Method and apparatus for processing image signal

Abstract

To provide a method for decoding an image signal.SOLUTION: The method comprises the steps of: determining, on the basis of a height and width of a current block, an input length and output length of a non-separable transform; determining a non-separable transform matrix related to the input length and output length of the non-separable transform; applying the non-separable transform matrix to coefficients of the current block, where the number of the coefficients is related to the input length of the non-separable transform; and performing primary inverse-transform on the coefficients to which the non-separable transform is applied. The input length and the output length of the non-separable transform are respectively determined, the input length of the non-separable transform being determined as 16 based on each of the height and the width of the current block being larger than 8, the output length of the non-separable transform being larger than the input length of the non-separable transform, the input length of the non-separable transform being determined as 8 based on each of the height and the width of the current block being equal to 8, and the output length of the non-separable transform being larger than the input length of the non-separable transform.SELECTED DRAWING: Figure 33

Description

[Technical Field]

[0001] The present invention relates to a method and apparatus for processing a video signal, and in particular to a method and apparatus for encoding or decoding a video signal by performing a transformation. [Background technology]

[0002] Compression encoding refers to a series of signal processing techniques used to transmit digitized information over communication lines or store it in a suitable format for storage media. Media such as video, images, and audio can be compressed and encoded, and the technology that compresses and encodes video in particular is called video compression.

[0003] Next-generation video content will be characterized by high spatial resolution, high frame rate, and high dimensionality of scene representation, and processing such content will bring about a huge increase in memory storage, memory access rate, and processing power.

[0004] Therefore, it is necessary to design coding tools for more efficiently processing next-generation video content. In particular, video codec standards following the high efficiency video coding (HEVC) standard require more accurate prediction techniques as well as efficient conversion techniques for converting spatial domain video signals into frequency domain. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE INVENTION Embodiments of the present invention seek to provide an image signal processing method and apparatus that applies a transform that has high coding efficiency and low complexity.

[0006] The technical problems to be solved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description. [Means for solving the problem]

[0007] A method for decoding an image signal according to an embodiment of the present invention includes the steps of determining an input length and an output length of a non-separable transform based on a height and a width of a current block, determining a non-separable transform matrix corresponding to the input length and the output length of the non-separable transform, and applying the non-separable transform matrix to a number of coefficients in the current block corresponding to the input length, wherein the height and width of the block are greater than or equal to 8, and if the height and width of the current block are each 8, the input length of the non-separable transform is determined to be 8.

[0008] Also, if the height and width of the current block are not 8, the input length of the non-separable transform is determined to be 16.

[0009] The output length is also determined to be 48 or 64.

[0010] In addition, the step of applying the non-separable transformation matrix to the current block includes the step of applying the non-separable transformation matrix to a 4x4 region in the upper left corner of the current block if the height and width are not each 8 and the product of the width and height is less than a critical value.

[0011] In addition, the step of determining the non-separable transform matrix includes the steps of determining a non-separable transform set index based on the intra prediction mode of the current block, determining a non-separable transform kernel corresponding to the non-separable transform index within a non-separable transform set included in the non-separable transform set index, and determining the non-separable transform matrix from the non-separable transform kernel based on the input length and the output length.

[0012] An image signal processing apparatus according to another embodiment of the present invention includes a memory for storing the image signal, and a processor coupled to the memory, wherein the processor is configured to determine an input length and an output length of a non-separable transform based on a height and a width of a current block, determine a non-separable transform matrix corresponding to the input length and the output length of the non-separable transform, and apply the non-separable transform matrix to a number of coefficients in the current block corresponding to the input length, wherein if the height and width of the current block are greater than or equal to 8, and if the height and width of the current block are each 8, the input length of the non-separable transform is determined to be 8, and the output length is determined to be greater than the input length and less than or equal to 64. [Effects of the Invention]

[0013] According to an embodiment of the present invention, a video coding method and apparatus having high coding efficiency and low complexity can be provided by applying a transformation based on the size of a current block.

[0014] The effects obtained by the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description. [Brief explanation of the drawings]

[0015] To facilitate understanding of the present invention, the accompanying drawings, which are included as part of the detailed description, provide embodiments of the present invention and, together with the detailed description, explain the technical features of the present invention.

[0016] [Figure 1] 1 shows a schematic block diagram of an encoding device according to an embodiment of the present invention, in which a video / image signal is encoded. [Figure 2] 1 shows a schematic block diagram of a decoding device according to an embodiment of the present invention, in which an image signal is decoded. [Figure 3] These are embodiments to which the present invention can be applied, and are figures for explaining block division structures using QT (quadtree: QT), FIG. 3B (binary tree: BT), FIG. 3C (ternary tree: TT), and FIG. 3D (asymmetric tree: AT). [Figure 4] 2 shows a schematic block diagram of a transform and quantization unit and an inverse quantization and inverse transform unit in the encoding device of FIG. 1, which is an embodiment to which the present invention is applied. [Figure 5] 1 shows a schematic block diagram of an inverse quantization and inverse transform unit in a decoding device according to an embodiment of the present invention; [Figure 6] 1 shows a flowchart of an embodiment to which the present invention is applied, in which a video signal is encoded by a primary transform and a secondary transform. [Figure 7] 1 shows a flowchart of an embodiment to which the present invention is applied, in which a video signal is decoded by secondary inverse transform and primary inverse transform. [Figure 8] 1 illustrates an example of a transform configuration group to which adaptive multiple transform (AMT) is applied according to an embodiment of the present invention. [Figure 9] 1 shows an example of an encoding flowchart in which AMT is applied according to an embodiment of the present invention. [Figure 10] 10 shows an example of a decoding flowchart in which AMT is applied according to an embodiment of the present invention. [Figure 11] 1 illustrates an example of a flowchart for encoding an AMT flag and an AMT index according to an embodiment of the present invention. [Figure 12]10 shows an example of a decoding flowchart for performing conversion based on an AMT flag and an AMT index. [Figure 13] 1 shows a diagram for explaining Givens rotation, which is an embodiment to which the present invention is applied. [Figure 14] This shows an embodiment to which the present invention is applied, in which a configuration of one round in 4×4 NSST is configured from a Givens rotation layer and permutation. [Figure 15] 10 illustrates an example of a configuration of a non-separable transform set for each intra prediction mode according to an embodiment of the present invention. [Figure 16] Three forward scan orders for transform coefficients or transform coefficient blocks applied in the HEVC (high efficiency video coding) standard: (a) diagonal scan, (b) horizontal scan, and (c) vertical scan. [Figure 17] 10 shows the positions of transform coefficients when a forward diagonal scan is applied to a 4×8 block in an embodiment to which the present invention is applied. [Figure 18] An example of an embodiment to which the present invention is applied, in which valid transform coefficients of two 4x4 blocks are merged into one block, is shown. [Figure 19] 10 shows an example of a method for constructing an NSST set mixed for each intra prediction mode, which is an embodiment to which the present invention is applied. [Figure 20] An embodiment to which the present invention is applied will be described below, showing an example of a method for selecting an NSST set (or kernel) taking into consideration an intra prediction mode and a transform block size. [Figure 21A] 1 shows an embodiment to which the present invention is applied, in which forward and inverse contracted transforms are performed. [Figure 21B] 1 shows an embodiment to which the present invention is applied, in which forward and inverse contracted transforms are performed. [Figure 22]10 shows an example of a decoding flowchart using a reduced transform according to an embodiment of the present invention. [Figure 23] 1 illustrates an example of a flowchart for applying a conditional contracted transformation according to an embodiment of the present invention. [Figure 24] 10 illustrates an example of a decoding flowchart for a secondary inverse transform to which a conditional contracted transform is applied, according to an embodiment of the present invention. [Figure 25A] 1 illustrates an example of a reduced transform and a reduced inverse transform according to an embodiment of the present invention. [Figure 25B] 1 illustrates an example of a reduced transform and a reduced inverse transform according to an embodiment of the present invention. [Figure 26A] 1 illustrates an example of a reduced transform and a reduced inverse transform according to an embodiment of the present invention. [Figure 26B] 1 illustrates an example of a reduced transform and a reduced inverse transform according to an embodiment of the present invention. [Figure 27] 1 illustrates an example of a domain to which a reduced quadratic transform is applied according to an embodiment of the present invention. [Figure 28] 10 illustrates a downscaled transformation according to a downscaling factor in accordance with an embodiment of the present invention. [Figure 29] 10 shows an example of an encoding flowchart for performing conversion, which is an embodiment to which the present invention is applied. [Figure 30] 10 shows an example of a decoding flowchart for performing conversion, which is an embodiment to which the present invention is applied. [Figure 31] 1 shows an example of a detailed block diagram of a conversion unit in an encoding device, which is an embodiment to which the present invention is applied. [Figure 32] 1 shows an example of a detailed block diagram of an inverse transform unit in a decoding device, which is an embodiment to which the present invention is applied. [Figure 33] 10 shows an example of a decoding flowchart in which a transformation according to an embodiment of the present invention is applied; [Figure 34] 1 shows an example of a block diagram of an apparatus for processing a video signal, which is an embodiment to which the present invention is applied. [Figure 35]1 shows an example of an image coding system as an embodiment to which the present invention is applied. [Figure 36] 1 is a structural diagram of a content streaming system according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below together with the accompanying drawings is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiments in which the present invention can be practiced. The following detailed description includes specific details to provide a thorough understanding of the present invention. However, those skilled in the art will understand that the present invention can be practiced without such specific details.

[0018] In some cases, in order to avoid obscuring the concept of the present invention, well-known structures and devices may be omitted or shown in block diagram form, focusing on the core functions of each structure and device.

[0019] In some cases, in order to avoid obscuring the concept of the present invention, well-known structures and devices may be omitted or shown in block diagram form, focusing on the core functions of each structure and device.

[0020] Specific terms used in the following description are provided to facilitate understanding of the present invention, and the use of such specific terms may be changed to other forms without departing from the technical spirit of the present invention. For example, in the case of a signal, data, sample, picture, frame, block, etc., they may be appropriately substituted and interpreted in each coding process.

[0021] Hereinafter, in this specification, the term "processing unit" refers to a unit in which encoding / decoding processes such as prediction, transformation, and / or quantization are performed. The term "processing unit" may also be interpreted to include a unit of a luma component and a unit of a chroma component. For example, the processing unit may correspond to a block, a coding unit (CU), a prediction unit (PU), or a transform unit (TU).

[0022] Furthermore, the processing unit may be interpreted as a unit of a luminance component or a unit of a chrominance component. For example, the processing unit may correspond to a CTB, CB, PU, ​​or TB of the luminance component. Alternatively, the processing unit may correspond to a CTB, CB, PU, ​​or TB of the chrominance component. Furthermore, without being limited thereto, the processing unit may be interpreted to include a unit of a luminance component and a unit of a chrominance component.

[0023] Furthermore, the processing units are not necessarily limited to square blocks, but may be configured in the form of polygons having three or more vertices.

[0024] In the following description, pixels, picture elements, or coefficients (transform coefficients or transform coefficients that have undergone linear transformation) are collectively referred to as samples. Using a sample means using pixel values, picture element values, or coefficients (transform coefficients or transform coefficients that have undergone linear transformation).

[0025] Hereinafter, a method for designing and applying a reduced secondary transform (RST) that takes into account worst-case computational complexity for encoding / decoding still or moving images will be described.

[0026] Embodiments of the present invention provide a method and apparatus for compressing images and videos. Compressed data is in the form of a bitstream, which can be stored in various types of storage or streamed over a network and transmitted to a terminal having a decoder. If a display device is installed in the terminal, the decoded image can be displayed on the display device, or the bitstream data can simply be stored. The method and apparatus proposed in embodiments of the present invention can be applied to both encoders and decoders, and can be applied to devices that generate bitstreams or devices that receive bitstreams, regardless of whether the terminal outputs the data via a display device.

[0027] An image compression device is composed of a prediction unit, a transform and quantization unit, and an entropy coding unit, and the schematic block diagrams of the encoding device and decoding device are shown in Figures 1 and 2. Among them, the transform and quantization unit subtracts a predicted signal from an original signal, converts the residual signal into a frequency domain signal using a transform such as DCT (discrete cosine transform)-2, and then applies quantization to significantly reduce the number of non-zero signals, enabling image compression.

[0028] FIG. 1 is a schematic block diagram of an encoding device according to an embodiment of the present invention, in which a video / image signal is encoded.

[0029] The image division unit 110 divides an input image (or picture, frame) input to the encoding device 100 into one or more processing units. For example, the processing units may be called coding units (CUs). In this case, the coding units are recursively divided from a coding tree unit (CTU) or a largest coding unit (LCU) using a quad-tree binary-tree (QTBT) structure. For example, one coding unit is divided into multiple coding units of deeper depths based on a quad-tree structure and / or a binary tree structure. In this case, for example, the quad-tree structure may be applied first, and then the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The coding procedure according to the present invention is performed based on the final coding unit that is not further divided. In this case, the largest coding unit may be immediately used as the final coding unit based on coding efficiency according to image characteristics, or the coding unit may be recursively divided into coding units of lower depth as needed, and the coding unit of the optimal size may be used as the final coding unit. Here, the coding procedure includes procedures such as prediction, transformation, and reconstruction, which will be described later. As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit are each divided or partitioned from the final coding unit. The prediction unit is a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0030] The term "unit" may be mixed with terms such as "block" or "area" in some cases. In a general case, an MxN block refers to a set of samples or transform coefficients consisting of M columns and N rows. A sample generally refers to a pixel or a pixel value, and may refer to only the value of a pixel / pixel of the luma component, or only the value of a pixel / pixel of the chroma component. A sample can be used as a term corresponding to one pixel or pel of a picture (or image).

[0031] The encoding apparatus 100 subtracts a prediction signal (predicted block, prediction sample array) output from the inter prediction unit 180 or the intra prediction unit 185 from an input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is transmitted to the conversion unit 120. In this case, as shown in the figure, a unit in the encoder 100 that subtracts the prediction signal (predicted block, prediction sample array) from the input image signal (original block, original sample array) may be referred to as a subtraction unit 115. The prediction unit performs prediction on a current block to be processed (hereinafter, referred to as a current block) and generates a predicted block including prediction samples for the current block. The prediction unit may determine whether intra prediction or inter prediction is to be applied on a current block or CU basis. The prediction unit generates various information related to prediction, such as prediction mode information, as will be described later in the description of each prediction mode, and transmits the information to the entropy encoding unit 190. The prediction information is encoded in the entropy encoding unit 190 and output in the form of a bitstream.

[0032] The intra prediction unit 185 predicts the current block by referring to samples in the current picture. The referenced samples are located either in the neighborhood of the current block or far away from it depending on the prediction mode. Prediction modes in intra prediction include a plurality of non-directional modes and a plurality of directional modes. Non-directional modes include, for example, DC mode and planar mode. The directional modes may include, for example, 33 directional prediction modes or 65 directional prediction modes depending on the degree of precision of the prediction direction. However, this is merely an example, and more or less directional prediction modes may be used depending on the settings. The intra prediction unit 185 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0033] The inter prediction unit 180 may derive a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. Here, to reduce the amount of motion information transmitted in the inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information includes a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be called collocated reference blocks, collocated CUs (colCUs), etc., and the reference picture including the temporal neighboring blocks may be called a collocated picture (colPic). For example, the inter predictor 180 may construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of a skip mode or a merge mode, the inter predictor 180 may use motion information of neighboring blocks as motion information for the current block. In the case of a skip mode, unlike in the merge mode, a residual signal may not be transmitted. In the case of a motion vector prediction (MVP) mode, the motion vector of a neighboring block may be used as a motion vector predictor, and the motion vector of the current block may be indicated by signaling a motion vector difference.

[0034] The prediction signal generated by the inter predictor 180 or the intra predictor 185 is used to generate a reconstructed signal or a residual signal.

[0035] The transform unit 120 generates transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of DCT, DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to a transform obtained from a graph representing inter-pixel relationship information. CNT refers to a transform obtained based on a prediction signal generated using all previously reconstructed pixels. The transform process may be applied to pixel blocks having the same square size or to non-square blocks of variable size.

[0036] The quantizer 130 quantizes the transform coefficients and transmits them to the entropy encoder 190, which then encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantizer 130 may rearrange the quantized transform coefficients in a block format into a one-dimensional vector format based on a coefficient scan order and generate information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector format. The entropy encoder 190 may perform various encoding methods, such as exponential Golomb, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). In addition to the quantized transform coefficients, the entropy encoder 190 may also encode information required for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients. The encoded information (e.g., video / image information) is transmitted or stored in the form of a bitstream in NAL (network abstraction layer) units. The bitstream is transmitted via a network or stored in a digital storage medium. Here, the network includes a broadcasting network and / or a communication network, and the digital storage medium includes various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. A transmitter (not shown) that transmits and / or a storage unit (not shown) that stores the signal output from the entropy encoding unit 190 may be configured as an internal / external element of the encoding device 100, or the transmitter may be a component of the entropy encoding unit 190.

[0037] The quantized transform coefficients output from the quantization unit 130 may be used to generate a prediction signal. For example, the quantized transform coefficients may be subjected to inverse quantization and inverse transformation by the inverse quantization unit 140 and the inverse transform unit 150 in a loop to reconstruct a residual signal. The adder 155 generates a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185. When there is no residual for the current block, such as when a skip mode is applied, a predicted block may be used as the reconstructed block. The adder 155 may also be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next current block in the current picture, or may be used for inter prediction of the next picture after filtering, as described below.

[0038] The filtering unit 160 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 160 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and transmit the modified reconstructed picture to the decoded picture buffer 170. Examples of the various filtering methods include deblocking filtering, sample adaptive offset, adaptive loop filter, and bilateral filter. The filtering unit 160 generates various information related to filtering, as will be described later in the description of each filtering method, and transmits the generated information to the entropy encoding unit 190. The entropy encoding unit 190 encodes the filtering information and outputs it in the form of a bitstream.

[0039] The modified decoded picture sent to the decoded picture buffer 170 is used as a reference picture in the inter prediction unit 180. This allows the encoding device 100 to avoid prediction mismatch between the encoding device 100 and the decoding device when inter prediction is applied, and also improves coding efficiency.

[0040] The decoded picture buffer 170 can store the modified reconstructed picture for use as a reference picture in the inter predictor 180 .

[0041] FIG. 2 is a schematic block diagram of a decoding device that decodes an image signal, according to an embodiment of the present invention.

[0042] As shown in FIG. 2, the decoding device 200 includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an adder 235, a filtering unit 240, a decoded picture buffer (DPB) 250, an inter prediction unit 260, and an intra prediction unit 265. The inter prediction unit 260 and the intra prediction unit 265 may be collectively referred to as a prediction unit. That is, the prediction unit includes an inter prediction unit 180 and an intra prediction unit 185. The inverse quantization unit 220 and the inverse transform unit 230 may be collectively referred to as a residual processing unit. That is, the residual processing unit includes the inverse quantization unit 220 and the inverse transform unit 230. The entropy decoding unit 210, the inverse quantization unit 220, the inverse transform unit 230, the adder 235, the filtering unit 240, the inter prediction unit 260, and the intra prediction unit 265 may be configured as a single hardware component (e.g., a decoder or a processor) according to some embodiments. Also, the decoded picture buffer 250 may be implemented as a single hardware component (eg, a memory or a digital storage medium) depending on the embodiment.

[0043] When a bitstream including video / image information is input, the decoding device 200 can reconstruct an image corresponding to the process by which the video / image information was processed in the encoding device 100 of Fig. 2. For example, the decoding device 200 performs decoding using the processing unit applied in the encoding device 100. Therefore, the processing unit for decoding is, for example, a coding unit, and the coding unit is divided into a coding tree unit or a maximum coding unit according to a quad tree structure and / or a binary tree structure. The reconstructed image signal decoded and output by the decoding device 200 is reproduced by a playback device.

[0044] The decoding device 200 receives a signal output from the encoding device 100 of FIG. 2 in the form of a bitstream, and the received signal is decoded by an entropy decoding unit 210. For example, the entropy decoding unit 210 parses the bitstream to derive information (e.g., video / image information) necessary for image reconstruction (or picture reconstruction). For example, the entropy decoding unit 210 may decode information in the bitstream based on a coding method such as Exponential-Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements necessary for image reconstruction and quantized values ​​of transform coefficients related to residuals. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using information about the syntax element to be decoded and decoding information about neighboring and current blocks or information about symbols / bins decoded in a previous step, predicts the occurrence probability of the bin according to the determined context model, and performs arithmetic decoding of the bin to generate a symbol corresponding to the value of each syntax element. Here, after determining the context model, the CABAC entropy decoding method may update the context model using information about the decoded symbol / bin for the context model of the next symbol / bin. Prediction information from the information decoded by the entropy decoding unit 210 is provided to the prediction unit (inter prediction unit 260 and intra prediction unit 265), and residual values ​​entropy decoded by the entropy decoding unit 210, i.e., quantized transform coefficients and related parameter information, are input to the inverse quantization unit 220. In addition, information regarding filtering among the information decoded by the entropy decoding unit 210 is provided to the filtering unit 240. Meanwhile, a receiving unit (not shown) for receiving a signal output from the encoding device 100 may be further configured as an internal / external element of the decoding device 200, or the receiving unit may be a component of the entropy decoding unit 210.

[0045] The inverse quantization unit 220 inverse quantizes the quantized transform coefficients and outputs transform coefficients. The inverse quantization unit 220 rearranges the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement may be performed based on the coefficient scanning order performed in the encoding apparatus 100. The inverse quantization unit 220 inverse quantizes the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0046] The inverse transform unit 230 obtains a residual signal (residual block, residual sample array) by inverse transforming the transform coefficients.

[0047] The prediction unit performs prediction on the current block and generates a predicted block including prediction samples for the current block. The prediction unit determines whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 210, and may determine a specific intra / inter prediction mode.

[0048] The intra prediction unit 265 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or may be located far away from the current block depending on the prediction mode. Prediction modes in intra prediction include a plurality of non-directional modes and a plurality of directional modes. The intra prediction unit 265 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0049] The inter prediction unit 260 derives a predicted block for a current block based on a reference block (reference sample array) identified by a motion vector on a reference picture. Here, to reduce the amount of motion information transmitted in inter prediction mode, the motion information is predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include information on the inter prediction direction (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the inter prediction unit 260 constructs a motion information candidate list based on information related to the prediction of the neighboring blocks and derives a motion vector and / or a reference picture index for the current block based on the received candidate selection information. Inter prediction is performed based on various prediction modes, and the prediction information includes information indicating the inter prediction mode for the current block.

[0050] The adder 235 generates a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a predicted signal (predicted block, predicted sample array) output from the inter prediction unit 260 or the intra prediction unit 265. When there is no residual for the current block, such as when the skip mode is applied, the predicted block can be used as the reconstructed block.

[0051] The adder 235 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of the next block to be processed in the current picture, or may be used for inter prediction of the next picture after filtering as described below.

[0052] The filtering unit 240 can improve the subjective / objective image quality by applying filtering to the reconstructed signal. For example, the filtering unit 240 applies various filtering methods to the reconstructed picture to generate a modified reconstructed picture and transmits the modified reconstructed picture to the decoded picture buffer 250. The various filtering methods include, for example, deblocking filtering, sample adaptive offset (SAO), adaptive loop filter (ALF), and bilateral filter.

[0053] The modified decoded picture sent to the decoded picture buffer 250 can be used as a reference picture by the inter predictor 260 .

[0054] In this document, the embodiments described for the filtering unit 160, inter prediction unit 180, and intra prediction unit 185 of the encoding device 100 are also applied identically or correspondingly to the filtering unit 240, inter prediction unit 260, and intra prediction unit 265 of the decoding device 200, respectively.

[0055] Figure 3 shows an embodiment to which the present invention can be applied, with Figure 3A being a diagram for explaining a block division structure using QT (quadtree: QT), Figure 3B being a diagram for explaining a block division structure using BT (binary tree: BT), Figure 3C being a diagram for explaining a block division structure using TT (ternary tree: TT), and Figure 3D being a diagram for explaining a block division structure using AT (asymmetric tree: AT).

[0056] In video coding, a block can be divided based on the QT. A subblock divided by the QT can be further divided recursively using the QT. A leaf block that is not further divided by the QT is divided by at least one of BT, TT, or AT. BT can have two types of division: horizontal BT (2N×N, 2N×N) and vertical BT (N×2N, N×2N). TT can have two types of division: horizontal TT (2N×½N, 2N×N, 2N×½N) and vertical TT (½N×2N, N×2N, ½N×2N). The AT can have four types of division: horizontal-up AT (2N x 1 / 2N, 2N x 3 / 2N), horizontal-down AT (2N x 3 / 2N, 2N x 1 / 2N), vertical-left AT (1 / 2N x 2N, 3 / 2N x 2N), and vertical-right AT (3 / 2N x 2N, 1 / 2N x 2N). Each BT, TT, and AT may be further divided recursively using BT, TT, and AT.

[0057] Figure 3A shows an example of QT division. Block A is divided into four sub-blocks (A0, A1, A2, A3) by QT. Sub-block A1 is again divided into four sub-blocks (B0, B1, B2, B3) by QT.

[0058] 3B shows an example of BT division. Block B3, which is not further divided by QT, is divided into vertical BT(C0, C1) or horizontal BT(D0, D1). Like block C0, each sub-block is further divided recursively into horizontal BT(E0, E1) or vertical BT(F0, F1).

[0059] 3C shows an example of TT division. Block B3, which is not further divided by QT, is divided into vertical TTs (C0, C1, C2) or horizontal TTs (D0, D1, D2). Like block C1, each sub-block is further divided recursively into horizontal TTs (E0, E1, E2) or vertical TTs (F0, F1, F2).

[0060] 3D shows an example of AT division. Block B3, which is not further divided by QT, is divided into vertical AT(C0, C1) or horizontal AT(D0, D1). Like block C1, each sub-block can be further divided recursively, such as horizontal AT(E0, E1) or vertical TT(F0, F1).

[0061] Meanwhile, BT, TT, and AT divisions may be used together. For example, sub-blocks divided by BT can be divided by TT or AT. Also, sub-blocks divided by TT can be divided by BT or AT. Sub-blocks divided by AT can be divided by BT or TT. For example, after horizontal BT division, each sub-block can be divided by vertical BT, or after vertical BT division, each sub-block can be divided by horizontal BT. In this case, the division order is different, but the final divided shape is the same.

[0062] Also, when a block is divided, the order of searching the block can be defined in various ways. Generally, the search is performed from left to right and from top to bottom, and searching the block means the order of determining whether each divided sub-block can be further divided into blocks, or the order of encoding each sub-block if the block is not further divided, or the order of searching when referring to information of other neighboring blocks in a sub-block.

[0063] Transforms can be performed for each processing unit (or transform block) divided according to the division structure shown in Figures 3A to 3D, and in particular, transform matrices can be applied by dividing the processing unit into rows and columns. According to an embodiment of the present invention, different transform types can be used depending on the length of the processing unit (or transform block) in the row or column direction.

[0064] 4 and 5 show embodiments to which the present invention is applied. FIG. 4 shows a schematic block diagram of the transform and quantization unit 120 / 130 and the inverse quantization and inverse transform unit 140 / 150 in the encoding device 100 of FIG. 1, and FIG. 5 shows a schematic block diagram of the inverse quantization and inverse transform unit 220 / 230 in the decoding device 200.

[0065] 4, the transform and quantization unit 120 / 130 includes a primary transform unit 121, a secondary transform unit 122, and a quantization unit 130. The inverse quantization and inverse transform unit 140 / 150 includes an inverse quantization unit 140, an inverse secondary transform unit 151, and an inverse primary transform unit 152.

[0066] As shown in FIG. 5, the inverse quantization and inverse transform unit 220 / 230 includes an inverse quantization unit 220, an inverse secondary transform unit 231 and an inverse primary transform unit 232.

[0067] In the present invention, a transformation is performed through multiple stages. For example, as shown in Figure 4, two stages, a primary transform and a secondary transform, can be applied, or more stages can be used depending on the algorithm. Here, the primary transform may be referred to as a core transform.

[0068] The linear transform unit 121 applies a linear transform to the residual signal, where the linear transform may already be defined as a table in the encoder and / or decoder.

[0069] The secondary transform unit 122 applies a secondary transform to the primary transformed signal, where the secondary transform may already be defined as a table in the encoder and / or decoder.

[0070] In one embodiment, a non-separable secondary transform (NSST) may be conditionally applied as the secondary transform. For example, the NSST may be applied only to a prediction block within a frame, and each prediction mode group may have a set of applicable transforms.

[0071] Here, the prediction mode group is set based on symmetry with respect to the prediction direction. For example, since prediction mode 52 and prediction mode 16 are symmetric with respect to prediction mode 34 (diagonal direction), they form one group and the same transform set can be applied. Here, when applying a transform to prediction mode 52, the input data is transposed before application, because the transform set is the same as that of prediction mode 16.

[0072] On the other hand, in the case of planar mode and DC mode, there is no symmetry with respect to direction, so each mode has its own transformation set, which can be composed of two transformations, while for the remaining directional modes, each transformation set can be composed of three transformations.

[0073] The quantization unit 130 quantizes the secondary transformed signal.

[0074] The inverse quantization and inverse transformation unit 140 / 150 performs the above-described process in reverse, and a duplicated description will be omitted.

[0075] FIG. 5 shows a schematic block diagram of the inverse quantization and inverse transform unit 220 / 230 in the decoding device 200.

[0076] As shown in FIG. 5, the inverse quantization and inverse transform unit 220 / 230 includes an inverse quantization unit 220, an inverse secondary transform unit 231 and an inverse primary transform unit 232.

[0077] The inverse quantization unit 220 obtains transform coefficients from the entropy decoded signal using the quantization step size information.

[0078] Inverse secondary transform unit 231 performs an inverse secondary transform on the transform coefficients. Here, the inverse secondary transform refers to the inverse transform of the secondary transform described with reference to FIG.

[0079] The inverse primary transform unit 232 performs an inverse primary transform on the inverse secondary transformed signal (or block) to obtain a residual signal. Here, the inverse primary transform refers to the inverse transform of the primary transform described in FIG. 4.

[0080] 6 is a flowchart showing an embodiment of the present invention, in which a video signal is encoded by primary and secondary conversion. The operations shown in FIG. 6 are performed by the conversion unit 120 of the encoding device 100.

[0081] The encoding apparatus 100 determines (or selects) a forward secondary transform based on at least one of the prediction mode, block shape, and / or block size of the current block (S610).

[0082] The encoding device 100 can determine an optimal forward secondary transform by rate-distortion optimization (RD optimization). The optimal forward secondary transform corresponds to one of a plurality of transform combinations, and the plurality of transform combinations are defined by transform indexes. For example, for RD optimization, the encoding device 100 can compare the results of performing all of forward secondary transform, quantization, residual coding, etc. for each candidate.

[0083] The encoding device 100 signals (S620) a secondary transform index corresponding to the optimal forward secondary transform. Here, the secondary transform index can be applied to other embodiments described in this specification.

[0084] On the other hand, the encoding device 100 performs a forward primary transform on the current block (residual block) (S630).

[0085] The encoding device 100 performs a forward secondary transform on the current block using the optimal forward secondary transform (S640). On the other hand, the forward secondary transform can be an RST described below. RST means a transform in which N pieces of residual data (N×1 residual vector) are input and R pieces of transform coefficient data (R×1 transform coefficient vector) (R < N) are output.

[0086] As one embodiment, the RST can be applied to a specific region of the current block. For example, when the current block is N×N, the specific region can mean the upper left N / 2×N / 2 region. However, the present invention is not limited to this, and is set to vary depending on at least one of the prediction mode, block shape, or block size. For example, when the current block is N×N, the specific region can mean the upper left M×M region (M≦N).

[0087] On the other hand, the encoding device 100 generates a transform coefficient block by performing quantization on the current block (S650).

[0088] The encoding device 100 can perform entropy encoding on the transform coefficient blocks to generate a bitstream.

[0089] 7 is a flowchart showing an embodiment of the present invention, in which a video signal is decoded by secondary inverse transform and primary inverse transform. Each operation shown in FIG. 7 is performed by the inverse transform unit 230 of the decoding device 200.

[0090] The decoding device 200 obtains the secondary transform index from the bitstream (S710).

[0091] The decoding device 200 derives a secondary transform corresponding to the secondary transform index (S720).

[0092] However, steps S710 and S720 are merely exemplary embodiments, and the present invention is not limited thereto. For example, the decoding apparatus 200 may derive the secondary transform based on at least one of the prediction mode, block shape, and / or block size of the current block without obtaining the secondary transform index.

[0093] Meanwhile, the decoder 200 entropy decodes the bitstream to obtain a transform coefficient block, and performs inverse quantization on the transform coefficient block (S730).

[0094] The decoder 200 performs an inverse secondary transform on the dequantized transform coefficient block (S740). For example, the inverse secondary transform may be an inverse RST. The inverse RST is a transpose matrix of the RST described in FIG. 6, and refers to a transform in which R transform coefficient data (Rx1 transform coefficient vector) are input and N residual data (Nx1 residual vector) are output.

[0095] In one embodiment, the scaled secondary transform may be applied to a specific region of the current block. For example, if the current block is an N×N block, the specific region may refer to the N / 2×N / 2 region on the upper left side. However, the present invention is not limited thereto and may be set differently depending on at least one of the prediction mode, the block shape, and the block size. For example, if the current block is an N×N block, the specific region may refer to an M×M region (M≦N) or an M×L region (M≦N, L≦N) on the upper left side.

[0096] Then, the decoder 200 performs an inverse linear transform on the result of the inverse secondary transform (S750).

[0097] The decoder 200 generates a residual block in step S750, and generates a reconstructed block by adding the residual block and the predicted block.

[0098] FIG. 8 illustrates an example of a transform configuration group to which an adaptive multiple transform (AMT) is applied according to an embodiment of the present invention.

[0099] 8, the transform setting group is determined based on the prediction mode, and there may be a total of six groups (G0 to G5). G0 to G4 correspond to cases where intra prediction is applied, and G5 indicates a transform combination (or a transform set, or a transform combination set) applied to a residual block generated by inter prediction.

[0100] A transformation combination consists of a horizontal transform (or row transform) applied to the rows of the corresponding two-dimensional block and a vertical transform (or column transform) applied to the columns.

[0101] Here, each of all transform setting groups includes four transform combination candidates, which are selected or determined by transform combination indexes of 0 to 3, and the transform combination indexes are transmitted from the encoding device 100 to the decoding device 200 in an encoding procedure.

[0102] In one embodiment, residual data (or residual signals) obtained by intra prediction have different statistical characteristics depending on the intra prediction mode. Therefore, a transform other than the general cosine transform may be applied depending on the intra prediction mode as shown in Figure 8. In this document, the transform type may be expressed as, for example, DCT-Type 2, DCT-II, or DCT-2.

[0103] 8, transform set configurations are shown for the cases where 35 intra prediction modes are used and where 67 intra prediction modes are used. A plurality of transform combinations can be applied to each transform setting group, which is divided into the intra prediction mode column. For example, a plurality of transform combinations (row-direction transform, column-direction transform) are configured into four combinations. More specifically, in group 0, DST-7 and DCT-5 can be applied to both the row (horizontal) and column (vertical) directions, so four combinations are possible.

[0104] Since a total of four transform kernel combinations can be applied to each intra prediction mode, a transform combination index for selecting one of them is transmitted for each transform unit. In this document, the transform combination index is referred to as an AMT index and may be expressed as amt_idx.

[0105] In addition to the transform kernels shown in Fig. 8, there are cases where DCT-2 is optimal for both the row and column directions due to the characteristics of the residual signal. Therefore, adaptive transforms can be performed by defining an AMT flag for each coding unit. Here, when the AMT flag is 0, DCT-2 is applied to both the row and column directions, and when the AMT flag is 1, one of four combinations can be selected or determined by the AMT index.

[0106] In one embodiment, when the AMT flag is 0, if the number of transform coefficients for one transform unit is less than 3, the transform kernel of FIG. 8 is not applied, and DST-7 is applied to all rows and columns.

[0107] In one embodiment, the transform coefficient values ​​are parsed first, and if the number of transform coefficients is less than 3, the amount of additional information to be transmitted can be reduced by applying DST-7 without parsing the AMT index.

[0108] In one embodiment, AMT is applicable only if the width and height of the transformation unit are both 32 or less.

[0109] In one embodiment, FIG. 8 may be pre-configured through off-line training.

[0110] In one embodiment, the AMT index may be defined by a single index that can simultaneously indicate a combination of horizontal and vertical transforms, or the AMT index may be defined separately by a horizontal transform index and a vertical transform index.

[0111] A technique for applying a selected transform from multiple transform kernels (e.g., DCT-2, DST-7, DCT-8) such as the aforementioned AMT may be referred to as MTS (multiple transform selection) or EMT (enhanced multiple transform), and an AMT index may be referred to as an MT index.

[0112] 9 shows an example of an encoding flowchart to which AMT is applied according to an embodiment of the present invention. The operations shown in FIG.

[0113] Although this document describes an embodiment that primarily applies transforms separately for the horizontal and vertical directions, the transform combination can also consist of non-separable transforms.

[0114] It can also be configured as a mixture of separable and non-separable transforms. In this case, if a non-separable transform is used, there is no need to select a transform by row / column or by horizontal / vertical direction, and the transform combinations shown in Figure 8 are used only when a separable transform is selected.

[0115] In addition, the method proposed in this specification can be applied regardless of whether it is a primary transform or a secondary transform. That is, there is no restriction that it must be applied to only one of the two, and it can be applied to both. Here, the primary transform may refer to a transform for first transforming a residual block, and the secondary transform may refer to a transform for applying a transform to a block generated as a result of the primary transform.

[0116] First, the encoding apparatus 100 determines a transform setting group corresponding to a current block (S910). Here, the transform setting group may be configured as shown in FIG.

[0117] The encoding device 100 performs a transform on a combination of candidate transforms available in the transform setting group (S920).

[0118] As a result of the transformation, the encoding apparatus 100 determines or selects the transformation combination with the smallest RD (rate distortion) cost (S930).

[0119] The encoding device 100 encodes a transform combination index corresponding to the selected transform combination (S940).

[0120] 10 shows an example of a decoding flowchart to which AMT is applied according to an embodiment of the present invention. The operation shown in FIG.

[0121] First, the decoding device 200 determines a transform setting group for a current block (S1010). The decoding device 200 parses (or obtains) a transform combination index from the video signal, where the transform combination index corresponds to one of a plurality of transform combinations in the transform setting group (S1020). For example, the transform setting group may include DCT-2, DCT-7, or DCT-8.

[0122] The decoding apparatus 200 derives a transform combination corresponding to the transform combination index (S1030). Here, the transform combination includes a horizontal transform and a vertical transform, and includes at least one of DCT-2, DCT-7, and DCT-8. The transform combination may be the transform combination described in FIG. 8.

[0123] The decoding apparatus 200 performs an inverse transform on the current block based on the derived transform combination (S1040). If the transform combination is composed of a row (horizontal) transform and a column (vertical) transform, the row (horizontal) transform may be applied first, followed by the column (vertical) transform. However, the present invention is not limited thereto, and if the transform combination is composed of inversely applied or non-separable transforms, the non-separable transform may be applied immediately.

[0124] In one embodiment, when the vertical or horizontal transform is a DST-7 or DCT-8, the inverse DST-7 or DCT-8 transform is applied to each column and then to each row, and a different vertical or horizontal transform is applied to each row and / or each column.

[0125] In one embodiment, the transform combination index may be obtained based on an AMT flag indicating whether AMT is performed. That is, the transform combination index may be obtained only if AMT is performed according to the AMT flag. In addition, the decoding apparatus 200 checks whether the number of non-zero transform coefficients is greater than a threshold value. Here, the transform combination index may be parsed only if the number of non-zero transform coefficients is greater than the threshold value.

[0126] In one embodiment, the AMT flag or AMT index is defined at least at one level of a sequence, a picture, a slice, a block, a coding unit, a transform unit, or a prediction unit.

[0127] Meanwhile, in another embodiment, the process of determining the transform setting group and the process of parsing the transform combination index may be performed simultaneously, or step S1010 may be omitted since they are already configured in the encoding device 100 and / or the decoding device 200.

[0128] 11 shows an example of a flowchart for encoding an AMT flag and an AMT index according to an embodiment of the present invention. The operation of FIG.

[0129] Encoding apparatus 100 determines whether AMT is applied to the current block (S1110).

[0130] If AMT is applied, encoding device 100 encodes with AMT flag=1 (S1120).

[0131] Then, the encoding apparatus 100 determines an AMT index based on at least one of the prediction mode, horizontal transform, and vertical transform of the current block (S1130). Here, the AMT index indicates an index that points to one of a plurality of transform combinations for each intra prediction mode, and the AMT index is transmitted for each transform unit.

[0132] When the AMT index is determined, the encoding device 100 encodes the AMT index (S1140).

[0133] On the other hand, if AMT is not applied, the encoding device 100 performs encoding with the AMT flag set to 0 (S1150).

[0134] FIG. 12 shows an example of a decoding flowchart for performing conversion based on the AMT flag and the AMT index.

[0135] The decoding device 200 parses the AMT flag from the bitstream (S1210), where the AMT flag indicates whether AMT is applied to the current block.

[0136] The decoding device 200 determines whether AMT is applied to the current block based on the AMT flag (S1220). For example, it determines whether the AMT flag is 1.

[0137] If the AMT flag is 1, the decoding apparatus 200 parses the AMT index (S1230). Here, the AMT index refers to an index indicating one of a plurality of transform combinations for each intra prediction mode, and the AMT index can be transmitted for each transform unit. Alternatively, the AMT index refers to an index indicating one of the transform combinations defined in a pre-established transform combination table, and here, the pre-established transform combination table may refer to FIG. 8, but the present invention is not limited thereto.

[0138] The decoding device 200 derives or determines horizontal and vertical transforms based on at least one of the AMT index or the prediction mode (S1240).

[0139] Alternatively, the decoding device 200 derives a transform combination corresponding to the AMT index. For example, the decoding device 200 derives or determines a horizontal transform and a vertical transform corresponding to the AMT index.

[0140] On the other hand, if the AMT flag is 0, the decoding apparatus 200 applies the previously set inverse vertical transform to each column (S1250). For example, the inverse vertical transform may be the inverse transform of DCT-2.

[0141] Then, the decoding device 200 applies the previously set horizontal inverse transform to each row (S1260). For example, the horizontal inverse transform may be an inverse transform of DCT-2. That is, when the AMT flag is 0, a transform kernel previously set in the encoding device 100 or the decoding device 200 is used. For example, a commonly used transform kernel may be used instead of one defined in a transform combination table such as that shown in FIG. 8.

[0142] NSST (non-separable secondary transform)

[0143] A secondary transform is a process in which a transform kernel is applied again to the result of a primary transform. Primary transforms include DCT-2, DST-7, and the AMT described above in HEVC. A non-separable transform is a process in which an NxN 2D residual block is divided into N 2 ×1 vector, and then apply N 2 ×N 2 This refers to applying the transformation kernel only once.

[0144] That is, NSST refers to a non-separable square matrix applied to a vector composed of coefficients of a transform block. Although the embodiments of this document will be described focusing on NSST as an example of a non-separable transform applied to the upper left region (low frequency region) determined by the size of the block, the embodiments of the present invention are not limited to the term NSST, and any type of non-separable transform can be applied to the embodiments of the present invention. For example, a non-separable transform applied to the upper left region (low frequency region) determined by the size of the block is called an LFNST (low frequency non-separable transform). In this document, an M×N transform (or transform matrix) refers to a matrix composed of M rows and N columns.

[0145] In NSST, the two-dimensional block data obtained by applying a linear transformation is divided into M × M blocks, and then M 2 ×M 2A non-separable transform is applied. The value of M can be 4 or 8. NSST can be applied only to a portion of the 2D block obtained by the linear transform, rather than to the entire region. For example, NSST can be applied only to the top-left 8x8 block. Also, only if the width and height of the 2D block obtained by the linear transform are both 8 or greater, can a 64x64 non-separable transform be applied to the top-left 8x8 region. The remaining cases can be divided into 4x4 blocks and the corresponding 16x16 non-separable transform be applied to each.

[0146] M 2 ×M 2 Non-separable transformations can be applied in the form of matrix multiplication, but to reduce computational complexity and memory requirements, they can be approximated by a combination of a Givens rotation layer and a permutation layer. Figure 13 shows one Givens rotation. As shown in Figure 13, it can be explained by one angle of one Givens rotation.

[0147] 13 and 14 show embodiments to which the present invention is applied. FIG. 13 shows a diagram for explaining Givens rotation, and FIG. 14 shows the configuration of one round in 4x4 NSST consisting of a Givens rotation layer and permutation.

[0148] Both the 8x8NSST and the 4x4NSST can be constructed by a hierarchical combination of Givens rotations. The matrix corresponding to one Givens rotation is shown in Equation 1, and the matrix multiplication can be represented diagrammatically as shown in Figure 13.

[0149]

number

[0150] In Figure 13, the output t m and t ncan be calculated as in Equation 2.

[0151]

number

[0152] As shown in Figure 13, one Givens rotation rotates two data pieces, so 32 or 8 Givens rotations are required to process 64 data pieces (for 8x8 NSST) or 16 data pieces (for 4x4 NSST), respectively. Therefore, a bundle of 32 or 8 Givens rotations can form a Givens rotation layer. As shown in Figure 15, the output data for one Givens rotation layer is shuffled and transmitted to the input data for the next Givens rotation layer. As shown in Figure 15, the permutation pattern is regularly defined. In the case of 4x4 NSST, four Givens rotation layers and corresponding permutations form one round. 4x4 NSST is performed in two rounds, while 8x8 NSST is performed in four rounds. Different rounds use the same permutation pattern, but the applied Givens rotation angles are different. Therefore, angle data for all Givens rotations that make up each transformation must be stored.

[0153] As a final step, a final permutation is applied to the data output from the Givens rotation layer, and information about this permutation is stored separately for each transformation. This permutation is applied at the end of the forward NSST, and the inverse permutation is applied first in the inverse NSST.

[0154] In the reverse NSST, the Givens rotation layers and permutations applied in the forward NSST are performed in the reverse order, and the rotation is performed by taking a negative (-) value for each Givens rotation angle.

[0155] FIG. 15 illustrates an example of a configuration of a non-separable transform set for each intra prediction mode according to an embodiment of the present invention.

[0156] Intra prediction modes that use the same NSST or NSST set may form a group. In Figure 15, 67 intra prediction modes are classified into 35 groups. For example, modes 20 and 48 both belong to group 20 (hereinafter referred to as mode group).

[0157] For each mode group, multiple NSSTs can be configured as a set, rather than a single NSST. Each set includes a case where NSST is not applied. For example, if three different NSSTs can be applied to one mode group, it can be configured to select one of four cases, including the case where NSST is not applied. Here, an index is transmitted in TU units to distinguish one of the four cases. The number of NSSTs can be configured to differ for each mode group. For example, mode groups 0 and 1 are signaled to select one of three cases, including the case where NSST is not applied.

[0158] Embodiment 1: RST applicable to 4x4 blocks

[0159] A non-separable transform that can be applied to a 4x4 block is a 16x16 transform. That is, when the data elements constituting the 4x4 block are arranged in a row-first or column-first order, a 16x1 vector is formed, and the non-separable transform can be applied to the 16x1 vector. A forward 16x16 transform is composed of 16 row-wise transform basis vectors. The inner product of the 16x1 vector and each transform basis vector is used to obtain the transform coefficients for the corresponding transform basis vector. The process of obtaining the transform coefficients for all 16 transform basis vectors is similar to multiplying a 16x16 non-separable transform matrix by the input 16x1 vector. The transform coefficients obtained by the matrix multiplication have the form of a 16x1 vector, but the statistical properties of each transform coefficient may differ. For example, if a 16x1 transform coefficient vector is composed of the 0th element to the 15th element, the variance of the 0th element may be greater than the variance of the 15th element. That is, the earlier an element is located, the greater its variance value and therefore the greater its energy value.

[0160] Applying an inverse 16x16 non-separable transform to the 16x1 transform coefficients can restore the original 4x4 block signal (ignoring the effects of quantization, integerization, etc.). If the forward 16x16 non-separable transform is an orthonormal transform, the inverse 16x16 transform can be obtained by taking the matrix transpose of the forward 16x16 transform. Simply put, multiplying the 16x1 transform coefficient vector by the inverse 16x16 non-separable transform matrix results in data in the form of a 16x1 vector, which can be arranged in the row-major or column-major order as originally applied to restore the 4x4 block signal.

[0161] As mentioned above, the elements of a 16×1 transform coefficient vector may have different statistical characteristics. As in the previous example, if the transform coefficients located earlier (closer to the 0th element) have greater energy, a signal fairly close to the original signal can be restored even if an inverse transform is applied to some of the earlier transform coefficients without using all of the transform coefficients. For example, if an inverse 16×16 non-separable transform is composed of 16 column basis vectors, only L column basis vectors are left to form a 16×L matrix. After leaving only the most important L transform coefficients from the transform coefficients (an L×1 vector, which can appear first as in the previous example), multiplying the 16×L matrix by the L×1 vector restores a 16×1 vector with a small error compared to the original input 16×1 vector data. As a result, because only L coefficients are involved in data restoration, an L×1 transform coefficient vector, not a 16×1 transform coefficient vector, can be obtained to obtain the transform coefficients. That is, in a forward 16x16 non-separable transformation matrix, L relevant row-direction transformation vectors can be selected to form an Lx16 transformation, and then multiplied with a 16x1 input vector to obtain L significant transformation coefficients.

[0162] Embodiment 2: Setting the application area of ​​4x4RST and arranging the conversion coefficients

[0163] 4x4RST can be applied as a secondary transform, and in this case, it can be applied secondarily to a block that has already undergone a primary transform such as DCT-type 2. When the size of a block to which a primary transform has been applied is NxN, it is usually larger than 4x4. Therefore, when applying 4x4RST to an NxN block, the following two methods are possible:

[0164] 1) Instead of applying 4x4RST to the entire NxN region, it can be applied only to a portion of the region. For example, it can be applied only to the top-left MxM region (M<=N).

[0165] 2) After dividing the region to which the secondary transformation is applied into 4x4 blocks, 4x4 RST can be applied to each divided block.

[0166] The above methods 1) and 2) can be mixed and applied. For example, after dividing only the upper left M×M region into 4×4 blocks, 4×4 RST can be applied.

[0167] In a specific embodiment, a secondary transformation is applied only to the upper left 8x8 region, and if the NxN block is greater than or equal to 8x8, an 8x8 RST is applied. If the NxN block is smaller than 8x8 (4x4, 8x4, 4x8), it is divided into 4x4 blocks as in 2) above, and then a 4x4 RST is applied to each.

[0168] When L transform coefficients (1<=L<16) are generated after applying 4×4 RST, there is a degree of freedom in how to arrange the L transform coefficients. However, since there is a predetermined order when reading and processing the transform coefficients in the residual coding part, coding performance may vary depending on how the L transform coefficients are arranged in a two-dimensional block. In the high efficiency video coding (HEVC) standard, residual coding starts from the position farthest from the DC position. This is to improve coding performance by taking advantage of the fact that the farther away from the DC position is, the more likely the quantized coefficient value is 0 or close to 0. Therefore, it may be advantageous in terms of coding performance to arrange the more important coefficients, which have high energy, among the L transform coefficients so that they are coded later in the residual coding order.

[0169] Figure 16 shows three forward scan orders for transform coefficients or transform coefficient blocks applied in the HEVC standard: (a) diagonal scan, (b) horizontal scan, and (c) vertical scan.

[0170] 16 illustrates three forward scan orders for transform coefficients or transform coefficient blocks (4x4 blocks, Coefficient Groups (CGs)) applied in the HEVC standard, and residual coding is performed in the reverse order of the scan orders (a), (b), or (c) (i.e., coding is performed in the order from 16 to 1). Since the three scan orders shown in (a), (b), and (c) are selected depending on the intra-prediction mode, the scan order for the L transform coefficients can also be configured to be determined depending on the intra-prediction mode.

[0171] The value of L has a range of 1<=L<16, and generally, L of the 16 transformation basis vectors can be selected in any manner. However, from the viewpoint of encoding and decoding, it may be advantageous in terms of coding efficiency to select transformation basis vectors that are highly important in terms of signal energy, as in the example presented above.

[0172] Figures 17 and 18 are embodiments to which the present invention is applied, where Figure 17 shows the positions of transform coefficients when a forward diagonal scan is applied when applying 4x4RST to a 4x8 block, and Figure 18 shows an example of merging valid transform coefficients of two 4x4 blocks into one block.

[0173] When the upper left 4x8 block is divided into 4x4 blocks according to the diagonal scan order of (a) and 4x4 RST is applied, if the L value is 8 (i.e., only 8 of the 16 transform coefficients are left), the transform coefficients are positioned as shown in Figure 17, but only half of each 4x4 block can have transform coefficients, and positions marked with X are filled with a value of 0 as a default. Therefore, L transform coefficients are arranged for each 4x4 block according to the scan order presented in (a), and the remaining (16-L) positions of each 4x4 block can be filled with 0s, and residual coding (e.g., residual coding in HEVC) can be applied assuming that these positions are filled with 0s.

[0174] Also, as shown in Figure 18, L transform coefficients arranged in two 4x4 blocks can be configured into one block. In particular, when the value of L is 8, the transform coefficients of the two 4x4 blocks completely fill one 4x4 block, leaving no transform coefficients in the other blocks. Therefore, since residual coding is not required for the 4x4 block that is empty of transform coefficients, in the case of HEVC, a flag (coded_sub_block_flag) indicating whether residual coding is applicable to the block is coded to 0. There are various combination methods for the positions of the transform coefficients of the two 4x4 blocks. For example, the positions can be combined in any order, but the following methods may also be applied.

[0175] 1) The transform coefficients of two 4x4 blocks are alternately combined in the scan order. That is, in FIGS. 8A, 8B, and 8C, the transform coefficients for the upper block are JPEG2025129322000004.jpg556, and the transformation coefficients of the lower block are When the image is JPEG2025129322000005.jpg552, They can be combined one by one, like JPEG2025129322000006.jpg568. JPEG2025129322000007.jpg55 and You can change the order of JPEG2025129322000008.jpg55 JPEG2025129322000009.jpg1190.

[0176] 2) The transform coefficients for the first 4x4 block can be arranged first, followed by the transform coefficients for the second 4x4 block. It can be linked and placed like JPEG2025129322000010.jpg667. You can also change the order, for example, JPEG2025129322000011.jpg669.

[0177] Embodiment 3: A method for coding a non-separable secondary transform (NSST) index for a 4x4 RST

[0178] When the 4x4 RST is applied as shown in Figure 17, values ​​of 0 are filled from the L+1th position to the 16th position according to the transform coefficient scan order for each 4x4 block. Therefore, if there is a non-zero value from the L+1th position to the 16th position in at least one of two 4x4 blocks, it is derived that the 4x4 RST is not applied. If the 4x4 RST has a structure that applies a transform selected from a prepared transform set, such as the joint experiment model (JEM) NSST, an index (hereinafter referred to as an NSST index) indicating which transform to apply is signaled.

[0179] In some decoders, the NSST index can be determined by bitstream parsing, which can be performed after residual coding. In this case, if a non-zero transform coefficient exists between the L+1th position and the 16th position through residual decoding, the decoder can be sure that 4x4 RST will not be applied, and therefore does not parse the NSST index. Therefore, by selectively parsing the NSST index only when necessary, signaling costs can be reduced.

[0180] 17, when a 4x4 RST is applied to a plurality of 4x4 blocks within a specific region (in this case, the same 4x4 RST may be applied to all of them, or different 4x4 RSTs may be applied to each of them), one NSST index specifies the 4x4 RST (the same or different) to be applied to all of the 4x4 blocks. Since one NSST index determines whether the 4x4 RST and 4x4 RST are applied to all of the 4x4 blocks, if a non-zero transform coefficient is found at an impermissible position (L+1-th position to 16-th position) in a 4x4 block as a result of checking whether a non-zero transform coefficient exists at the L+1-th position to 16-th position for all of the 4x4 blocks during a residual decoding process, encoding device 100 may be configured not to code the NSST index.

[0181] The encoding apparatus 100 may signal separate NSST indices for each luminance block and chrominance block, and in the case of a chrominance block, may signal separate NSST indices for the Cb and Cr components, or may use one common NSST index. If one NSST index is used, the NSST index is signaled only once. If one NSST index is shared for the Cb and Cr components, a 4x4 RST indicated by the same NSST index is applied. In this case, the 4x4 RSTs for the Cb and Cr components may be the same, or separate 4x4 RSTs may be set for the Cb and Cr components with the same NSST index. When a shared NSST index is used for the Cb and Cr components, for the above-mentioned conditional signaling, it is checked whether there are any non-zero transform coefficients from the L+1th position to the 16th position for all 4x4 blocks for the Cb and Cr components, and if a non-zero transform coefficient is found in the L+1th position to the 16th position, the signaling for the NSST index may be omitted.

[0182] 18, even when transform coefficients for two 4x4 blocks are merged into one 4x4 block, encoding apparatus 100 may determine whether to signal an NSST index after checking whether a non-zero transform coefficient appears in a position where no valid transform coefficient exists when 4x4 RST is applied. In particular, since the L value is 8 as shown in FIG. 18, if a valid transform coefficient does not exist in one 4x4 block when 4x4 RST is applied (a block indicated by X in (b) of FIG. 18), a flag (coded_sub_block_flag) indicating whether residual coding of the corresponding block is enabled may be checked. If the flag is 1, the NSST index may be set not to be signaled. As described above, the following description will focus on NSST as an example of a non-separable transform, but other known terms (e.g., LFNST) may also be used for a non-separable transform. For example, an NSST set and an NSST index may be substituted for an LFNST set and an LFNST index. In addition, the RST described in this document is an example of a non-separable transform (e.g., LFNST) that uses a non-rectangular transform matrix with a reduced input length and / or a reduced output length in a square non-separable transform matrix applied to at least some regions of a transform block (the upper left 4x4, 8x8 region, or the remaining region in an 8x8 block excluding the lower right 4x4 region), and RST can also be used in place of LFNST.

[0183] Embodiment 4: Optimization method for when coding for 4x4 indexes is performed before residual coding

[0184] When coding for an NSST index is performed before residual coding, whether to apply 4x4 RST is determined in advance, so that residual coding for positions where transform coefficients are filled with 0 may be omitted. Here, whether to apply 4x4 RST may be configured to be determined based on the NSST index value (e.g., if the NSST index is 0, 4x4 RST is not applied), or whether to apply 4x4 RST may be signaled by a separate syntax element (e.g., an NSST flag). For example, if the separate syntax element is an NSST flag, decoding apparatus 200 may determine whether to apply 4x4 RST by first parsing the NSST flag, and then, if the NSST flag value is 1, may omit residual coding (decoding) for positions where no valid transform coefficients exist, as described above.

[0185] In the case of HEVC, when residual coding is performed, coding is first performed at the position of the last non-zero coefficient in a TU. If coding for the NSST index is performed after coding for the position of the last non-zero coefficient, and assuming the application of 4x4 RST, if the position of the last non-zero coefficient is a position where a non-zero coefficient cannot exist, the decoding device 200 may be configured not to code the NSST index and not to apply 4x4 RST. For example, in the case of a position marked X in FIG. 17, when 4x4 RST is applied, a valid transform coefficient is not located (it may be filled with a zero value). Therefore, if the last non-zero coefficient is located in the area marked X, the decoding device 200 may omit coding for the NSST index. If the last non-zero coefficient is not located in the area marked X, the decoding device 200 may perform coding for the NSST index.

[0186] If it is determined whether 4x4RST is applicable by conditionally coding the NSST index after coding the positions of non-zero coefficients, the remaining part where residual coding is performed can be processed in the following two ways.

[0187] 1) When 4x4 RST is not applied, general residual coding is performed, i.e., coding is performed under the assumption that a non-zero transform coefficient can exist anywhere from the position of the last non-zero coefficient to the DC.

[0188] 2) When 4x4 RST is applied, since a transform coefficient does not exist at a specific position or a specific 4x4 block (e.g., position X in FIG. 17) (it is filled with 0 by default), residual coding may be omitted for that position or block. For example, when a position marked X is reached while scanning according to the scan order in FIG. 17, coding of a flag (sig_coeff_flag) indicating whether a non-zero coefficient exists at that position in the HEVC standard may be omitted. When transform coefficients of two blocks are merged into one block as in FIG. 18, coding of a flag (e.g., code_sub_block_flag in the HEVC standard) indicating whether residual coding of the 4x4 block filled with 0 is performed may be omitted, and the corresponding value may be set to 0, and the 4x4 block may be filled with all 0 values ​​without additional coding.

[0189] When coding the NSST index after coding the position of the last non-zero coefficient, if the x-position (Px) and y-position (Py) of the last non-zero coefficient are less than Tx and Ty, respectively, coding of the NSST index is omitted, and 4x4RST is not applied. For example, if Tx = 1, Ty = 1, and the last non-zero coefficient is located at the DC position, NSST index coding is omitted. The method of determining whether to code the NSST index for comparison with the threshold value can be applied differently to the luminance component and the chrominance component. For example, different Tx and Ty may be applied to the luminance component and the chrominance component, respectively, or a threshold value may be applied to the luminance component and not to the chrominance component. Conversely, a threshold value may be applied to the chrominance component and not to the luminance component.

[0190] The above two methods (NSST index coding is omitted if the last non-zero coefficient is located in an area where no valid transform coefficient exists, and NSST index coding is omitted if the X and Y coordinates of the last non-zero coefficient are each smaller than a threshold value) can be applied simultaneously. For example, the threshold value for the position coordinate of the last non-zero coefficient can be checked first, and then it can be checked whether the last non-zero coefficient is located in an area where no valid transform coefficient exists, and the order of the two methods can be changed.

[0191] The method presented in embodiment 4) can also be applied to an 8x8 RST. That is, if the last non-zero coefficient is located in a region that is not the upper left 4x4 region within the upper left 8x8 region, coding for the NSST index can be omitted; otherwise, coding for the NSST index can be performed. Also, if the X and Y coordinate values ​​for the position of the last non-zero coefficient are all less than a certain critical value, coding for the NSST index can be omitted. The two methods can also be applied simultaneously.

[0192] 5. When RST is applied, different NSST index coding and residual coding methods are applied to the luminance component and the chrominance component.

[0193] The schemes described in embodiment 3 and embodiment 4 may be applied differently to the luminance component and the chrominance component. That is, the NSST index coding and residual coding schemes may be applied differently to the luminance component and the chrominance component. For example, the scheme described in embodiment 4 may be applied to the luminance component, and the scheme described in embodiment 3 may be applied to the chrominance component. In addition, the conditional NSST index coding proposed in embodiment 3 or embodiment 4 may be applied to the luminance component, but not to the luminance component, or vice versa (conditional NSST index coding may be applied to the chrominance component, but not to the luminance component).

[0194] Embodiment 6

[0195] In one embodiment of the present invention, a mixed NSST transform set (MNTS) for applying various NSST conditions in the process of applying NSST and a method for constructing the MNTS are provided.

[0196] According to JEM, a 4x4 NSST set contains only 4x4 kernels, and an 8x8 NSST set contains only 8x8 kernels, depending on the preselected size of the subblock. An embodiment of the present invention additionally proposes a method for constructing a mixed NSST set as follows.

[0197] -The size of the NSST kernels available in an NSST set is not fixed, and one or more NSST kernels with variable sizes can be included in the NSST set (for example, one NSST set includes both a 4x4 NSST kernel and an 8x8 NSST kernel).

[0198] The number of NSST kernels available in the NSST set may not be fixed but may be variable (e.g., the first set may contain three kernels, the second set may contain four kernels).

[0199] - The order of NSST kernels may not be fixed and may be defined differently depending on the NSST set (e.g., in the first set, NSST kernels 1, 2, and 3 are mapped to NSST indices 1, 2, and 3, respectively, while in the second set, NSST kernels 3, 2, and 1 are mapped to NSST indices 1, 2, and 3, respectively).

[0200] More specifically, an example of a mixed NSST transformation set construction method is as follows.

[0201] The priority of the NSST kernels available in the -NSST transformation set may be determined by the size of the NSST kernel (e.g., 4x4NSST and 8x8NSST).

[0202] For example, if the block is large, the 8x8 NSST kernel may be more important than the 4x4 NSST kernel, so we assign the 8x8 NSST kernel an NSST index with a lower value.

[0203] The priority of the NSST kernels available in the -NSST transformation set can be determined by the order of the NSST kernels.

[0204] For example, a given 4x4 NSST first kernel may take precedence over a 4x4 NSST second kernel.

[0205] Since the NSST index is transmitted coded, the NSST index can be signaled using fewer bits by assigning higher priority (smaller index) to frequently occurring NSST kernels.

[0206] Tables 1 and 2 below show examples of mixed NSST sets proposed in this embodiment.

[0207] [Table 1]

[0208] [Table 2]

[0209] Embodiment 7

[0210] In one embodiment of the present invention, a method for determining an NSST set is proposed, taking into consideration an intra prediction mode and a block size in the process of determining a secondary transform set.

[0211] The method proposed in this embodiment, in conjunction with the sixth embodiment, configures a set of transformations suited to intra-prediction modes, and configures kernels of various sizes to be applied to blocks.

[0212] FIG. 19 illustrates an example of a method for constructing an NSST set mixed for each intra prediction mode, which is an embodiment to which the present invention is applied.

[0213] 19 is an example of a table in which the method proposed in the second embodiment is applied in conjunction with the sixth embodiment. That is, as shown in FIG. 19, an index ("Mixed Type") is defined for each intra prediction mode, indicating whether to follow an existing NSST set construction method or a different NSST set construction method.

[0214] More specifically, in the case of an intra prediction mode in which the index ("Mixed Type") is defined as "1" in Fig. 19, the NSST set is constructed using a system-defined NSST set construction method, not the JEM NSST set construction method. Here, the system-defined NSST set construction method refers to the mixed NSST set proposed in embodiment 6.

[0215] In another embodiment, the table of Figure 19 illustrates two types of transform set construction methods (JEM-based NSST set construction and the mixed type NSST set construction method proposed in an embodiment of the present invention) based on mixed type information (flag) associated with the intra prediction mode, but there may be more than one mixed type NSST construction method, and here the mixed type information can be expressed as N (N>2) types of various values.

[0216] In another embodiment, it may be determined whether to configure a transform set suitable for the current block as a mixed type by considering both the intra prediction mode and the size of the transform block. For example, if the mode type corresponding to the intra prediction mode is 0, the NSST set setting of JEM is followed; otherwise (Mode Type=1), various mixed type NSST sets may be determined depending on the size of the transform block.

[0217] FIG. 20 shows an example of a method for selecting an NSST set (or kernel) taking into consideration the intra prediction mode and the size of the transform block, in an embodiment to which the present invention is applied.

[0218] Once the transform set is determined, the decoding device 200 can determine the NSST kernel to be used by using the NSST index information.

[0219] Embodiment 8

[0220] In one embodiment of the present invention, when constructing a transform set considering the intra prediction mode and block size during the application of secondary transform, a method for efficiently encoding the NSST index is provided by considering the change in the statistical distribution of the NSST index values transmitted after encoding. Embodiments of the present invention provide a method for selecting a kernel applied using a syntax indicating the kernel size.

[0221] Also, in embodiments of the present invention, since the number of available NSST kernels is different for each transform set, a method of truncated unary binary evolution as shown in Table 3 below is provided according to the maximum NSST index value available for each set for an efficient binary evolution method.

[0222]

Table 3

[0223] Table 3 shows a method of binary evolution of NSST index values. Since the number of available NSST kernels is different for each transform set, the NSST index can be binary-evolved by the maximum NSST index value.

[0224] Embodiment 9: Reduced Transform

[0225] Due to complexity issues in the transform (e.g., large block transform or non-separable transform), reduced transforms applicable to core transforms (e.g., DCT, DST, etc.) and secondary transforms (e.g., NSST) are provided.

[0226] The main idea of the reduced transform is to map an N-dimensional vector to an R-dimensional vector from another space, where R / N (R < N) is the reduction factor. The reduced transform is an R×N matrix as shown in Equation 3 below.

[0227]

Number

[0228] In Equation 1, the R rows of the transformation are the R bases of the new N-dimensional space. Therefore, the reason it is called a reduced transformation is that the number of elements of the vector output by the transformation is smaller than the number of elements of the input vector (R < N). The inverse transform matrix for the reduced transformation is the transpose of the forward transformation. The forward and inverse reduced transformations will be described with reference to FIGS. 13A and 13B.

[0229] FIGS. 21A and 21B show embodiments to which the present invention is applied and show forward and inverse reduced transformations.

[0230] The number of elements of the reduced transformation is RxN, which is smaller than the size of the complete matrix (N×N) by R / N, which means that the required memory is R / N of the complete matrix.

[0231] Also, the number of multiplications required is R×N, which is less than the original N×N by R / N.

[0232] [[ID=2=3]]If X is a vector of N levels, R coefficients are obtained after applying the reduced transformation, which means that only R values need to be transmitted instead of the original N coefficients.

[0233] FIG. 22 shows an example of a decode flowchart using the reduced transformation according to an embodiment of the present invention.

[0234] The proposed downscaled transform (inverse transform at the decoder) can be applied to the coefficients (dequantized coefficients) as shown in Figure 21. A predetermined downscale factor (R or R / N) and a transform kernel for performing the transform may be required. Here, the transform kernel can be determined based on available information such as the block size (width, height), intra prediction mode, and Cidx. If the current coding block is a luma block, CIdx is 0. Otherwise (Cb or Cr block), CIdx is a non-zero value such as 1.

[0235] The operators used in the rest of this document are defined as shown in Tables 4 and 5 below.

[0236] [Table 4]

[0237] [Table 5]

[0238] 23 shows an example of a flowchart for applying a conditional contracted transform according to an embodiment of the present invention. The operations of FIG. 23 are performed by the inverse quantization unit 140 and the inverse transform unit 150 of the decoding device 200.

[0239] In one embodiment, a reduced transform can be used if certain conditions are met. For example, a reduced transform can be applied to blocks larger than a certain size as follows:

[0240] - Width > TH && Height > HT (where TH is a predefined value (e.g., 4))

[0241] or

[0242] - Width * Height > K && MIN(width, height) > TH (K and TH are predefined values)

[0243] That is, as in the above condition, if the width of the current block is greater than the predefined value (TH) and the height of the current block is greater than the predefined value (TH), the reduced conversion can be applied. Alternatively, if the product of the width and height of the current block is greater than the predefined value (K) and the smaller of the width and height of the current block is greater than the predefined value (TH), the reduced conversion can be applied.

[0244] A reduced transformation can be applied to a predetermined group of blocks as follows:

[0245] - Width == TH && Height == TH

[0246] or

[0247] - Width == Height

[0248] That is, if the width and height of the current block are equal to the predetermined value (TH), or if the width and height of the current block are equal (if the current block is a square block), the scaled transformation can be applied.

[0249] If the conditions for using the reduced transform are not met, a regular transform is applied. The regular transform may be a transform that is predefined and available in the video coding system. Examples of regular transforms are as follows:

[0250] -DCT-2, DCT-4, DCT-5, DCT-7, DCT-8

[0251] or

[0252] -DST-1, DST-4, DST-7

[0253] or

[0254] -Non-separable transformation

[0255] or

[0256] -JEM-NSST(HyGT)

[0257] As shown in Figure 23, the reduced transform condition depends on an index (Transform_idx) that indicates which transform (e.g., DCT-4, DST-1) is used or which kernel is applied (when multiple kernels are available). In particular, Transform_idx can be sent twice: once as an index indicating a horizontal transform (Transform_idx_h) and once as an index indicating a vertical transform (Transform_idx_v).

[0258] 23, the decoding device 200 performs inverse quantization on the input bitstream (S2305). Then, the decoding device 200 determines whether to apply a transform (S2310). The decoding device 200 determines whether to apply a transform based on a flag indicating whether to skip a transform.

[0259] If a transform is applied, the decoding device 200 parses a transform index (Transform_idx) indicating the transform to be applied (S2315). Then, the decoding device 200 selects a transform kernel (S2330). For example, the decoding device 200 selects a transform kernel corresponding to the transform index (Transform_idx). Then, the decoding device 200 selects the transform kernel taking into account the block size (width, height), intra prediction mode, and CIdx (luma, chroma).

[0260] The decoding device 200 determines whether the conditions for applying the reduced transform are satisfied (S2320). The conditions for applying the reduced transform include the conditions described above. If the reduced transform is not applicable, the decoding device 200 applies a normal inverse transform (S2325). For example, the decoding device 200 determines an inverse transform matrix from the transform kernel selected in step S2330 and applies the determined inverse transform matrix to the current block including the transform coefficients.

[0261] If a scaled transform is applied, the decoding device 200 applies the scaled inverse transform (S2335). For example, the decoding device 200 determines a scaled inverse transform matrix by taking into account the scale-down factor from the transform kernel selected in step S2330, and applies the scaled inverse transform matrix to the current block including the transform coefficients.

[0262] 24 shows an example of a decoding flowchart for a secondary inverse transform to which a conditional contracted transform is applied according to an embodiment of the present invention. The operation of FIG. 24 is performed by the inverse transform unit 230 of the decoding device 200.

[0263] In one embodiment, the reduced transform can be applied to the secondary transform as in Figure 24. Once the NSST index is parsed, the reduced inverse transform can be applied.

[0264] 24, the decoding apparatus 200 performs inverse quantization (S2405). The decoding apparatus 200 determines whether to apply NSST to the transform coefficients generated by the inverse quantization (S2410). That is, the decoding apparatus 200 determines whether parsing of the NSST index (NSST_idx) is necessary depending on whether NSST is applied.

[0265] If NSST is applied, the decoding device 200 parses the NSST index (S2415) and determines whether the NSST index is greater than 0 (S2420). The NSST index is restored by the entropy decoding unit 210 using a technique such as CABAC. If the NSST index is 0, the decoding device 200 omits the secondary inverse transform and applies the core inverse transform or the primary inverse transform (S2445).

[0266] If NSST is applied, the decoding device 200 selects a transform kernel for secondary inverse transform (S2435). For example, the decoding device 200 selects a transform kernel corresponding to an NSST index (NSST_idx). The decoding device 200 selects the transform kernel taking into account the block size (width, height), intra prediction mode, and CIdx (luma, chroma).

[0267] If the NSST index is greater than 0, the decoding device 200 determines whether the conditions for applying a reduced transform are satisfied (S2425). The conditions for applying a reduced transform include the conditions described above. If a reduced transform is not applied, the decoding device 200 applies a normal secondary inverse transform (S2430). For example, the decoding device 200 determines a secondary inverse transform matrix from the transform kernel selected in step S2435 and applies the determined secondary inverse transform matrix to the current block including transform coefficients.

[0268] If a reduced transform is applied, the decoding device 200 applies a reduced secondary inverse transform (S2440). For example, the decoding device 200 may determine a reduced inverse transform matrix by considering a reduction factor from the transform kernel selected in step S2335, and apply the reduced inverse transform matrix to the current block including the transform coefficients. Thereafter, the decoding device 200 applies a core inverse transform or a linear inverse transform (S2445).

[0269] Embodiment 10: Reduced Transform as a Secondary Transform with Different Block Sizes

[0270] 25A, 25B, 26A, and 26B show examples of reduced transforms and reduced inverse transforms according to embodiments of the present invention.

[0271] In one embodiment of the present invention, a reduced transform in a video codec for different block sizes such as 4x4, 8x8, 16x16, etc. can be used as the secondary transform and the secondary inverse transform. As an example for an 8x8 block size and a reduction factor R=16, the secondary transform and the secondary inverse transform can be set as shown in Figures 25A and 25B.

[0272] The pseudocode for the reduced transform and the reduced inverse transform is set out in FIG.

[0273] Embodiment 11: Reduced Transform as a Secondary Transform with Non-Rectangular Shape

[0274] FIG. 27 shows an example of a region to which a reduced quadratic transform is applied according to an embodiment of the present invention.

[0275] As mentioned above, due to the complexity issues in quadratic transformations, quadratic transformations can be applied to 4x4 and 8x8 corners. Reduced transformations can also be applied to non-rectangular shapes.

[0276] As mentioned above, due to the complexity issues in quadratic transformations, quadratic transformations can be applied to 4x4 and 8x8 corners. Reduced transformations can also be applied to non-rectangular shapes.

[0277] In another example, when RST is applied to an 8x8 block, the non-separable transform (RST) can be applied only to the remaining three 4x4 blocks (48 transform coefficients in total) on the top-left, top-right, and bottom-left sides, excluding the bottom-right 4x4 block.

[0278] Embodiment 12: Reduction Factor

[0279] FIG. 28 illustrates a reduced transformation with a reduction factor according to an embodiment of the present invention.

[0280] Changing the reduction factor can change the memory and multiplication complexity. As mentioned above, changing the reduction factor reduces the memory and multiplication complexity by a factor R / N. For example, for an 8x8 NSST, if R=16, the memory and multiplication complexity are reduced by 1 / 4.

[0281] Embodiment 13: High Level Syntax

[0282] The syntax elements shown in Table 6 below can be used to process RST in video coding. Semantics related to the reduced transform can be present in the sequence parameter set (SPS) or slice header.

[0283] Reduced_transform_enabled_flag equal to 1 indicates that the reduced transform is enabled and applied. Reduced_transform_enabled_flag equal to 0 indicates that the reduced transform is not enabled. If Reduced_transform_enabled_flag is not present, it is inferred to be equal to 0 (Reduced_transform_enabled_flag equals to 1 specifies that reduced transform is enabled and applied. Reduced_transform_enabled_flag equal to 0 specifies that reduced transform is not enabled. When Reduced_transform_enabled_flag is not present, it is inferred to be equal to 0).

[0284] Reduced_transform_factor specifies that the number of reduced dimensions to keep for reduced transform. When Reduced_transform_factor is not present, it is inferred to be equal to R.

[0285] min_reduced_transform_size specifies that the minimum transform size to apply reduced transform. When min_reduced_transform_size is not present, it is inferred to be equal to 0.

[0286] max_reduced_transform_size indicates the maximum transform size to apply the reduced transform to. If max_reduced_transform_size is not present, it is inferred to be 0.

[0287] reduced_transform_size specifies the number of reduced dimensions to keep for the reduced transform. When Reduced_transform_factor is not present, it is inferred to be equal to 0.

[0288] [Table 6]

[0289] Embodiment 14: Conditional application of 4x4RST for worst case handling

[0290] The non-separable quadratic transform (4x4 NSST) that can be applied to a 4x4 block is a 16x16 transform. 4x4 NSST is applied secondarily to a block that has already undergone a primary transform such as DCT-2, DST-7, or DCT-8. If the size of the block to which the primary transform has been applied is NxM, the following methods can be considered when applying 4x4 NSST to an NxM block:

[0291] 1) The conditions for applying 4x4 NSST to an NxM region are as follows: a) and b).

[0292] a) N >= 4

[0293] b) M >= 4

[0294] 2) 4x4 NSST is not applied to the entire NxM region, but only to a portion of the region. For example, 4x4 NSST can be applied only to the KxJ region in the upper left. The conditions for this are as follows: a) and b).

[0295] a) K >= 4

[0296] b) J >= 4

[0297] 3) After dividing the region to which the secondary transformation is applied into 4x4 blocks, 4x4 NSST can be applied to each divided block.

[0298] The computational complexity of 4x4 NSST is a very important consideration for both the encoder and decoder, so we will analyze it in detail. In particular, we will analyze the computational complexity of 4x4 NSST based on the number of multiplications. In the case of forward NSST, a 16x16 quadratic transform is composed of 16 row-wise transform basis vectors. The transform coefficients for each transform basis vector are obtained by taking the inner product of a 16x1 vector with each transform basis vector. The process of obtaining all transform coefficients for the 16 transform basis vectors is equivalent to multiplying a 16x16 non-separable transform matrix by the input 16x1 vector. Therefore, the total number of multiplications required for 4x4 forward NSST is 256.

[0299] At the decoder, applying an inverse 16x16 non-separable transform to the 16x1 transform coefficients (ignoring the effects of quantization, integerization, etc.) recovers the coefficients of the original 4x4 linear transform block. In other words, multiplying the 16x1 transform coefficient vector by the inverse 16x16 non-separable transform matrix obtains data in the form of a 16x1 vector, and arranging the data in the row-major or column-major order originally applied recovers the 4x4 block signal (linear transform coefficients). Therefore, the total number of multiplications required for a 4x4 inverse NSST is 256.

[0300] As mentioned above, when 4x4 NSST is applied, the number of multiplications required per sample is 16. This is the number obtained by dividing the total number of multiplications (256) obtained in the inner product process of the 16x1 vector and each transformation basis vector, which is the process of executing 4x4 NSST, by the total number of samples (16). The number of multiplications required is the same for both forward 4x4 NSST and backward 4x4 NSST, which is 16.

[0301] If we have an 8x8 block, the number of multiplications per sample required when applying 4x4 NSST is determined as follows depending on the region to which 4x4 NSST is applied:

[0302] 1. 4x4 NSST applied only to the upper left 4x4 region: 256 (number of multiplications required for the 4x4 NSST process) / 64 (total number of samples in the 8x8 block) = 4 multiplications per sample

[0303] 2. 4x4 NSST applied to the upper left 4x4 region and the upper right 4x4 region: 512 (multiplications required for the two 4x4 NSST processes) / 64 (total number of samples in the 8x8 block) = 8 multiplications per sample

[0304] 3. When 4x4 NSST is applied to all 4x4 regions of an 8x8 block: 1024 (number of multiplications required for four 4x4 NSST processes) / 64 (total number of samples in the 8x8 block) = 16 multiplications / sample

[0305] As mentioned above, if the size of the block is large, the extent to which the 4x4 NSST is applied can be reduced to reduce the number of worst-case multiplications required for each sample.

[0306] Therefore, when using 4x4 NSST, the worst case occurs when the TU size is 4x4. In this case, the method to reduce the worst case complexity is as follows.

[0307] Method 1. Do not apply 4x4 NSST to small TUs (i.e., 4x4 TUs).

[0308] Method 2. For 4x4 blocks (4x4TU), apply 4x4RST instead of 4x4NSST.

[0309] In Method 1, experiments showed that not applying 4x4 NSST resulted in a significant degradation in coding performance. In Method 2, due to the statistical characteristics of the elements that make up the 16x1 transform coefficient vector, it was found that applying inverse transform to some of the front transform coefficients could restore a signal that was fairly close to the original signal, even without using all transform coefficients, and that most coding performance could be maintained.

[0310] Specifically, in the case of a 4x4 RST, if the inverse (or forward) 16x16 non-separable transform is composed of 16 column basis vectors, only L column basis vectors are left to form a 16xL matrix. By leaving only the most important L transform coefficients among the transform coefficients, multiplying the 16xL matrix by an Lx1 vector can restore a 16x1 vector with a small error compared to the original 16x1 vector data.

[0311] As a result, since only L coefficients are involved in data recovery, an L×1 transform coefficient vector is required to obtain the transform coefficients, rather than a 16×1 transform coefficient vector. That is, an L×16 transform matrix is ​​constructed by selecting L row-direction transform vectors in a forward 16×16 non-separable transform matrix, and L transform coefficients are obtained by multiplying the L×16 transform matrix by a 16×1 input vector.

[0312] The value of L has a range of 1<=L<16, and generally, L can be selected from 16 transformation basis vectors in any way, but as mentioned above, it may be advantageous in terms of coding efficiency to select transformation basis vectors with high signal energy significance from the perspective of encoding and decoding. The worst-case number of multiplications per sample in a 4x4 block due to transformation of the value of L is as shown in Table 7 below.

[0313] [Table 7]

[0314] As mentioned above, to reduce worst-case multiplication complexity, 4x4 NSST and 4x4 RST can be used in combination as shown in Table 8 below (however, the following example explains the conditions for applying 4x4 NSST and 4x4 RST under the conditions for applying 4x4 NSST (i.e., when the width and height of the current block are all greater than or equal to 4)).

[0315] As mentioned above, 4x4 NSST for a 4x4 block is a square (16x16) transform matrix that inputs 16 pieces of data and outputs 16 pieces of data, while 4x4 RST refers to a non-square (8x16) transform matrix that inputs 16 pieces of data from the encoder side and outputs R pieces of data (e.g., 8 pieces) that are less than 16. 4x4 RST refers to a non-square (16x8) transform matrix that inputs R pieces of data (e.g., 8 pieces) that are less than 16 and outputs 16 pieces of data from the decoder side.

[0316] [Table 8]

[0317] Referring to Table 8, if the width and height of the current block are each 4, a 4x4 RST based on an 8x16 matrix is ​​applied to the current block; otherwise (if either the width or height of the current block is not 4), a 4x4 NSST is applied to the 4x4 region in the upper left corner of the current block. More specifically, if the size of the current block is 4x4, a non-separable transform with an input length of 16 and an output length of 8 can be applied. In the case of an inverse non-separable transform, a non-separable transform with an input length of 8 and an output length of 16 can be applied.

[0318] As mentioned above, to reduce worst-case multiplication complexity, 4x4 NSST and 4x4 RST can be used in combination as shown in Table 9 below (however, the following example explains the conditions for applying 4x4 NSST and 4x4 RST under the conditions for applying 4x4 NSST (i.e., when the width and height of the current block are all greater than or equal to 4)).

[0319] [Table 9]

[0320] Referring to Table 9, if the width and height of the current block are each 4, a 4x4 RST based on an 8x16 matrix is ​​applied; if the product of the width and height of the current block is less than a threshold value (TH), a 4x4 NSST is applied to the 4x4 region in the upper left corner of the current block; if the width of the current block is greater than or equal to the height, a 4x4 NSST is applied to the 4x4 region in the upper left corner of the current block and the 4x4 region located to the right of the 4x4 region in the upper left corner; and in the remaining cases (if less than the height of the current block), a 4x4 NSST is applied to the 4x4 region in the upper left corner of the current block and the 4x4 region located below the 4x4 region in the upper left corner of the current block.

[0321] In conclusion, for the reduction of the worst-case computational complexity of multiplication, 4x4 RST can be applied instead of 4x4 NSST for 4x4 blocks (e.g., 8x16 matrix).

[0322] Embodiment 15: Conditional application of 8x8RST for worst case handling

[0323] The non-separable secondary transform (8x8NSST) that can be applied to an 8x8 block is a 64x64 transform. The 8x8NSST is applied secondarily to a block that has already undergone a primary transform such as DCT-2, DST-7, or DCT-8. If the size of the block to which the primary transform has been applied is NxM, the following methods can be considered when applying the 8x8NSST to an NxM block:

[0324] 1) The conditions for applying 8x8 NSST to an NxM region are as follows: c) and d).

[0325] c) N >= 8

[0326] d) M >= 8

[0327] 2) The 8x8 NSST is not necessarily applied to the entire NxM region, but may be applied only to some regions. For example, the 8x8 NSST is applied only to the KxJ region in the upper left. The conditions for this are as follows: c) and d)

[0328] c) K >= 8

[0329] d) J >= 8

[0330] 3) After dividing the region to which the secondary transformation is applied into 8x8 blocks, 8x8 NSST can be applied to each divided block.

[0331] The computational complexity of 8x8 NSST is a very important consideration for both the encoder and decoder, so we will analyze it in detail. In particular, we will analyze the computational complexity of 8x8 NSST based on the number of multiplications. In the case of forward NSST, a 64x64 non-separable quadratic transform is composed of 64 row-wise transform basis vectors. The transform coefficients for each transform basis vector are obtained by taking the inner product of a 64x1 vector with each transform basis vector. The process of obtaining all transform coefficients for the 64 transform basis vectors is equivalent to multiplying the input 64x1 vector by a 64x64 non-separable transform matrix. Therefore, the total number of multiplications required for 8x8 forward NSST is 4096.

[0332] In the decoder, applying an inverse 64x64 non-separable transform to the 64x1 transform coefficients (ignoring the effects of quantization, integerization, etc.) recovers the coefficients of the original 8x8 linear transform block. In other words, multiplying the 64x1 transform coefficient vector by the inverse 64x64 non-separable transform matrix obtains data in the form of a 64x1 vector, and arranging the data in the row-major or column-major order originally applied recovers the 8x8 block signal (linear transform coefficients). Therefore, the total number of multiplications required for an 8x8 inverse NSST is 4096.

[0333] As mentioned above, when 8x8 NSST is applied, the number of multiplications required per sample is 64. This is the number obtained by dividing the total number of multiplications (4096) obtained in the inner product process of the 64x1 vector and each transformation basis vector, which is the process of executing 8x8 NSST, by the total number of samples (64). The number of multiplications required is the same for both forward 8x8 NSST and backward 8x8 NSST, which is 64.

[0334] If the block is 16x16, the number of multiplications required per sample when applying 8x8 NSST is determined by the region to which 8x8 NSST is applied as follows:

[0335] 1. 8x8 NSST applied only to the upper left 8x8 region: 4096 (number of multiplications required for the 8x8 NSST process) / 256 (total number of samples in the 16x16 block) = 16 multiplications per sample

[0336] 2. 8x8 NSST applied to the upper left 8x8 region and the upper right 8x8 region: 8192 (multiplications required for the two 8x8 NSST processes) / 256 (total number of samples in the 16x16 block) = 32 multiplications per sample

[0337] 3. When 8x8 NSST is applied to all 8x8 regions of a 16x16 block: 16384 (number of multiplications required for four 8x8 NSST processes) / 256 (total number of samples in the 16x16 block) = 64 multiplications / sample

[0338] As mentioned above, if the block size is large, the extent to which the 8x8 NSST is applied can be reduced to reduce the number of worst-case multiplications required per sample.

[0339] When 8x8 NSST is applied, since an 8x8 block is the smallest TU to which 8x8 NSST can be applied, the worst case occurs when the TU size is 8x8 in terms of the number of multiplications required per sample. In this case, the method for reducing the worst case complexity is as follows.

[0340] Method 1. Do not apply 8x8 NSST to small TUs (i.e., 8x8 TUs).

[0341] Method 2. For 8x8 blocks (8x8TU), apply 8x8RST instead of 8x8NSST.

[0342] In Method 1, experiments showed that not applying 8x8 NSST resulted in a significant degradation in coding performance. In Method 2, due to the statistical characteristics of the elements that make up the 64x1 transform coefficient vector, it was found that applying inverse transform to some of the leading transform coefficients could restore a signal that was fairly close to the original, without using all of the transform coefficients, and that most of the coding performance could be maintained.

[0343] Specifically, in the case of an 8x8 RST, if the inverse (or forward) 64x64 non-separable transform is composed of 16 column basis vectors, only L column basis vectors are left to form a 64xL matrix. By leaving only the most important L transform coefficients among the transform coefficients, multiplying the 64xL matrix by an Lx1 vector can restore a 64x1 vector with a small error compared to the original 64x1 vector data.

[0344] Also, as described in embodiment 11, RST may not be applied to all 64 transform coefficients contained in an 8x8 block, but may be applied to a portion of the block (e.g., the remaining area of ​​the 8x8 block excluding the 4x4 area in the lower right corner).

[0345] As a result, since only L coefficients are involved in data recovery, an L×1 transform coefficient vector is required to obtain the transform coefficients, rather than a 64×1 transform coefficient vector. That is, an L×64 transform matrix is ​​constructed by selecting L row-direction transform vectors in a forward 64×64 non-separable transform matrix, and L transform coefficients are obtained by multiplying the L×64 transform matrix by a 64×1 input vector.

[0346] The value of L has a range of 1<=L<64, and generally, L can be selected from 64 transform basis vectors in any way, but as mentioned above, it may be advantageous in terms of coding efficiency to select transform basis vectors with high signal energy significance from the perspective of encoding and decoding. The worst-case multiplication number per sample in an 8x8 block depending on the value of L is as shown in Table 10 below.

[0347] [Table 10]

[0348] As mentioned above, to reduce worst-case multiplication complexity, 8x8 RSTs with different L values ​​can be used in combination as shown in Table 11 below (however, the following example explains the conditions for applying 8x8 RST under the conditions for applying 8x8 NSST (i.e., when the width and height of the current block are all greater than or equal to 8)).

[0349] [Table 11]

[0350] Referring to Table 11, if the width and height of the current block are each 8, an 8x8RST based on an 8x64 matrix is ​​applied to the current block; otherwise (if either the width or height of the current block is not 8), an 8x8RST based on a 16x64 matrix is ​​applied to the current block. More specifically, if the size of the current block is 8x8, a non-separable transform with an input length of 64 and an output length of 8 is applied; otherwise, a non-separable transform with an input length of 64 and an output length of 16 is applied. For the inverse non-separable transform, if the current block is 8x8, a non-separable transform with an input length of 8 and an output length of 64 is applied; otherwise, a non-separable transform with an input length of 16 and an output length of 64 is applied.

[0351] Also, as described in embodiment 11, RST can be applied only to a portion of an 8x8 block rather than to the entire block, so for example, when RST is applied to the remaining area excluding the 4x4 area in the lower right corner of an 8x8 block, an 8x8 RST based on an 8x48 or 16x18 matrix can be applied. That is, if the width and height of the current block are each 8, an 8x8 RST based on an 8x48 matrix is ​​applied, and if not (if the width or height of the current block is not 8), an 8x8 RST based on a 16x48 matrix is ​​applied.

[0352] For forward non-separable transforms, if the current block is 8x8, then a non-separable transform with input length 48 and output length 8 is applied, otherwise a non-separable transform with input length 48 and output length 16 is applied.

[0353] For the inverse non-separable transform, if the current block is 8x8, a non-separable transform with input length 8 and output length 48 is applied, otherwise a non-separable transform with input length 16 and output length 48 is applied.

[0354] In conclusion, when RST is applied to a block larger than 8x8, based on the encoder side, if the block height and width are each 8, a non-separable transform matrix (8x48 or 8x64 matrix) with an input length less than or equal to 64 (e.g., 48 or 64) and an output length less than 64 (e.g., 8) is applied; if the block height or width is not 8, a non-separable transform matrix (16x48 or 16x64 matrix) with an input length less than or equal to 64 (e.g., 48 or 64) and an output length less than 64 (e.g., 16) is applied.

[0355] Also, when RST is applied to a block larger than 8x8 on the decoder side, if the height and width of the block are both 8, a non-separable transform matrix (48x8 or 64x8 matrix) with an input length less than 64 (e.g., 8) and an output length less than or equal to 64 (e.g., 48 or 64) is applied; if the height or width of the block is not 8, a non-separable transform matrix (48x16 or 64x16 matrix) with an input length less than 64 (e.g., 16) and an output length less than or equal to 64 (e.g., 48 or 64) is applied.

[0356] Table 12 shows examples of various 8x8 RST applications under the condition for applying 8x8 NSST (ie, when the width and height of the current block are greater than or equal to 8).

[0357] [Table 12]

[0358] Referring to Table 12, if the width and height of the current block are each 8, an 8x8 RST based on an 8x64 matrix (or an 8x48 matrix) is applied; if the product of the width and height of the current block is less than the critical value (TH), an 8x8 RST based on a 16x64 matrix (or a 16x48 matrix) is applied to the 8x8 region at the top left of the current block; and in the remaining cases (if the width or height of the current block is not 8 and the product of the width and height of the current block is greater than or equal to the critical value), an 8x8 RST based on a 32x64 matrix (or a 32x48 matrix) is applied to the 8x8 region at the top left of the current block.

[0359] FIG. 29 shows an example of an encoding flowchart for performing conversion, according to an embodiment to which the present invention is applied.

[0360] The encoding apparatus 100 performs a primary transform on the residual block (S2910). The primary transform may be referred to as a core transform. In an embodiment, the encoding apparatus 100 performs the primary transform using the MTS. The encoding apparatus 100 also transmits an MTS index indicating a specific MTS from among the MTS candidates to the decoding apparatus 200. Here, the MTS candidate is configured based on the intra prediction mode of the current block.

[0361] The encoding apparatus 100 determines whether to apply a secondary transform (S2920). As an example, the encoding apparatus 100 determines whether to apply a secondary transform based on the primary-transformed residual transform coefficients. For example, the secondary transform may be NSST or RST.

[0362] The encoding apparatus 100 determines a secondary transform (S2930). At this time, the encoding apparatus 100 determines the secondary transform based on the NSST (or RST) transform set specified by the intra prediction mode.

[0363] Also, as an example, the encoding apparatus 100 determines the area to which the secondary transform is applied based on the size of the current block prior to step S2930.

[0364] The encoding apparatus 100 performs secondary transformation using the secondary transformation determined in step S2930 (S2940).

[0365] FIG. 30 shows an example of a decoding flowchart for performing conversion in an embodiment to which the present invention is applied.

[0366] The decoding device 200 determines whether to apply a secondary inverse transform (S3010). For example, the secondary inverse transform may be NSST or RST. As an example, the decoding device 200 determines whether to apply a secondary inverse transform based on a secondary transform flag received from the encoding device 100.

[0367] The decoding device 200 determines a secondary inverse transform (S3020). Here, the decoding device 200 may determine a secondary inverse transform to be applied to the current block based on the NSST (or RST) transform set specified by the above-described intra prediction mode.

[0368] Also, for example, the decoding apparatus 200 determines the area to which the secondary inverse transform is applied based on the size of the current block prior to step S3020.

[0369] The decoding device 200 performs a secondary inverse transform on the dequantized residual block using the secondary inverse transform determined in step S3020 (S3030).

[0370] The decoding apparatus 200 performs a primary inverse transform on the secondary inverse transformed residual block (S3040). The primary inverse transform may be referred to as a core inverse transform. In one embodiment, the decoding apparatus 200 performs the primary inverse transform using the MTS. As another example, the decoding apparatus 200 may determine whether MTS is applied to the current block prior to step S3040. In this case, the decoding flowchart of FIG. 30 may further include a step of determining whether MTS is applied.

[0371] For example, when MTS is applied to the current block (i.e., cu_mts_flag=1), the decoding apparatus 200 configures MTS candidates based on the intra prediction mode of the current block. In this case, the decoding flowchart of Figure 30 may further include a step of configuring MTS candidates. The decoding apparatus 200 may then determine a linear inverse transform to be applied to the current block using mts_idx, which indicates a specific MTS among the configured MTS candidates.

[0372] FIG. 31 shows an example of a detailed block diagram of the transform unit 120 in the encoding device 100, which is an embodiment to which the present invention is applied.

[0373] The encoding device 100 to which the embodiment of the present invention is applied includes a primary conversion unit 3110 , a secondary conversion application decision unit 3120 , a secondary conversion decision unit 3130 , and a secondary conversion unit 3140 .

[0374] The primary transform unit 3110 may perform a primary transform on the residual block. The primary transform may be referred to as a core transform. In an embodiment, the primary transform unit 3110 performs the primary transform using the MTS. The primary transform unit 3110 also transmits an MTS index indicating a specific MTS among the MTS candidates to the decoding apparatus 200. Here, the MTS candidate is configured based on the intra prediction mode of the current block.

[0375] The secondary transform application determining unit 3120 may determine whether to apply a secondary transform. For example, the secondary transform application determining unit 3120 may determine whether to apply a secondary transform based on transform coefficients of the primary transformed residual block. For example, the secondary transform may be NSST or RST.

[0376] The secondary transform determination unit 3130 determines the secondary transform based on the NSST (or RST) transform set specified by the intra prediction mode, as described above.

[0377] Also, for example, the secondary transformation determination unit 3130 may determine the area to which the secondary transformation is applied based on the size of the current block.

[0378] The secondary conversion unit 3140 can perform the secondary conversion using the determined secondary conversion.

[0379] FIG. 32 shows an example of a detailed block diagram of the inverse transform unit 230 in the decoding device 200, which is an embodiment to which the present invention is applied.

[0380] The decoding device 200 to which the present invention is applied includes a secondary inverse transform application determining unit 3210, a secondary inverse transform determining unit 3220, a secondary inverse transform unit 3230, and a primary inverse transform unit 3240.

[0381] The secondary inverse transform application determining unit 3210 determines whether to apply a secondary inverse transform. For example, the secondary inverse transform may be NSST or RST. As an example, the secondary inverse transform application determining unit 3210 determines whether to apply a secondary inverse transform based on a secondary transform flag received from the encoding apparatus 100. As another example, the secondary inverse transform application determining unit 3210 may determine whether to apply a secondary inverse transform based on transform coefficients of a residual block.

[0382] The secondary inverse transform decision unit 3220 decides a secondary inverse transform to be applied to the current block based on an NSST (or RST) transform set specified by the intra prediction mode.

[0383] Also, for example, the secondary inverse transform determination unit 3220 may determine the area to which the secondary inverse transform is applied based on the size of the current block.

[0384] Also, as an example, the secondary inverse transform unit 3230 may perform a secondary inverse transform on the inverse-quantized residual block using the determined secondary inverse transform.

[0385] The primary inverse transform unit 3240 performs a primary inverse transform on the secondary inverse transformed residual block. As an embodiment, the primary inverse transform unit 3240 performs the primary transform using the MTS. As an example, the primary inverse transform unit 3240 may determine whether MTS is applied to the current block.

[0386] For example, when MTS is applied to the current block (i.e., cu_mts_flag=1), the primary inverse transform unit 3240 configures MTS candidates based on the intra prediction mode of the current block, and determines the primary transform to be applied to the current block using mts_idx, which indicates a specific MTS among the configured MTS candidates.

[0387] 33 shows an example of a decoding flowchart in which a transform according to an embodiment of the present invention is applied. The operation of FIG.

[0388] In step S3305, decoding apparatus 200 determines the input length and output length of the non-separable transform based on the height and width of the current block. Here, if the height and width of the current block are each 8, the input length of the non-separable transform is determined to be 8, and the output length is determined to be a value greater than the input length and less than or equal to 64 (e.g., 48 or 64). For example, if the encoder applies a non-separable transform to all of the transform coefficients of an 8x8 block, the output length is determined to be 64, and if the encoder applies a non-separable transform to only a portion of the transform coefficients of the 8x8 block (e.g., a portion of the 8x8 block excluding the 4x4 area at the bottom right), the output length is determined to be 48.

[0389] In step S3310, decoding device 200 determines a non-separable transform matrix corresponding to the input length and output length of the non-separable transform. For example, if the input length of the non-separable transform is 8 and the output length is 48 or 64 (for example, if the size of the current block is 4×4), a 48×8 or 64×8 matrix derived from the transform kernel may be determined as the non-separable transform. If the input length of the non-separable transform is 16 and the output length is 48 or 64 (for example, if the current block is smaller than 8×8 and not 4×4), a 48×16 or 64×16 transform kernel may be determined as the non-separable transform.

[0390] According to an embodiment of the present invention, the decoding device 200 determines a non-separable transform set index (e.g., an NSST index) based on the intra prediction mode of the current block, determines a non-separable transform kernel corresponding to the non-separable transform index within the non-separable transform set included in the non-separable transform set index, and can determine a non-separable transform matrix from the non-separable transform kernel based on the input length and output length determined in step S3305.

[0391] In operation S3315, decoding apparatus 200 applies the non-separable transform matrix determined for the current block to coefficients of the determined input length (8 or 16) of the current block. For example, if the input length of the non-separable transform is 8 and the output length is 48 or 64, a 48×8 or 64×8 matrix derived from the transform kernel is applied to the eight coefficients included in the current block. If the input length of the non-separable transform is 16 and the output length is 48 or 64, a 48×16 or 64×16 matrix derived from the transform kernel is applied to the 16 coefficients in the upper left 4×4 region of the current block. Here, the coefficients to which the non-separable transform is applied are the coefficients from the DC position of the current block to a position corresponding to the input length (e.g., 8 or 16) according to a determined scan order (e.g., (a), (b), or (c) of FIG. 16).

[0392] In addition, in the case where the height and width of the current block are not 8, if the product of the width and height of the current block is less than a threshold value, the decoding device 200 applies a non-separable transform matrix (48×16 or 64×16 matrix) that inputs 16 coefficients in the 4×4 area on the upper left side of the current block and outputs transformed coefficients for the output length (e.g., 48 or 64); if the product of the width and height of the current block is greater than or equal to the threshold value, the decoding device 200 applies a non-separable transform matrix (48×32 or 64×32 matrix) that inputs 32 coefficients in the current block and outputs transformed coefficients for the output length (e.g., 48 or 64).

[0393] When the output length is 64, 64 pieces of transformed data (transformed coefficients) are arranged in an 8x8 block by applying a non-separable transformation matrix, and when the output length is 48, 48 pieces of transformed data (transformed coefficients) are arranged in the remaining area of ​​the 8x8 block excluding the 4x4 area in the bottom right corner by applying a non-separable transformation matrix.

[0394] Fig. 34 shows an example of a block diagram of an apparatus for processing a video signal, which is an embodiment to which the present invention is applied. An image processing apparatus 3400 in Fig. 34 may correspond to the encoding apparatus 100 in Fig. 1 or the decoding apparatus 200 in Fig. 2.

[0395] The image processing device 3400 for processing image signals includes a memory 3420 for storing image signals, and a processor 3410 coupled to the memory for processing the image signals.

[0396] The processor 3410 according to an embodiment of the present invention includes at least one processing circuit for processing an image signal, and can process the image signal by executing instructions for encoding or decoding the image signal. That is, the processor 3410 can encode original image data or decode an encoded image signal by executing the encoding or decoding method described above.

[0397] FIG. 35 shows an example of an image coding system to which the present invention is applied.

[0398] An image coding system includes a source device and a receiving device. The source device transmits encoded video / image information or data to the receiving device in the form of a file or streaming data via a digital storage medium or a network.

[0399] A source device includes a video source, an encoding device, and a transmitter. A receiving device includes a receiver, a decoding device, and a renderer. An encoding device may be called a video / image encoding device, and a decoding device may be called a video / image decoding device. A transmitter may be included in an encoding device. A receiver may be included in a decoding device. A renderer may include a display unit, which may be a separate device or an external component.

[0400] A video source acquires video / images through a video / image capture, synthesis, or generation process. A video source includes a video / image capture device and / or a video / image generation device. A video / image capture device includes, for example, one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device includes, for example, a computer, a tablet, a smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image can be generated by a computer, in which case the process of video / image capture can be replaced by a process of generating the associated data.

[0401] An encoder encodes the input video / image. The encoder may perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) is output in the form of a bitstream.

[0402] The transmitter transmits the encoded video / image information or data output in the form of a bitstream to the receiver of the receiving device via a digital storage medium or network in the form of a file or streaming. Digital storage media include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter includes elements for generating a media file in a predetermined file format and elements for transmission via a broadcast / communication network. The receiver extracts the bitstream and transmits it to a decoding device.

[0403] The decoding device decodes the video / image by performing a series of steps such as inverse quantization, inverse transformation, and prediction, which correspond to the operations of the encoding device.

[0404] The renderer renders the decoded video / image, which is then displayed via the display unit.

[0405] FIG. 36 is a structural diagram of a content streaming system, which is an embodiment to which the present invention is applied.

[0406] The content streaming system to which the present invention is applied includes an encoding server, a streaming server, a web server, a media repository, a user device, and a multimedia input device.

[0407] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, or camcorder into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, or camcorder generates a bitstream directly, the encoding server may be omitted.

[0408] The bitstream is generated by an encoding method or a bitstream generating method to which the present invention is applied, and the streaming server can temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0409] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as a medium for informing the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. Here, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0410] The streaming server receives content from a media repository and / or an encoding server. For example, when receiving content from an encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0411] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, head mounted displays (HMDs)), digital TVs, desktop computers, digital signage, and the like.

[0412] Each server in the content streaming system can be operated as a distributed server, in which case data received by each server can be processed in a distributed manner.

[0413] Furthermore, a processing method according to the present invention can be produced in the form of a computer-executable program and stored in a computer-readable storage medium. Multimedia data having a data structure according to the present invention can also be stored in a computer-readable storage medium. The computer-readable storage medium includes any type of storage device or distributed storage device in which computer-readable data is stored. The computer-readable storage medium can include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable storage medium can also include media realized in the form of a carrier wave (e.g., transmission over the Internet). The bitstream generated by the encoding method can be stored in a computer-readable storage medium or transmitted via a wired or wireless communication network.

[0414] Furthermore, embodiments of the present invention may be realized as a computer program product with program code, which may be executed on a computer according to embodiments of the present invention. The program code may be stored on a computer-readable carrier.

[0415] As described above, the embodiments described in the present invention may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip.

[0416] In addition, the decoder and encoder to which the present invention is applied may be included in real-time communication devices such as multimedia broadcasting transmitting / receiving devices, mobile communication terminals, home cinema video devices, digital cinema video devices, surveillance cameras, video interaction devices, video communications, etc., mobile streaming devices, storage media, camcorders, video-on-demand (VoD) service providing devices, over-the-top (OTT) video devices, internet streaming service providing devices, three-dimensional (3D) video devices, video phones, video devices, medical video devices, etc., and are used to process video signals or data signals. For example, over-the-top (OTT) video devices may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, digital video recorders (DVRs), etc.

[0417] Furthermore, a processing method according to the present invention can be produced in the form of a computer-executable program and stored in a computer-readable storage medium. Multimedia data having a data structure according to the present invention can also be stored in a computer-readable storage medium. The computer-readable storage medium includes any type of storage device or distributed storage device in which computer-readable data is stored. The computer-readable storage medium can include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable storage medium can also include media realized in the form of a carrier wave (e.g., transmission over the Internet). The bitstream generated by the encoding method can be stored in a computer-readable storage medium or transmitted via a wired or wireless communication network.

[0418] Furthermore, embodiments of the present invention may be realized as a computer program product with program code, which may be executed on a computer according to embodiments of the present invention. The program code may be stored on a computer-readable carrier.

[0419] The above-described embodiments are combinations of elements and features of the present invention in a predetermined form. Each element or feature should be considered optional unless otherwise explicitly stated. Each element or feature may be implemented without being combined with other elements or features. It is also possible to combine some elements and / or features to form an embodiment of the present invention. The order of operations described in the embodiments of the present invention may be changed. Some elements or features of any embodiment may be included in other embodiments, or may be substituted with corresponding elements or features of other embodiments. It is obvious that claims that do not have an explicit reference relationship in the claims may be combined to form an embodiment, or may be included as new claims by amendment after filing.

[0420] Embodiments of the present invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. In a hardware implementation, an embodiment of the present invention may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0421] In the case of implementation by firmware or software, an embodiment of the present invention may be implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above. The software code may be stored in a memory and driven by a processor. The memory may be located inside or outside the processor and may transmit and receive data to and from the processor by various means known in the art.

[0422] It is obvious to those skilled in the art that the present invention can be embodied in other specific forms without departing from the essential characteristics of the present invention. Therefore, the above detailed description should not be interpreted as limiting in all respects, but should be regarded as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention. [Industrial Applicability]

[0423] The above-described preferred embodiments of the present invention have been disclosed for illustrative purposes, and those skilled in the art may improve, modify, substitute or add various other embodiments within the technical idea and technical scope of the present invention disclosed in the appended claims below.

Claims

1. 1. A method for an apparatus to decode an image signal, comprising: determining the input and output lengths of a non-separable transform based on the height and width of the current block; determining a non-separable transform matrix associated with the input length and the output length of the non-separable transform; applying the non-separable transform matrix to the coefficients of the current block, the number of coefficients being related to the input length of the non-separable transform; performing a linear inverse transform on the coefficients to which the non-separable transform has been applied; the input length and the output length of the non-separable transform are determined independently; The size of the non-separable transformation matrix is ​​determined as one of four predefined sizes based on the height and the width of the current block; the input length of the non-separable transform is determined to be 16 based on both the height and the width of the current block being greater than 8, and the output length of the non-separable transform is greater than the input length of the non-separable transform; The method of claim 1, wherein the input length of the non-separable transform is determined to be 8 based on both the height and the width of the current block being equal to 8, and the output length of the non-separable transform is greater than the input length of the non-separable transform.

2. 1. A method for an apparatus for encoding an image signal, comprising: performing a linear transformation on the current block; determining an input length and an output length of a non-separable transform based on the height and width of the current block; determining a non-separable transform matrix for the current block, the size of the non-separable transform matrix being determined based on the input length and the output length of the non-separable transform; applying the non-separable transform matrix to the coefficients of the linearly transformed current block, the number of coefficients being related to the input length of the non-separable transform; encoding non-separable transform index information for the non-separable transform matrix for the current block; the input length and the output length of the non-separable transform are determined independently; The size of the non-separable transformation matrix is ​​determined as one of four predefined sizes based on the height and the width of the current block; the output length of the non-separable transform is determined to be 16 based on both the height and the width of the current block being greater than 8, and the input length of the non-separable transform is greater than the output length of the non-separable transform; The method of claim 1, wherein the output length of the non-separable transform is determined to be 8 based on both the height and the width of the current block being equal to 8, and the input length of the non-separable transform is greater than the output length of the non-separable transform.

3. 1. A method for transmitting a bitstream generated by a method for encoding an image signal, wherein the method for encoding an image signal comprises: performing a linear transformation on the current block; determining an input length and an output length of a non-separable transform based on the height and width of the current block; determining a non-separable transform matrix for the current block, the size of the non-separable transform matrix being determined based on the input length and the output length of the non-separable transform; applying the non-separable transform matrix to the coefficients of the linearly transformed current block, the number of coefficients being related to the input length of the non-separable transform; encoding non-separable transform index information for the non-separable transform matrix for the current block into the bitstream; the input length and the output length of the non-separable transform are determined independently; The size of the non-separable transformation matrix is ​​determined as one of four predefined sizes based on the height and the width of the current block; the output length of the non-separable transform is determined to be 16 based on both the height and the width of the current block being greater than 8, and the input length of the non-separable transform is greater than the output length of the non-separable transform; The method of claim 1, wherein the output length of the non-separable transform is determined to be 8 based on both the height and the width of the current block being equal to 8, and the input length of the non-separable transform is greater than the output length of the non-separable transform.

Images