Image encoding / decoding method and apparatus, and recording medium for storing bitstreams.

Abstract

The image decoding method and apparatus according to this disclosure can acquire residual information from a bitstream, derive transformation coefficients of the current block based on the residual information, perform at least one of inverse quantization or inverse transformation on the transformation coefficients of the current block to derive a residual sample of the current block, and reconstruct the current block based on the residual sample of the current block. Here, the inverse transformation may be performed based on at least one of unseparated linear transformation, low-frequency unseparated transformation, or separated linear transformation.

Description

[Technical Field]

[0001] The present invention relates to an image encoding / decoding method and apparatus, and a recording medium storing a bitstream. [Background technology]

[0002] In recent years, the demand for high-resolution, high-quality images such as HD (High Definition) and UHD (Ultra High Definition) images has increased in various application fields, leading to discussions about highly efficient image compression technologies.

[0003] Various image compression techniques exist, such as inter-prediction techniques that predict pixel values ​​contained in the current picture from previous or subsequent pictures; intra-prediction techniques that predict pixel values ​​contained in the current picture using pixel information within the current picture; and entropy coding techniques that assign short codes to frequently occurring values ​​and long codes to less frequently occurring values. By using such image compression techniques, image data can be effectively compressed and transmitted or stored. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] This disclosure aims to provide a method and apparatus for performing a transformation using an unseparated first-order transformation.

[0005] This disclosure aims to provide a method and apparatus for performing a transformation using a reduced-dimensional non-separable linear transformation kernel.

[0006] This disclosure aims to provide a method and apparatus for determining / signaling an unseparated transform kernel based on coding parameters. [Means for solving the problem]

[0007] The image decoding method and apparatus according to this disclosure can acquire residual information from a bitstream, derive transformation coefficients for the current block based on the residual information, perform at least one of inverse quantization or inverse transformation on the transformation coefficients for the current block to derive a residual sample for the current block, and restore the current block based on the residual sample for the current block.

[0008] In the image decoding method and apparatus relating to this disclosure, the inverse transform may be performed based on at least one of the following: a non-separable primary transform (NSPT), a low-frequency non-separable transform (LFNST), or a separable primary transform.

[0009] In the image decoding method and apparatus relating to this disclosure, the predefined allowable conversion block sizes may be distinguished into a plurality of groups, and the plurality of groups may include at least two of the following: a first group including block sizes with a minimum width and height of 4; a second group including block sizes with a minimum width and height of 8; a third group including block sizes with a minimum width and height of 16; a fourth group including block sizes greater than or equal to 16x16; a fifth group including block sizes greater than or equal to 32x32; or a sixth group including block sizes greater than or equal to 8x8.

[0010] In the image decoding method and apparatus relating to this disclosure, any one of the plurality of groups may include one or more block sizes to which the NSPT can be applied.

[0011] In the image decoding method and apparatus relating to this disclosure, any one of the plurality of groups may include one or more block sizes to which the NSPT and LFNST do not apply.

[0012] In the image decoding method and apparatus relating to this disclosure, any one of the plurality of groups may include one or more block sizes to which the LFNST can be applied.

[0013] In the image decoding method and apparatus according to this disclosure, the NSPT may be applied to 4xL blocks and Lx4 blocks (4≦L≦32) belonging to the first group, and the NSPT and LFNST may not be applied to 4xM blocks and Mx4 blocks (M>32) belonging to the first group.

[0014] In the image decoding method and apparatus according to this disclosure, the NSPT may be applied to 4x4 blocks belonging to the first group, and the LFNST may be applied to 4xL blocks and Lx4 blocks (L≧8) belonging to the first group.

[0015] In the image decoding method and apparatus according to this disclosure, the NSPT may be applied to 4x4 blocks belonging to the first group, and the NSPT and LFNST may not be applied to 4xL blocks and Lx4 blocks (L≧8) belonging to the first group.

[0016] In the image decoding method and apparatus according to this disclosure, the NSPT may be applied to 8xL blocks and Lx8 blocks (8≦L≦32) belonging to the second group, and the NSPT and LFNST may not be applied to 8xM blocks and Mx8 blocks (M>32) belonging to the second group.

[0017] In the image decoding method and apparatus according to this disclosure, the NSPT may be applied to 8x8 blocks belonging to the second group, and the LFNST may be applied to 8xL blocks and Lx8 blocks (L≧16) belonging to the second group.

[0018] In the image decoding method and apparatus according to the present disclosure, the NSPT may be applied to an 8x8 block belonging to the second group, and the NSPT and the LFNST may not be applied to an 8xL block and an Lx8 block (L≥16) belonging to the second group.

[0019] In the image decoding method and apparatus according to the present disclosure, the NSPT may not be applied to at least one of the third group to the sixth group.

[0020] The image encoding method and apparatus according to the present disclosure can induce a residual sample of a current block, perform at least one of conversion or quantization on the residual sample of the current block to induce a transform coefficient of the current block, and encode the transform coefficient of the current block. Here, the conversion may be performed based on at least one of a non-separable primary transform (NSPT), a separable primary transform, or a low frequency non-separable transform (LFNST).

[0021] In the image encoding method and apparatus according to the present disclosure, a predefined allowable transform block size is divided into a plurality of groups, and the plurality of groups may include at least two of a first group including a block size whose minimum value of width and height is 4, a second group including a block size whose minimum value of width and height is 8, a third group including a block size whose minimum value of width and height is 16, a fourth group including a block size greater than or equal to 16x16, a fifth group including a block size greater than or equal to 32x32, or a sixth group including a block size greater than or equal to 8x8.

[0022] A computer-readable digital storage medium is provided, which stores encoded video / image information that is configured to be decoded using the decoding device relating to this disclosure.

[0023] A computer-readable digital storage medium is provided on which video / image information generated by the image encoding method relating to this disclosure is stored.

[0024] A method and apparatus for transmitting video / image information generated by the image encoding method relating to this disclosure are provided. [Effects of the Invention]

[0025] This disclosure shows that the performance of the transformation can be improved by using an unseparated linear transformation as the first-order transformation.

[0026] This disclosure demonstrates that the performance of a transformation can be improved by performing the transformation using a reduced-dimensional, non-separable linear transformation kernel.

[0027] This disclosure can improve coding efficiency by effectively determining and / or signaling an unseparated transform kernel based on coding parameters. [Brief explanation of the drawing]

[0028] [Figure 1] This figure shows the video / image coding system related to this disclosure. [Figure 2] This is a schematic block diagram of an encoding device to which the embodiments of the present disclosure can be applied, in which video / image signals are encoded. [Figure 3] This is a schematic block diagram of a decoding device to which the embodiments of the present disclosure can be applied, in which video / image signals are decoded. [Figure 4] This figure shows an embodiment of the present disclosure, illustrating an image decoding method performed by a decoding device (300). [Figure 5]This figure illustrates the intra-prediction mode and its prediction direction related to this disclosure. [Figure 6] This figure shows a schematic configuration of a decoding device (300) that performs the image decoding method relating to this disclosure. [Figure 7] This figure shows an embodiment of the present disclosure, which is an image encoding method performed by an encoding device (200). [Figure 8] This figure shows a schematic configuration of an encoding device (200) that performs the image encoding method relating to this disclosure. [Figure 9] This figure shows an example of a content streaming system to which the embodiments of this disclosure can be applied. [Modes for carrying out the invention]

[0029] This disclosure may be modified in various ways and may have various embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the disclosure to any particular embodiment, but rather should be understood to include all modifications, equivalents, or substitutions that fall within the spirit and technical scope of this disclosure. In the description of each figure, similar reference numerals are used for similar components.

[0030] Terms such as "First," "Second," etc., may be used to describe various components, but these components should not be limited by such terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without exceeding the scope of rights of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related descriptions or any one of multiple related descriptions.

[0031] When it is stated that one component is "connected" or "linked" to another component, it should be understood that it may be directly connected or linked to the other component, and that there may be other components in between. On the other hand, when it is stated that one component is "directly connected" or "linked" to another component, it should be understood that there are no other components in between.

[0032] The terminology used in this application is solely for the purpose of describing specific embodiments and is not intended to limit the disclosure. Singular expressions include plural expressions unless otherwise specified in the context. In this application, terms such as “includes” or “having” are intended to specify the existence of features, figures, stages, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the possibility of the existence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0033] This disclosure relates to video / image coding. For example, the methods / examples disclosed herein may be applied to methods disclosed in the VVC (versatile video coding) standard. Also, the methods / examples disclosed herein may be applied to methods disclosed in the EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0034] This specification presents various embodiments relating to video / image coding, and unless otherwise specified, the above embodiments may be combined with each other.

[0035] In this specification, video can mean a collection of images over time. Picture generally means a unit representing a single image at a specific time point, and slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may contain one or more CTUs (coding tree units). A picture may consist of one or more slices / tiles. A tile is a rectangular area consisting of multiple CTUs in a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs having the same height as the picture height and the width specified by the syntax requirements of the picture parameter set. A tile row is a rectangular area of ​​CTUs having the same height as the picture parameter set and the width as the picture width. CTUs within a tile may be arranged consecutively by a CTU raster scan, while tiles within a picture may be arranged consecutively by a tile raster scan. A slice may contain an integer number of complete tiles or an integer number of consecutive complete CTU rows within a picture tile that can exclusively be contained within a single NAL unit. On the other hand, a picture may be divided into two or more subpictures. A subpicture may be a rectangular region of one or more slices in the picture.

[0036] A pixel, or pel, can refer to the smallest unit that makes up a picture (or image). The term "sample" may also be used as a counterpart to "pixel." A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value for the lumen component, or only the pixel / pixel value for the chroma component.

[0037] A unit can refer to a basic unit of image processing. A unit may contain at least one of the following: a specific region of a picture and information associated with that region. A unit may contain one lumen block and two chroma (e.g., cb, cr) blocks. The term unit may, as in some cases, be used interchangeably with terms such as block or area. Generally, an MxN block may contain a sample (or sample array) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0038] In this specification, "A or B" can mean "A only," "B only," or "both A and B." In other words, in this specification, "A or B" may be interpreted as "A and / or B." For example, in this specification, "A, B or C" can mean "A only," "B only," "C only," or "any combination of A, B and C."

[0039] In this specification, a slash ( / ) or comma can mean "and / or". For example, "A / B" can mean "A and / or B". Thus, "A / B" can mean "A only", "B only", or "both A and B". For example, "A, B, C" can mean "A, B or C".

[0040] In this specification, "at least one of A and B" can mean "A only," "B only," or "both A and B." Furthermore, in this specification, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as equivalent to "at least one of A and B."

[0041] Furthermore, in this specification, "at least one of A, B and C" can mean "A only," "B only," "C only," or "any combination of A, B and C." Also, "at least one of A, B or C" or "at least one of A, B and / or C" can mean "at least one of A, B and C."

[0042] Furthermore, parentheses used in this specification can mean "for example." Specifically, when "prediction (intra prediction)" is displayed, "intra prediction" may be proposed as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra prediction," and "intra prediction" may be proposed as an example of "prediction." Also, when "prediction (i.e., intra prediction)" is displayed, "intra prediction" may be proposed as an example of "prediction."

[0043] In this specification, technical features described individually in the same drawings may be embodied individually or simultaneously.

[0044] Figure 1 shows the video / image coding system related to this disclosure.

[0045] Referring to Figure 1, the video / image coding system may include a first device (source device) and a second device (receiving device).

[0046] A source device can transmit encoded video / image information or data to a receiving device in the form of a file or streaming via a digital storage medium or network. The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may also be called a video / image encoding device, and the decoding device may also be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, which may consist of a separate device or external component.

[0047] A video source can acquire video / images through processes such as video / image capture, synthesis, or generation. A video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include a computer, tablet, and smartphone, and can generate video / images (electronically). For example, virtual video / images may be generated through a computer, in which case the video / image capture process may be replaced by the process of generating the associated data.

[0048] An encoding device can encode input video / images. The encoding device can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) may be output in the form of a bitstream.

[0049] The transmission unit can transmit encoded video / image information or data output in bitstream format to the receiving unit of a receiving device via a digital storage medium or network in file or streaming format. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for generating media files in a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0050] A decoding device can decode video / images by performing a series of steps, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of an encoding device.

[0051] The renderer can render the decoded video / image. The rendered video / image may be displayed on the display unit.

[0052] Figure 2 is a schematic block diagram of an encoding device to which the embodiments of this disclosure can be applied, in which video / image signals are encoded.

[0053] Referring to Figure 2, the encoding device 200 may include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-prediction unit (221) and an intra-prediction unit (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit 250 may be called a reconstructor or a reconstructed block generator. The image segmentation unit 210, prediction unit 220, residual processing unit 230, entropy encoding unit 240, addition unit 250, and filtering unit 260 described above may be composed of one or more hardware components (e.g., an encoding device chipset or processor) depending on the embodiment. The memory 270 may also include a DPB (decoded picture buffer) and may be composed of a digital storage medium. The hardware components may further include the memory 270 as an internal / external component.

[0054] The image splitting unit 210 can split an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) using a QTBTTT (Quad-tree binary-tree ternary-tree) structure.

[0055] As an example, a single coding unit may be divided into multiple coding units having deeper depths based on a quad-tree structure, a binary tree structure, and / or a tertiary structure. In this case, for example, the quad-tree structure may be applied first, followed by the binary tree structure and / or tertiary structure. Alternatively, the binary tree structure may be applied before the quad-tree structure. The coding procedure according to this specification may be performed based on a final coding unit that is not further divided. In this case, based on coding efficiency due to image characteristics, the largest coding unit may be immediately used as the final coding unit, or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and reconstruction, which will be described later.

[0056] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be separated or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit that derives a transform coefficient and / or a unit that derives a residual signal from the transform coefficient.

[0057] The term "unit" may be used interchangeably with terms such as "block" or "area." Generally, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumen component, or only the pixel / pixel value of the chroma component. The term "sample" may be used in conjunction with a single picture (or image), pixel, or pel.

[0058] The encoding device 200 can generate a residual signal (residual block, residual sample array) by subtracting the prediction signal (prediction block, prediction sample array) output from the inter-prediction unit 221 or intra-prediction unit 222 from the input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, the unit that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) within the encoding device 200 may be called the subtraction unit 231.

[0059] The prediction unit 220 can make predictions for the block to be processed (hereinafter referred to as the current block) and generate a predicted block containing prediction samples for the current block. The prediction unit 220 can determine whether intra-prediction or inter-prediction is applied to the current block or on a CU basis. As will be described later in the explanation of each prediction mode, the prediction unit 220 can generate various information related to prediction, such as prediction mode information, and transmit it to the entropy encoding unit 240. The information related to prediction may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0060] The intra-prediction unit 222 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block, or it may be located at a certain distance from the current block, depending on the prediction mode. In intra-prediction, the prediction mode may include one or more non-directional modes and multiple directional modes. The non-directional mode may include at least one of DC mode or planar mode. The directional mode may include 33 or 65 directional modes, depending on the degree of fineness of the prediction direction. However, this is an example, and more or fewer directional modes may be used depending on the settings. The intra-prediction unit 222 can also determine the prediction mode to be applied to the current block using the prediction modes applied to the surrounding blocks.

[0061] The interprediction unit 221 can guide a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between the surrounding block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, the surrounding block may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture containing the reference block and the reference picture containing the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, colCU, etc., and the reference picture containing the temporal neighboring block may be called a collocated picture (colPic). For example, the interpretation unit 221 can construct a motion information candidate list based on surrounding blocks and generate information indicating which candidates are used to derive the motion vector and / or reference picture index of the current block. Interpretation may be performed based on various prediction modes; for example, in skip mode and merge mode, the interpretation unit 221 can use the motion information of surrounding blocks as the motion information of the current block. In skip mode, unlike merge mode, the residual signal does not need to be transmitted.In motion vector prediction (MVP) mode, the motion vectors of surrounding blocks are used as motion vector predictors, and the motion vector difference is signaled to indicate the motion vector of the current block.

[0062] The prediction unit 220 can generate prediction signals based on various prediction methods described later. For example, the prediction unit can apply intra-prediction or inter-prediction for predictions for a single block, and can also apply intra-prediction and inter-prediction simultaneously. This may be called CIIP (combined inter and intra prediction) mode. The prediction unit may also be based on intra-block copy (IBC) prediction mode or palette mode for predictions for blocks. The IBC prediction mode or palette mode may be used for coding content images / videos such as games, as in SCC (screen content coding). IBC basically performs predictions within the current picture, but can be performed similarly to inter-prediction in that it derives reference blocks within the current picture. That is, IBC can use at least one of the inter-prediction methods described herein. Palette mode may be considered an example of intra-coding or intra-prediction. When palette mode is applied, in-picture sample values ​​can be signaled based on information about the palette table and palette index. The prediction signal generated by the prediction unit 220 may be used to generate a restoration signal or to generate a residual signal.

[0063] The transformation unit 232 can generate transformation coefficients by applying a transformation method to the residual signal. For example, the transformation method may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to the transformation obtained from a graph when the relationship information between pixels is represented by a graph. CNT refers to the transformation obtained by generating a prediction signal using all previously restored pixels and obtaining a transformation based on it. Furthermore, the transformation process may be applied to pixel blocks of the same size that are square, or to blocks of variable size that are not square.

[0064] The quantization unit 233 quantizes the conversion coefficients and transmits them to the entropy encoding unit 240, which can encode the quantized signal (information regarding the quantized conversion coefficients) and output it as a bitstream. The information regarding the quantized conversion coefficients may be called residual information. The quantization unit 233 can rearrange the block-shaped quantized conversion coefficients into a one-dimensional vector based on the coefficient scan order, and can generate information regarding the quantized conversion coefficients based on the one-dimensional vector-shaped quantized conversion coefficients.

[0065] The entropy encoding unit 240 can perform various encoding methods such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 240 can also encode information necessary for video / image restoration (e.g., the values ​​of syntax elements) together with or separately from the quantized conversion coefficients.

[0066] The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information about various parameter sets, such as an adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. In this specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded by the encoding procedure described above and included in the bitstream. The bitstream may be transmitted over a network or stored on a digital storage medium. Here, the network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 may be transmitted by a transmission unit (not shown) and / or stored by a storage unit (not shown) which are configured as internal / external elements of the encoding device 200, or the transmission unit may be included in the entropy encoding unit 240.

[0067] The quantized conversion coefficients output from the quantization unit 233 may be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized conversion coefficients in the inverse quantization unit 234 and the inverse transformation unit 235, the residual signal (residual block or residual sample) can be reconstructed. The adder unit 250 may add the reconstructed residual signal to the prediction signal output from the inter-prediction unit 221 or the intra-prediction unit 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block may be used as the reconstructed block. The adder unit 250 may be called the reconstruction unit or the reconstructed block generation unit. The generated reconstructed signal may be used for intra-prediction of the next block to be processed in the current picture, and may be used for inter-prediction of the next picture after filtering, as described later. On the other hand, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or reconstruction process.

[0068] The filtering unit 260 can apply filtering to the restored signal to improve subjective / objective image quality. For example, the filtering unit 260 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 270, specifically in the DPB of the memory 270. The various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit 260 can generate various filtering-related information and transmit it to the entropy encoding unit 240. The filtering-related information may be encoded by the entropy encoding unit 240 and output in bitstream format.

[0069] The corrected restored picture transmitted to memory 270 may be used as a reference picture in the interpretation unit 221. This allows the encoding device to avoid prediction mismatches between the encoding device 200 and the decoding device when interpretation is applied, and also improves encoding efficiency.

[0070] The DPB in memory 270 can store the corrected restored picture for use as a reference picture in the inter-prediction unit 221. Memory 270 can store motion information of blocks from which motion information in the current picture has been derived (or encoded), and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 221 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 270 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 222.

[0071] Figure 3 is a schematic block diagram of a decoding device to which the embodiments of this disclosure can be applied, in which video / image signals are decoded.

[0072] Referring to Figure 3, the decoding device 300 may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (331) and an intra-prediction unit (332). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321).

[0073] The entropy decoding unit 310, the residual processing unit 320, the prediction unit 330, the addition unit 340, and the filtering unit 350 described above may be configured by a single hardware component (e.g., a decoding device chipset or processor) depending on the embodiment. The memory 360 may include a DPB (decoded picture buffer) and may be configured by a digital storage medium. The hardware component may further include the memory 360 as an internal / external component.

[0074] When a bitstream containing video / image information is input, the decoding device 300 can reconstruct the image in accordance with the process by which the video / image information was processed in the encoding device shown in Figure 2. For example, the decoding device 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device 300 can perform decoding using the processing units applied in the encoding device. Therefore, the decoding processing units may be coding units, which may be divided from a coding tree unit or a maximum coding unit according to a quad-tree structure, a binary tree structure, and / or a tertiary tree structure. One or more conversion units may be derived from the coding unit. The reconstructed image signal decoded and output by the decoding device 300 may then be reproduced by a playback device.

[0075] The decoding device 300 can receive the signal output from the encoding device shown in Figure 2 in the form of a bitstream, and the received signal may be decoded by the entropy decoding unit 310. For example, the entropy decoding unit 310 can parse the bitstream and derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information about various parameter sets such as the adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. The decoding device can decode the picture based on the parameter set information and / or the general constraint information. The signal / received information and / or syntax elements described later in this specification may be decoded by the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image reconstruction and the quantized values ​​of conversion coefficients related to the residual. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded and the decoding information of the surrounding and decoded blocks or symbol / bin information decoded in a previous stage, predicts the probability of bin occurrence based on the determined context model, and generates symbols corresponding to the values ​​of each syntax element by performing arithmetic decoding of the bins. At this time, after determining the context model, the CABAC entropy decoding method can update the context model using the symbol / bin information decoded for the context model of the next symbol / bin.Information related to prediction from the information decoded by the entropy decoding unit 310 is provided to the prediction unit (inter-prediction unit 332 and intra-prediction unit 331), and residual values ​​that have been entropy decoded by the entropy decoding unit 310, i.e., quantized conversion coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). In addition, information related to filtering from the information decoded by the entropy decoding unit 310 may be provided to the filtering unit 350. On the other hand, a receiving unit (not shown) that receives signals output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310.

[0076] On the other hand, the decoding device according to this specification may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoding device (video / image / picture information decoding device) and a sample decoding device (video / image / picture sample decoding device). The information decoding device may include the entropy decoding unit 310, and the sample decoding device may include at least one of the inverse quantization unit 321, inverse transformation unit 322, addition unit 340, filtering unit 350, memory 360, inter-prediction unit 332, and intra-prediction unit 331.

[0077] The inverse quantization unit 321 can inverse quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 321 can rearrange the quantized transformation coefficients in a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 321 can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain the transformation coefficients.

[0078] The inverse conversion unit 322 performs an inverse conversion on the conversion coefficients to obtain a resistive signal (residual block, resistive sample array).

[0079] The prediction unit 320 can make predictions for the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit 310, the prediction unit 320 can determine whether intra-prediction or inter-prediction is applied to the current block and determine a specific intra / inter-prediction mode.

[0080] The prediction unit 320 can generate prediction signals based on various prediction methods described later. For example, the prediction unit 320 can apply intra-prediction or inter-prediction for prediction of a single block, or it can apply intra-prediction and inter-prediction simultaneously. This may be called CIIP (combined inter and intra prediction) mode. The prediction unit may also be based on intra-block copy (IBC) prediction mode or palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / video coding such as SCC (screen content coding) for games. IBC basically performs prediction within the current picture, but can be performed similarly to inter-prediction in that it derives a reference block within the current picture. That is, IBC can use at least one of the inter-prediction methods described herein. Palette mode can be considered an example of intra-coding or intra-prediction. When palette mode is applied, information about the palette table and palette index may be included in the video / image information and signaled.

[0081] The intra-prediction unit 331 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block, or it may be located at a certain distance from the current block, depending on the prediction mode. In intra-prediction, the prediction mode may include one or more non-directional modes and multiple directional modes. The intra-prediction unit 331 can determine the prediction mode to be applied to the current block using the prediction modes applied to the surrounding blocks.

[0082] The interprediction unit 332 can derive a predicted block for the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between surrounding blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, surrounding blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the interprediction unit 332 can construct a motion information candidate list based on surrounding blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interprediction may be performed based on various prediction modes, and the prediction information may include information indicating the interprediction mode for the current block.

[0083] The adder 340 can generate a restored signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the prediction unit (including the inter-prediction unit 332 and / or intra-prediction unit 331). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block may be used as the restored block.

[0084] The summing unit 340 may be called the restoration unit or the restoration block generation unit. The generated restoration signal may be used for intra-prediction of the next block to be processed in the current picture, and may be output after filtering as described later, or may be used for intra-prediction of the next picture. On the other hand, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0085] The filtering unit 350 can apply filtering to the restored signal to improve subjective / objective image quality. For example, the filtering unit 350 can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory 360, specifically to the DPB of the memory 360. The various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like.

[0086] The restored picture stored (modified) in the DPB of memory 360 may be used as a reference picture in the inter-prediction unit 332. Memory 360 can store motion information of blocks from which motion information in the current picture has been derived (or decoded), and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 260 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 360 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 331.

[0087] In this specification, the embodiments described for the filtering unit 260, the inter-prediction unit 221, and the intra-prediction unit 222 of the encoding device 200 may also be applied identically or in a corresponding manner to the filtering unit 350, the inter-prediction unit 332, and the intra-prediction unit 331 of the decoding device 300, respectively.

[0088] Figure 4 is a diagram illustrating an embodiment of the present disclosure, which shows an image decoding method performed by a decoding device.

[0089] Referring to Figure 4, the transform coefficients of the current block can be derived from the bitstream (S400). That is, the bitstream may contain the current block's residual information, and the transform coefficients of the current block can be derived by decoding this residual information.

[0090] Referring to Figure 4, residual samples of the current block can be derived by performing at least one of dequantization or inverse-transform on the transformation coefficients of the current block (S410).

[0091] When Adaptive Multiple Transform Selection (MTS) is applied, the inverse transform may be performed based on at least one of DCT-2, DST-7, or DCT-8. Here, DCT-2, DST-7, DCT-8, etc. may be called the transform type, transform kernel, or transform core.

[0092] In this disclosure, the inverse transform can mean a separable transform. However, it is not limited to this, and the inverse transform can also mean a non-separable transform, or it may be a concept that includes both separable and non-separable transforms. Furthermore, in this disclosure, the inverse transform means a primary transform, but it is not limited to this, and it may be transformed into the same / similar form and applied to a secondary transform.

[0093] For example, as a method for the inverse transform, only DCT-2 and the inseparable transform may be used, or the inseparable transform may be used in addition to at least one of DCT-2, DST-7, or DCT-8, or the inseparable transform may be used in place of one or more of the transform kernels from DCT-2, DST-7, or DCT-8.

[0094] As a more specific example, if the candidate transformation kernels for delimited transformations are (DCT-2,DCT-2), (DST-7,DST-7), (DCT-8,DST-7), (DST-7,DCT-8), and (DCT-8,DCT-8), then the non-delimited transformation may substitute for or add to one or more of the five candidate transformation kernels. Here, notation such as (transform 1, transform 2) indicates that transform 1 is applied horizontally and transform 2 is applied vertically. When the non-delimited transformation substitutes for some of the candidate transformation kernels, the remaining candidate transformation kernels, excluding (DCT-2,DCT-2) and (DST-7,DST-7), can be replaced by the non-delimited transformation. However, the candidate transformation kernels are merely examples, and other types of DCT and / or DST may be included, and a transform skip may be included as a candidate transformation kernel.

[0095] An inseparable transform can refer to a transform or inverse transform based on a non-separable transform matrix. That is, unlike a separable transform, which separates vertical and horizontal transforms and performs them independently, an inseparable transform allows both horizontal and vertical transforms to be performed in a single step.

[0096] For example, when an inseparable transformation is performed on a 4x4 block, the input data X for the inseparable transformation is given by the following equation 1.

[0097] [Formula 1]

number

[0098] When the input data X is represented in vector form, the vector X' may be represented as follows:

[0099] [Formula 2]

number

[0100] In this case, the non-separable transformation may be performed as shown in the following equation 3.

[0101] [Formula 3]

number

[0102] In equation 3, F represents the transformation coefficient vector, T represents the 16x16 inseparable transformation matrix, and · signifies the multiplication of a matrix and a vector.

[0103] The above equation 3 may derive a 16x1 transformation coefficient vector F, which may be reconstructed into a 4x4 block according to a predetermined scan sequence. The scan sequence may be a horizontal scan, a vertical scan, a diagonal scan, a z scan, a raster scan, or a predefined scan.

[0104] The set of inseparable transforms and / or transform kernels for the inseparable transforms may be configured in various ways based on at least one of the following: prediction mode (e.g., intra mode, inter mode, etc.), width, height, or number of pixels of the current block, position of subblocks within the current block, explicitly signaled syntactic elements, statistical characteristics of the surrounding samples, whether or not a quadratic transform is used, or quantization parameters (QP).

[0105] Specifically, in intra-mode, predefined intra-prediction modes are grouped to correspond to n inseparable transformation sets, and each inseparable transformation set may contain k transformation kernel candidates. Here, n and k may be arbitrary constants that follow the same rules (conditions) defined for the encoding and decoding devices.

[0106] The number of non-separable transformation sets and / or the number of transformation kernel candidates included in each non-separable transformation set may be configured to differ according to the width and / or height of the current block. For example, for a 4x4 block, n1 non-separable transformation sets and k1 transformation kernel candidates may be configured. For a 4x8 block, n2 non-separable transformation sets and k2 transformation kernel candidates may be configured. Furthermore, the number of non-separable transformation sets and the number of transformation kernel candidates included in each non-separable transformation set may be configured to differ according to the product of the width and height of the current block. For example, if the product of the width and height of the current block is 256 or greater (or exceeds 256), n3 non-separable transformation sets and k3 transformation kernel candidates may be configured; otherwise, n4 non-separable transformation sets and k4 transformation kernel candidates may be configured. In other words, since the degree of change in the statistical characteristics of the residual signal differs according to the block size, the number of non-separable transformation sets and transformation kernel candidates may be configured to reflect this.

[0107] Currently, when a block is divided into multiple subblocks, the statistical characteristics of the residual signal may differ for each subblock. Therefore, the number of non-separable transform sets and transform kernel candidates may be configured to be different for each subblock. For example, when a 4x8 or 8x4 block is divided into two 4x4 subblocks and a non-separable transform is applied to each subblock, n5 non-separable transform sets and k5 transform kernel candidates may be configured for the upper left 4x4 subblock, and n6 non-separable transform sets and k6 transform kernel candidates may be configured for the other 4x4 subblocks.

[0108] The number of inseparable transformation sets and candidate transformation kernels may be configured to be different from each other based on an explicitly signaled syntactic element. The syntactic element may be information indicating one of several inseparable transformation configurations. For example, if three types of inseparable transformation configurations are supported (i.e., n7 inseparable transformation sets and k7 candidate transformation kernels, n8 inseparable transformation sets and k8 candidate transformation kernels, and n9 inseparable transformation sets and k9 candidate transformation kernels), the syntactic element may have values ​​of 0, 1, or 2, and the value of the signaled syntactic element may determine which inseparable transformation configuration is applied to the current block.

[0109] The number of non-separable transformation sets and transformation kernel candidates may be configured differently depending on whether a quadratic transformation is applied and / or what kind of quadratic transformation is applied. For example, if no quadratic transformation is applied, n 10 A set of inseparable transformations and k 10 A non-separable transform configuration can be applied, containing n transform kernel candidates. When a quadratic transform is applied, n 11 A set of inseparable transformations and k 11 A non-separable transformation configuration can be applied, including several candidate transformation kernels.

[0110] Different non-separable transformation configurations may be applied depending on the quantization parameter (QP) and / or the range of QP values. For example, when the QP value is small, n 12 A set of inseparable transformations and k 12A non-separable transformation configuration including n transformation kernel candidates can be applied. On the other hand, when the QP value is large, n 13 A set of inseparable transformations and k 13 An inseparable transformation configuration containing n transformation kernel candidates can be applied. If the QP value is less than or equal to a threshold (e.g., 32), it can be classified as having a small QP value; otherwise, it can be classified as having a large QP value. Alternatively, the range of QP values ​​can be divided into three or more ranges, and a different inseparable transformation configuration can be applied to each range.

[0111] For relatively large blocks, instead of using an inseparable transformation corresponding to the width and height of the block, the block can be divided into multiple subblocks, and inseparable transformations corresponding to the width and height of the subblocks can be used. For example, when performing an inseparable transformation on a 4x8 block, the 4x8 block can be divided into two 4x4 subblocks, and a 4x4 block-based inseparable transformation can be used for each 4x4 subblock. Alternatively, for an 8x16 block, it can be divided into two 8x8 subblocks, and an 8x8 block-based inseparable transformation can be used.

[0112] The aforementioned non-separated transformation set may be determined based on the current block's intra-prediction mode and mapping table. The mapping table can define the mapping relationship between a predefined intra-prediction mode and the non-separated transformation set. The predefined intra-prediction mode may include two non-directional modes and 65 directional modes. In general, non-separated transformations have a larger transformation kernel size than separated transformations. This means that the computational complexity required for the transformation process is higher, and the memory required to store the transformation kernel is larger. On the other hand, separated transformations can only consider statistical properties existing in the horizontal and / or vertical directions, while non-separated transformations can simultaneously consider statistical properties in a two-dimensional space including horizontal and vertical directions, thus providing better compression efficiency. Because the statistical properties and diversity of residuals differ depending on the direction of the intra-prediction mode, there may be cases where non-separated transformations are absolutely necessary, and there may be intra-prediction modes in which the characteristics of residuals can be sufficiently captured by separated transformations alone. Therefore, by pre-defining which transformations to use in the encoding and decoding devices according to the intra-prediction mode, the transformation process can be designed with optimized complexity and memory requirements. The non-directional mode may include a planar mode (number 0) and a DC mode (number 1), and the directional mode may include intra-predictive modes (numbers 2 through 66). However, this is illustrative, and the disclosure may also apply to cases where the number of predefined intra-predictive modes differs.

[0113] By applying WAIP (wide angle intra prediction), the predefined intra-prediction modes may further include intra-prediction modes -14 to -1 and intra-prediction modes 67 to 80.

[0114] Figure 5 illustrates the intra-prediction modes and their prediction directions related to this disclosure. Referring to Figure 5, modes -14 to -1 and 2 to 33, and modes 35 to 80 are symmetrical in terms of prediction direction with respect to mode 34. For example, modes 10 and 58 are symmetrical with respect to the direction corresponding to mode 34, and mode -1 is symmetrical with mode 67. Therefore, for vertical modes that are symmetrical with respect to horizontal modes with respect to mode 34, the input data can be used after transposing. Transposing the input data means that in the 2D block of input data MxN, rows become columns and columns become rows, forming NxM data.

[0115] For example, when a 4x4 block is used, the 16 data points forming the 4x4 block can be appropriately arranged for the inseparable transformation to construct a 16x1 one-dimensional vector. In this case, the one-dimensional vector may be constructed in row-major order or column-major order. The residual samples resulting from the inseparable transformation can be arranged in the above order to construct a two-dimensional block.

[0116] For modes -14 to -1 and 2 to 33, the data array order for constructing the 16x1 input vector is row-major, while for modes 35 to 80, the input vector may be constructed using column-major order.

[0117] Mode 34 can be said to be neither a horizontal nor a vertical mode, but in this disclosure, it is classified as belonging to the horizontal mode. That is, for modes -14 to -1 and 2 to 33, the input data sorting method for horizontal modes, i.e., row-major ordering, is used, and the input data can be transposed and used for vertical modes that are symmetrical with respect to mode 34.

[0118] In non-square blocks, the symmetries found in square blocks (i.e., the symmetry between the P mode and the (68-P) mode in an NxN block (2<=P<=33) or between the Q mode and the (66-Q) mode (-14<=Q<=-1)) cannot be utilized. Therefore, in addition to symmetries based solely on the intra-predicted modes, it is also possible to utilize the symmetries between block configurations that have a prior relationship with each other, namely the symmetry between KxL blocks and LxK blocks. Specifically, a symmetry relationship exists between a KxL block predicted to be the P mode and an LxK block predicted to be the (68-P) mode. Or, a symmetry relationship exists between a KxL block predicted to be the Q mode and an LxK block predicted to be the (66-Q) mode.

[0119] Since the KxL block having mode 2 and the LxK block having mode 66 are considered symmetrical to each other, the same transformation kernel can be applied to both the KxL and LxK blocks. Assuming that an inseparable transformation set is mapped to the intra-prediction mode of the KxL block, in order to apply an inseparable transformation to the LxK block, the inseparable transformation set can be induced by the mapping table corresponding to the KxL block, based on mode (68-P) instead of mode P applied to the LxK block. Alternatively, the inseparable transformation set can be induced by the mapping table corresponding to the KxL block, based on mode (66-Q) instead of mode Q applied to the LxK block.

[0120] For example, to apply an inseparable transformation to an LxK block, the set of inseparable transformations can be selected based on mode 2 instead of mode 66. For a KxL block, the input data can be read in a predetermined order (e.g., row-major or column-major) to construct a one-dimensional vector, and then the inseparable transformation can be applied. For an LxK block, the input data can be read in a transposed order to construct a one-dimensional vector, and then the inseparable transformation can be applied. That is, if a KxL block is read in row-major order, an LxK block can be read in column-major order. Conversely, if a KxL block is read in column-major order, an LxK block can be read in row-major order.

[0121] Furthermore, when mode 34 is applied to a KxL block, the inseparable transformation set can be determined based on mode 34, the input data can be read in a predetermined order to construct a one-dimensional vector, and the inseparable transformation can be performed. Similarly, when mode 34 is applied to an LxK block, the inseparable transformation set can be determined based on mode 34, but the input data can be read in a transposed order to construct a one-dimensional vector, and the inseparable transformation can be performed.

[0122] This disclosure describes a method for determining an inseparable transformation set and a method for constructing input data based on a KxL block. However, the inseparable transformation can be performed using an LxK block as the basis, utilizing the same symmetry as described above for the KxL block. Alternatively, the use of a base block may be restricted to blocks where the width is greater than the height. Alternatively, the use of symmetry may be restricted to non-square blocks. In this case, non-square blocks may use a different number of inseparable transformation sets and / or transformation kernel candidates than square blocks, and may also use a different mapping table than square blocks to select an inseparable transformation set.

[0123] An example of a mapping table for selecting an unseparated set of transformations is as follows:

[0124] [Table 1]

[0125] Table 1 shows an example of assigning non-separable transformation sets to each intra-prediction mode when there are five non-separable transformation sets. The value of predModeIntra represents the value of the intra-prediction mode considering WAIP, and TrSetIdx is an index indicating a specific non-separable transformation set. From Table 1, it can be seen that the same non-separable transformation set is applied to modes located in a symmetrical direction relative to each other by the intra-prediction mode. Table 1 is just one example of using five non-separable transformation sets and does not limit the total number of non-separable transformation sets for non-separable transformation.

[0126] Alternatively, as shown in Table 2, non-destructive conversion may not be applied to WAIP for the sake of compression performance.

[0127] [Table 2]

[0128] Alternatively, as shown in Table 3, instead of configuring a separate non-separated conversion set for WAIP, a non-separated conversion set corresponding to an adjacent intra-predictive mode may be shared.

[0129] [Table 3]

[0130] The aforementioned non-separable transformation set may include multiple transformation kernel candidates, and one of the multiple transformation kernel candidates may be selectively used. For this purpose, an index signaled by a bitstream may be used. Alternatively, one of the multiple transformation kernel candidates may be implicitly determined based on the context information of the current block. Here, the context information can mean the size of the current block, or whether or not a non-separable transformation is applied to surrounding blocks. Here, the size of the current block may be defined as the width, height, the maximum / minimum values ​​of the width and height, the sum of the width and height, or the product of the width and height.

[0131] The following describes in detail how to determine the transformation kernel for the inverse transformation of the current block.

[0132] Example 1

[0133] As mentioned above, inverse transformations can be distinguished into separable and inseparable transformations. A separable transformation means performing transformations on a 2D block in both the horizontal and vertical directions, while an inseparable transformation means performing a single transformation on samples that constitute the entire 2D block or a portion of it. When describing a separable transformation, it can be described as a pair of horizontal and vertical transformations, and in this disclosure, it will be described as (horizontal transformation, vertical transformation).

[0134] Multiple transformation sets may be defined for the inverse transformation of a block. Each transformation set may contain one or more candidate transformation kernels.

[0135] For example, any one of (DST-7,DST-7), (DCT-8,DST-7), (DST-7,DCT-8), or (DCT-8,DCT-8) may be applied as a separate transform, and these four candidate transform kernels may be considered as a single transform set. Also, (DCT-2,DCT-2) may be considered as a single transform set. A transform skip, where no transform is applied, may also be considered as a single transform set, and (DCT-2,DCT-2) and the transform skip may be considered as a single transform set. In this disclosure, a transform kernel may represent a single transform (e.g., DCT-2, DST-7) or a pair of transforms (e.g., (DCT-2,DCT-2)).

[0136] Furthermore, another example of a set of transformations is the non-separable transformation set mentioned above. In this disclosure, a non-separable transformation applied as a primary transformation can be denoted as an NSPT (Non-Separable Primary Transform). An NSPT may consist of multiple sets of non-separable transformations, each of which may contain one or more transformation kernels as candidate transformation kernels. In the case of an NSPT, one of the multiple sets of non-separable transformations is selected by an intra-prediction mode, and the multiple sets of non-separable transformations for an NSPT can be denoted as an NSPT set list. This is as described above, and a detailed explanation is omitted here.

[0137] A group of one or more transformation sets available to the current block can be formed from a predefined set of transformations. This group of one or more transformation sets may consist of a predetermined region unit to which the current block belongs, and will hereinafter be referred to as a collection. Here, the predetermined region unit may be at least one of a picture, a slice, a coding tree unit row (CTU row), or a coding tree unit (CTU).

[0138] For example, let the transformation set composed of (DCT-2, DCT-2) be called S1, and the transformation sets composed of (DST-7, DST-7), (DCT-8, DST-7), (DST-7, DCT-8), and (DCT-8, DCT-8) be called S2, respectively. Also, the aforementioned NSPT set list may include N non-separable transformation sets, and the N non-separable transformation sets are called S 3,1 , S 3,2 ,..., S 3,N , respectively. Here, N may be 35, but is not limited thereto.

[0139] When S 3,13 is selected as the non-separable transformation set for NSPT according to the intra prediction mode of the current block, the transformation kernel applicable to the current block may belong to any one of S1, S2, or S 3,13 . In this case, the available collection of the current block can be denoted as {S1, S2, S 3,13}.

[0140] As described above, since the collection according to the present disclosure is a group of one or more transformation sets available to the current block, the collection may be configured differently according to the context of the current block. Here, the context may include at least one of form, size, or intra prediction mode. If a total of K contexts are defined, K collections may be generated, and each collection may be denoted as C i (i = 1, 2,..., N). For example, when the sizes of the blocks to which NSPT can be applied are 4x4, 8x8, 16x16, 32x32, and any one of a total of 35 non-separable transformation sets is selected according to the intra prediction mode, if the transformation kernels applied for each block size are different, a total of 4x35 = 140 contexts may be defined.

[0141] A collection may be constructed based on the context of the current block. In this case, one of several transformation sets belonging to the collection may be selected, and one of several transformation kernel candidates belonging to the selected transformation set may be selected. Here, the selection of the transformation set and the transformation kernel candidate may be performed implicitly based on the context of the current block, or based on an explicitly signaled index. Alternatively, the process of selecting one of several transformation sets belonging to the collection and the process of selecting one of several transformation kernel candidates belonging to the selected transformation set may be performed separately. For example, an index for selecting a transformation set may be signaled first, and based on this, one of several transformation sets belonging to the collection can be selected. Then, an index pointing to one of several transformation kernel candidates belonging to the transformation set may be signaled, and based on the signaled index, one transformation kernel candidate can be selected from the transformation set. The transformation kernel of the current block may be determined based on the selected transformation kernel candidate. Alternatively, the selection of one transformation set from the collection may be implicitly performed based on the current block's context, and the selection of one transformation kernel candidate from the selected transformation set may be performed based on a signaled index. Alternatively, the selection of one transformation set from the collection may be performed based on a signaled index, and the selection of one transformation kernel candidate from the selected transformation set may be implicitly performed based on the current block's context. Alternatively, the selection of one transformation set from the collection may be implicitly performed based on the current block's context, and the selection of one transformation kernel candidate from the selected transformation set may also be implicitly performed based on the current block's context. Of course, if the number of transformation sets belonging to the collection is one, an index for selecting a transformation set does not need to be signaled.Similarly, if there is only one candidate transformation kernel belonging to the selected transformation set, an index pointing to that candidate does not need to be signaled. Alternatively, an index pointing to any one of the transformation kernel candidates currently belonging to the collection may be signaled. In this case, the process of selecting one transformation set from the collection may be omitted. At this time, all transformation sets belonging to the collection may be rearranged considering priority. For example, when assigning small-length binary code to small-value indices, such as truncated unary code, it may be advantageous to assign small-value indices to transformation kernel candidates that are relatively advantageous in improving coding performance. When rearranging all transformation kernel candidates belonging to a collection according to priority (shuffling), different shuffling can be applied to each collection. Also, instead of rearranging all transformation kernel candidates belonging to a collection, only some can be selectively rearranged.

[0142] Example 2

[0143] The transform kernel for the inverse transform of the current block may be determined based on MTS (Multiple Transform Selection).

[0144] The MTS relating to this disclosure may use at least one of DST-7, DCT-8, DCT-5, DST-4, DST-1, or IDT (identity transform) as a transformation kernel. The MTS relating to this disclosure may further include a DCT-2 transformation kernel.

[0145] In this disclosure, multiple sets of MTS may be defined for an MTS. Currently, one of the multiple sets of MTS may be determined based on the block size and / or intra-prediction mode. For example, when determining one set of MTS, 16 transformation block sizes may be considered, and for directional modes, the morphology of the transformation block and the symmetry between intra-prediction modes may be considered. For WAIP (Wide Angle Intra Prediction) modes (i.e., -1 to -14 (or -15), 67 to 80 (or 81)), the MTS set corresponding to mode 2 may be applied to modes -1 to -14 (or -15), and the MTS set corresponding to mode 66 may be applied to modes 67 to 80 (or 81). A different set of MTS may be assigned to MIP (Matrix-based Intra Prediction) modes.

[0146] For example, the MTS set based on the conversion block size and intra prediction mode may be allocated / defined as shown in Table 4 below.

[0147] [Table 4]

[0148] Table 4 shows the assignment of MTS sets based on 16 different conversion block sizes and intra-prediction modes. There are 80 predefined MTS sets, and the index that points to any one of these 80 MTS sets may range from 0 to 79, as shown in Table 4.

[0149] [Table 5-1] [Table 5-2]

[0150] Table 5 shows the transformation kernel candidates included in each MTS set described in Table 4. Each MTS set may consist of six transformation kernel candidates. The transformation kernel candidate index has one value from 0 to 5 and can indicate any one of the six transformation kernel candidates. Here, each transformation kernel candidate may be a combination of a horizontal transformation kernel and a vertical transformation kernel for decoupling, and 25 transformation kernel candidates with indices from 0 to 24 may be defined.

[0151] [Table 6]

[0152] Table 6 shows an example of the 25 transformation kernel candidates described in Table 5. Specifically, the horizontal and vertical transformations of the transformation kernel candidates are shown as (horizontal transformation, vertical transformation). For each transformation kernel candidate index, the horizontal / vertical transformation when the intra-prediction mode is less than 35 may be the opposite of the horizontal / vertical transformation when the intra-prediction mode is 35 or greater. When the value of the intra-prediction mode is 35 or greater, a mode that is symmetrical around mode 34 can be induced, and an MTS set can be selected from Table 4 based on that mode. Furthermore, the symmetry of the block shape may be further considered. When the original transformation block has a WxH size, the original transformation block can be made symmetrical and considered to have an HxW size, and an MTS set can be selected from Table 4. Here, the value of the intra-prediction mode may be the value of the modified intra-prediction mode. In other words, for WAIP, modes -14 (or -15) through -1 are modified to mode 2, modes 67 through 80 (or 81) are modified to mode 66, and the remaining modes can be set to the modified intra-prediction mode values ​​using the original intra-prediction mode values. In this case, the extended modes for WAIP are also configured symmetrically around mode 34, so symmetry around mode 34 can be used for all directional modes except Planar mode and DC mode.

[0153] For example, if a 16x32 block is predicted to be mode 54, then mode 14 (=68-54) is induced as a mode symmetric to mode 54, and the block size may be considered to be 32x16. In this case, an MTS set with 72 indices may be selected, as defined in Table 4.

[0154] When MIP mode is applied, the MTS set assigned to MIP mode may be selected based on the current block size, without considering the symmetry of the block configuration. Alternatively, when MIP mode is applied, the MTS set assigned to MIP mode may be selected based on the symmetrical block size, taking into account the symmetry of the block configuration. For example, when MIP mode is applied to an 8x16 block, the 8x16 block may be considered a symmetrical 16x8 block, and the MTS set with 49 indices, as defined in Table 4, may be selected. Alternatively, when MIP mode is applied, the intra-prediction mode may be considered Planar mode. In this case, the MTS set assigned to MIP mode may be selected based on the current block size, without considering the symmetry of the block configuration. Alternatively, the MTS set assigned to MIP mode may be selected based on the symmetrical block size, taking into account the symmetry of the block configuration.

[0155] In MIP mode, a flag may be used to indicate whether or not MIP mode is applied as transpose mode. When MIP mode is applied to the current block of MxN and the flag indicates the application of transpose mode, the intra-prediction mode is considered to be Planar mode, and the current block of MxN may be considered to be an NxM block. That is, an MTS set corresponding to the NxM block size and Planar mode may be selected from Table 4. As explained in Table 6, when the value of intra-prediction mode is 35 or greater, horizontal and vertical transformations are swapped with each other, but since the intra-prediction mode of the current block is considered to be Planar mode, it is not necessary to swap the horizontal and vertical transformations of the transformation kernel candidates. Alternatively, when MIP mode is applied to the current block of MxN and the flag indicates the application of transpose mode, the intra-prediction mode is not considered to be Planar mode, and the current block of MxN may be considered to be an NxM block. That is, an MTS set corresponding to the NxM block size and MIP mode may be selected from Table 4.

[0156] In Table 5, the transformation kernel candidate selected by the transformation kernel candidate index may be set as the transformation kernel for the current block. Alternatively, depending on the size of the current block, at least one of the horizontal or vertical transformations of the selected transformation kernel candidate may be changed to another transformation kernel. For example, if the transformation kernel candidate index is 3 and the width and height of the current block are both 16 or less, at least one of the horizontal or vertical transformations of the transformation kernel candidate corresponding to the transformation kernel candidate index of 3 may be changed to another transformation kernel. In this case, the horizontal and vertical transformations may be changed independently of each other. If the difference (or the absolute value of the difference) between the intra-prediction mode value and the horizontal mode value of the current block is less than or equal to a predetermined threshold, the vertical transformation of the selected transformation kernel candidate may be changed to the IDT (identity transformation). If the difference (or the absolute value of the difference) between the intra-prediction mode value and the vertical mode value of the current block is less than or equal to a predetermined threshold, the horizontal transformation of the selected transformation kernel candidate may be changed to the IDT (identity transformation). Here, the threshold may be determined based on the width and height of the current block as shown in Table 7.

[0157] [Table 7]

[0158] Table 7 defines thresholds based on the size of the transformation block for changing the horizontal and / or vertical transformations of the transformation kernel candidate selected by the transformation kernel candidate index to other transformation kernels.

[0159] The six translation kernel candidates constituting a single MTS set may be distinguished by translation kernel candidate indices from 0 to 5, as defined in Table 5. These translation kernel candidate indices may be signaled by a bitstream. A flag indicating whether an MTS set is available / applicable (MTS enabled flag or MTS flag) may be signaled, and if the flag indicates that the MTS set is available / applicable, the translation kernel candidate index may be signaled. The MTS flag may consist of a single bin, and one or more CABAC-based entropy coding contexts (hereinafter referred to as CABAC contexts) may be assigned to that bin. For example, different CABAC contexts may be assigned for when MIP mode is not active and for when MIP mode is active.

[0160] The number of available transformation kernel candidates for the current block may be set differently depending on the context of the current block as described above. For example, the sum of the absolute values ​​of all or some of the transformation coefficients in the current block may be considered as the context of the current block. This sum of the absolute values ​​of the transformation coefficients is called AbsSum. If AbsSum is less than or equal to T1, only one transformation kernel candidate corresponding to the transformation kernel candidate index of 0 may be available. If AbsSum is greater than T1 and less than or equal to T2, four transformation kernel candidates corresponding to the transformation kernel candidate indexes from 0 to 3 may be available. If AbsSum is greater than T2, six transformation kernel candidates corresponding to the transformation kernel candidate indexes from 0 to 5 may be available. Here, T1 may be 6 and T2 may be 32, but this is just an example.

[0161] If AbsSum is less than or equal to T1, there is currently one available translation kernel candidate for the block, so the translation kernel candidate corresponding to the translation kernel candidate index of 0 may be set as the translation kernel for the block without signaling of the translation kernel candidate index. If AbsSum is greater than T1 and less than or equal to T2, there are four available translation kernel candidates, so one of the four translation kernel candidates may be selected based on the translation kernel candidate index having two bins. That is, the translation kernel candidate indices 0 to 3 may be signaled as 00, 01, 10, and 11, respectively. The MSB (Most Significant Bit) may be signaled first for the two bins, and the LSB (Least Significant Bit) may be signaled later. Different CABAC contexts may be assigned to each bin. For example, a different CABAC context than the one assigned for the MTS flag may be assigned to each of the two bins. Alternatively, a CABAC context may not be assigned to two bins, and bypass coding may be applied. When AbsSum is greater than T2, the candidate translation kernel index has values ​​from 0 to 5, so the candidate translation kernel index cannot be represented by only two bins. In this case, the candidate translation kernel index can be represented by assigning two or more bins, as in truncated binary coding. A CABAC context may be assigned to each bin assigned by the truncated binary coding scheme, or bypass coding may be applied without assigning a CABAC context. Alternatively, a CABAC context may be assigned to some of the bins (for example, the first bin, or the first and second bins), and bypass coding may be applied to the remaining bins.

[0162] Example 3

[0163] The current block's translation kernel may be determined based on a translation set containing one or more translation kernel candidates. The current block's translation kernel may be induced to be one of the one or more translation kernel candidates belonging to the translation set.

[0164] The process of determining the current block's translation kernel may include at least one of the following: 1) the process of determining the current block's translation set, or 2) the process of selecting one translation kernel candidate from the current block's translation set. The process of determining the translation set may be the process of selecting one of a plurality of translation sets that are identically predefined for the encoding device and the decoding device. Alternatively, the process of determining the translation set may be the process of configuring one or more translation sets that are available to the current block from a plurality of translation sets that are identically predefined for the encoding device and the decoding device, and selecting one of the configured translation sets. Alternatively, the process of determining the translation set may be the process of configuring one translation set based on the translation kernel candidate that is available to the current block from a plurality of translation kernel candidates that are identically predefined for the encoding device and the decoding device.

[0165] If the current block's translation set contains multiple translation kernel candidates, the process of selecting one of the multiple translation kernel candidates for the current block may be performed. However, if the current block's translation set contains only one translation kernel candidate (i.e., if there is only one translation kernel candidate available for the current block), the current block's translation kernel may be set to that translation kernel candidate.

[0166] The transformation set relating to this disclosure may mean the (non-separable) transformation set in Example 1 described above, or the MTS set in Example 2. Alternatively, the transformation set may be defined separately from the (non-separable) transformation set of Example 1 or the MTS set of Example 2. In this case, the transformation set may include one or more specific transformation kernels as candidate transformation kernels. A specific transformation kernel may be defined as a pair of transformation kernels for horizontal transformation and transformation kernels for vertical transformation, or as a single transformation kernel that is applied identically to both horizontal and vertical transformations.

[0167] In the embodiments of this disclosure, the NSPT application process, which is an unseparable transformation applied as a linear transformation, will be described in detail. The NSPT may be applied to all or part of the transformation block. Based on the forward NSPT, the residual samples present in the region to which the NSPT is applied may be input to a one-dimensional vector of the NSPT. That is, the residual samples present in the entire or a part of a transformation block (referred to in this disclosure as a Region of Interest, ROI) can be collected into a one-dimensional vector and configured as input. Subsequently, by applying the forward NSPT, linear transformation coefficients can be obtained. Conversely, by applying the reverse NSPT to the linear transformation coefficients, a one-dimensional vector output can be obtained. By placing each element value constituting the output vector at a fixed position inside the 2D transformation block, residual samples for the ROI can be obtained.

[0168] The dimension of the matrix of the non-separable transform kernel for NSPT may be determined by the size of the ROI. In this disclosure, the transform kernel may be referred to as the transform type or the transform matrix, and the non-separable transform kernel for NSPT may be referred to as the NSPT kernel. For example, if the current block is an MxN transform block, the ROI is the entire area of ​​the MxN transform block, and a square NSPT is applied, the dimension of the transform matrix may be MNxMN. For example, if the ROI is the entire area of ​​an 8x8 transform block, the dimension of the NSPT kernel may be 64x64.

[0169] According to one embodiment of the present disclosure, when NSPT is applied to a residual generated by intra-prediction, the NSPT kernel can be adaptively determined by the intra-prediction mode. Since the statistical properties of the residual block may differ for each intra-prediction mode, the compression efficiency can be improved by adaptively determining the NSPT kernel by the intra-prediction mode.

[0170] The NSPT kernel can be configured to share an application for one or more intra-prediction modes. As mentioned above, the non-separable transformation set may be determined based on the intra-prediction mode of the current block and the mapping table. The mapping table can define the mapping relationships between predefined intra-prediction modes and the non-separable transformation set. The predefined intra-prediction modes may include two non-directional modes and 65 directional modes.

[0171] As one embodiment, intra-prediction modes can be grouped into intra-prediction mode groups. An intra-prediction mode group may be assigned one NSPT kernel, or multiple NSPT kernels. In other words, an intra-prediction mode group may be assigned an inseparable transformation set (NSPT set) containing one or more NSPT kernels. An inseparable transformation set is mapped to an intra-prediction mode, and one of the N NSPT kernels included in the inseparable transformation set may be selected.

[0172] As an example, an intra-prediction group may include adjacent prediction modes (e.g., modes 17, 18, and 19). Furthermore, an intra-prediction group may include modes that are symmetrical. For example, directional modes may be symmetrical around the diagonal mode in Figure 5 (i.e., intra-prediction mode 34). In this case, two symmetrical modes can form a group (or pair). For example, modes 18 and 50 are symmetrical around mode 34 and can therefore be included in the same group. However, for symmetrical modes, an additional step may be taken to transpose the 2D input block before applying the forward NSPT kernel and then construct a one-dimensional input vector. For example, if the intra-prediction mode is 34 or less, a one-dimensional input vector can be derived from the input block in row-first order without transposing the 2D input block. If the intra-prediction mode is greater than 34, a one-dimensional input vector can be constructed by first transposing the 2D input block and then reading the input block in row-first order, or by leaving the 2D input block as is and reading the input block in column-first order.

[0173] Table 8 below illustrates an NSPT set assignment mapping table for intra-predictive mode. Referring to Table 8, a total of 35 NSPT sets, from 0 to 34, may be defined. Extended WAIP modes (i.e., modes -14 to -1 and modes 67 to 80 in Figure 5) may be assigned the NSPT set assigned to the nearest general directional mode. That is, extended WAIP modes may be assigned the second NSPT set.

[0174] [Table 8]

[0175] An NSPT set may contain one or more NSPT kernels (or kernel candidates). That is, an NSPT set may contain N NSPT kernel candidates. For example, N may be set to a value equal to or greater than 1, such as 1, 2, 3, 4, etc. Of the one or more NSPT kernels contained in the NSPT set, the kernel that applies to the current block can be signaled using an index. In this disclosure, such index may be referred to as an NSPT index. For example, an NSPT index may have values ​​of 0, 1, 2, ..., N-1.

[0176] Furthermore, as one embodiment, if there is only one candidate NSPT kernel, the NSPT index value may be fixed to 0. In this case, the NSPT index may be inferred without being separately signaled. In addition, a flag indicating whether or not NSPT is applied may be signaled separately from the NSPT index. In this disclosure, this flag may be referred to as the NSPT flag.

[0177] If the value of the NSPT flag is 1, NSPT may be applied. If the value of the NSPT flag is 0, NSPT may not be applied. If the NSPT flag is not signaled, the value of the NSPT flag may be inferred to be 0. As an example, if the value of the NSPT flag is 1, the NSPT index may be signaled. Based on the signaled NSPT index, one of the N kernel candidates included in the NSPT set selected by the intra-prediction mode may be identified.

[0178] In one embodiment, the entropy coding method for NSPT indices may be defined in various ways, taking into account the number (N) of NSPT kernels included in the NSPT set. For example, truncated unary binarization, truncated binary, or fixed-length binary may be used as methods for mapping values ​​from 0 to N-1 to binstrings (i.e., binary methods).

[0179] For example, if the number of kernel candidates constituting the NSPT set (N-value) is 2, one bin can identify one of the two candidates. For example, 0 could indicate the first candidate, and 1 could indicate the second candidate. Also, if the N-value is 3 and truncated unary binary evolution is applied, two bins may be used to identify the candidates. For example, the first, second, and third candidates can be signaled by binary evolution to 0, 10, and 11, respectively. In practice, the binary-evolved bins may be coded by context coding or bypass coding.

[0180] This disclosure describes a reduced primary transform (RPT) method using a reduced-dimensional transformation kernel as a primary transformation. As mentioned above, when forward NSPT is applied, samples belonging to a 2D residual block may be arranged (or rearranged) into a 1D vector according to row priority (or column priority). Then, a transformation matrix for NSPT may be multiplied by the arranged vector. When the 2D residual block is an MxN block (M is horizontal, N is vertical), the length of the rearranged 1D vector may be M*N. That is, the 2D residual block can be represented by a column vector having M*Nx1 dimensions. In this disclosure, M*N may be denoted as MN for convenience. In this case, the dimension of the transformation matrix may be MNxMN. In short, forward NSPT can operate in a way that obtains an MNx1 transformation coefficient vector by multiplying the left side of the MNx1 vector by the MNxMN transformation matrix.

[0181] When RPT is applied, instead of multiplying the forward NSPT transformation matrix by an MNxMN matrix as described above, r transformation coefficients can be obtained by multiplying by an rxMN matrix. Here, r represents the number of rows in the transformation matrix, and MN represents the number of columns in the transformation matrix. According to the embodiments of this disclosure, the value of r may be set to be less than or equal to MN. That is, an existing forward NSPT transformation matrix contains MN rows, each row being a 1xMN row vector, which is the transform basis vector of the NSPT transformation matrix. The transformation coefficients may be obtained by multiplying each transform basis vector by an MNx1 sample column vector.

[0182] An existing forward NSPT transformation matrix consists of MN row vectors, so applying forward NSPT can yield MN transformation coefficients (i.e., an MNx1 transformation coefficient column vector). On the other hand, in the case of forward RPT, the transformation matrix may be constructed with r transformation basis vectors instead of MN transformation basis vectors. This allows for obtaining r transformation coefficients (i.e., an rx1 transformation coefficient column vector) instead of MN when forward RPT is applied.

[0183] An RPT kernel can be constructed by selecting r transformation basis vectors, which are part of the transformation basis vectors that constitute an MNxMN forward NSPT kernel. In this disclosure, a transformation kernel may be referred to as a transformation type or transformation matrix, and an inseparable transformation kernel for NSPT may be referred to as an RPT kernel. That is, when selecting r 1xMN row vectors from an MNxMN forward NSPT kernel, it may be advantageous to select the transformation basis vectors that are most important in terms of coding performance. Specifically, in terms of energy compaction by transformation, greater energy may be concentrated in the transformation coefficients that appear earlier when multiplied by the forward NSPT transformation matrix. In other words, the higher the transformation basis vector is located on the forward NSPT transformation matrix, the greater the energy of the transformation coefficients that can be generated. Taking these points into consideration, an rxMN forward RPT kernel can be constructed (or induced) by taking r vectors from the top of the forward NSPT kernel.

[0184] The RPT relating to this disclosure takes only a portion (i.e., r) of the conversion coefficients obtained by applying the existing NSPT, which may result in a loss of some of the energy inherent in the signal. In other words, this process may cause distortion between the original signal and the original signal. Nevertheless, by applying RPT, only r conversion coefficients are generated instead of MN, thus reducing the number of bits required to code these conversion coefficients. Therefore, in the case of signals where a large amount of energy is concentrated in a small number of conversion coefficients (e.g., image resistive signals), the gain obtained by reducing the signaling bits is significantly large, thereby improving coding performance.

[0185] The inverse NSPT is a transformation matrix, which may be the transpose of the forward NSPT kernel described above. In this case, the input data may be a transformation coefficient signal instead of a sample signal such as a residual signal. Specifically, if the forward NSPT transformation matrix is ​​G and the sample signal rearranged into a 1D vector is x, the transformation coefficient vector obtained by multiplying the transformation matrix to the left may be expressed as shown in equation 4 below.

[0186] [Equation 4]

number

[0187] Referring to Equation 4, x and y may be MNx1 column vectors. G may have the form of an MNxMN matrix. The inverse NSPT process may be expressed using the same variables as in Equation 5 below.

[0188] [Formula 5]

number

[0189] In formula 5, G Trepresents the transpose matrix of G. The forward and reverse RPT operations according to this disclosure may also be expressed by the two formulas above. However, when RPT is applied, y is an rx1 column vector instead of an MNx1 column vector, and G is an rxMN matrix instead of an MMxMN matrix. That is, applying RPT instead of NSPT does not change the dimension of the sample signal (e.g., image residual signal), but this means that the original number of sample signals (i.e., MN sample signals) can be restored using only r transformation coefficients by reverse RPT. In other words, the original MN sample signals can be restored by coding only r transformation coefficients, which is fewer than MN, resulting in improved coding performance.

[0190] In one embodiment of this disclosure, we propose an RPT structure that defines an r-value considering the statistical properties of a residual block and derives a residual block of the existing transformed block size from a residual block of reduced size determined by the defined r-value. If an additional transformation (i.e., a quadratic transformation) is applied to predict the statistical distribution of the linear transformation coefficients, a quantization process is applied to the linear transformation coefficients, so that quantized non-zero coefficients may be concentrated in the relatively low frequency domain. Thus, a reduced secondary transform for the statistical distribution of the linear transformation coefficients can relatively easily define the statistical properties of the linear transformation coefficients in the form of setting an r-value for a given low frequency domain. However, the RPT according to this disclosure is a technique for defining an r-value considering the statistical properties of a sample in a residual block, which have properties very different from the distribution of the linear transformation coefficients, and thus has a fundamental difference from a reduced secondary transform. Below, we describe various embodiments for determining the RPT kernel, which is a transformation matrix of reduced dimensions. In other words, we describe below a method for determining or defining the r-value in RPT.

[0191] In one embodiment of this disclosure, the r value in RPT can be determined by considering the worst-case complexity tolerable by the conversion system. In one embodiment, the worst-case complexity can be calculated based on the number of multiplications per sample. To apply RPT in both the forward and reverse directions based on an MxN block, MN*r multiplications are required. Since a 2D block consists of a total of MN samples, the number of multiplications per sample can be calculated as (MN*r) / MN=r. Therefore, the r value can be configured to be kept below the maximum allowable number of multiplications per sample. For example, if the maximum possible number of multiplications per sample for a 16x16 block is set to 16, the r value may be determined to be 16 or less. That is, the forward RPT kernel may be set to 16x256.

[0192] In other embodiments, memory usage can be considered as a measure of worst-case complexity. For example, the allowable memory size per kernel can be set. For instance, if p bytes are required for each kernel coefficient (in this disclosure, each element constituting the translation kernel is referred to as a kernel coefficient), and the memory usage is set to be q bytes or less per kernel, then the r value may be set to q / (MN*p) or less. For example, if p is 1 byte for a forward RPT kernel for a 16x16 block, and the memory usage is set to be 8KB or less per kernel (q=8KB=2 13 (Bytes), the r value may be set to 32 or less.

[0193] Another example is to consider memory usage and / or the multiplication per sample as measures of worst-case complexity. For example, if the maximum possible multiplication per sample for a 16x16 block is set to 16 and the memory usage is set to 8KB or less per kernel (kernel coefficients are represented by 1 byte), then the r value may be set to 16 or less.

[0194] Furthermore, in one embodiment, the r value constituting the RPT kernel may be determined by specific information. In other words, the r value constituting the RPT kernel may be determined based on predefined coding parameters. For example, the r value may be determined by the size of the block. In other words, the RPT kernel may be variably determined by the size of the block. Here, the block may be at least one of a coding block, a transformation block, and a prediction block. Also, for example, the r value may be determined based on prediction information. Here, the prediction information may include information about inter / intra prediction, intra prediction mode information, etc. Also, for example, the r value may be determined based on signaled information (values ​​of syntax elements). For example, the r value may be variably determined by quantization parameter values. Also, in terms of complexity improvement, a predefined fixed value may be used as the r value, and the predefined fixed value may be determined based on signaled information.

[0195] Applying the RPT kernel rxMN to a sampled signal yields r conversion coefficients. These r conversion coefficients may be arranged according to a predefined scan order of conversion coefficients (e.g., forward / reverse zig-zag scan order, forward / reverse horizontal scan order, forward / reverse vertical scan order, forward / reverse diagonal scan order, scan order determined based on an intra-prediction mode, etc.). When arranging the conversion coefficients obtained from forward RPT application according to such a scan order (e.g., a coefficient group (CG) unit scan order is also applicable), if the r value is smaller than MN, it may not be necessary to fill the entire MxN block with r conversion coefficients, resulting in empty space. In an embodiment of this disclosure, the aforementioned empty space may be predicted in the following way, taking into account the characteristics of the residual signal.

[0196] - The values ​​of the available surrounding pixels can be used to fill in the values ​​in the empty space.

[0197] - The values ​​of the empty spaces can be filled in based on the values ​​of the available surrounding pixels and the intra-prediction mode. For example, the values ​​of the empty spaces can be predicted by performing intra-prediction based on the values ​​of the available surrounding pixels and the intra-prediction mode.

[0198] - The empty spaces can be filled with values ​​using predefined, fixed values ​​(for example, 0).

[0199] - The empty space can be filled with values ​​from available peripheral pixels using a predetermined intra-prediction mode (e.g., planar mode).

[0200] In this disclosure, filling empty spaces with zeros, as described above, may be referred to as the zero-out process. When filling empty spaces with zeros, the following embodiments may be applied: If a non-zero conversion coefficient is detected (or parsed) in the relevant empty space portion during the passing of conversion coefficients on the decoding device side, it can be considered (or inferred) that RPT will not be applied. In other words, if a non-zero conversion coefficient exists in a predefined region representing the relevant empty space, it can be considered that RPT will not be applied. In this case, signaling (or parsing) is not performed for a flag indicating whether or not RPT is applied and / or for an index specifying one of several RPT kernel candidates. As an example, if a non-zero conversion coefficient exists in a predefined region representing the relevant empty space, a predefined variable value may be updated, and based on the updated variable value, it can be inferred that RPT will not be applied.

[0201] In one embodiment of this disclosure, the application of RPT may be determined by the size and / or shape of the block. Furthermore, the RPT kernel may be variably determined by the size and / or shape of the block. Since the r value may differ depending on the size and / or shape of the block (i.e., for each MxN block), the available space may vary depending on the size and / or shape of the block. Consequently, the region for checking whether a non-zero conversion coefficient is detected may be defined differently for each block size and / or shape. In other words, the zero-out region may be variably determined.

[0202] For example, when a 16x64 matrix is ​​applied as the forward RPT matrix for an 8x8 block, the r value may be 16. In this case, when the CG is a 4x4 subblock, a non-zero RPT conversion coefficient may be filled only in the top-left 4x4 block, and the remaining three 4x4 subblocks (i.e., the top-right, bottom-left, and bottom-right subblocks) may be filled with zero values. If a non-zero conversion coefficient is detected in these remaining three 4x4 subblock regions during the decoding process, it can be considered that RPT is not applied. Furthermore, as mentioned above, a flag indicating whether or not RPT is applied, or an index specifying one of several RPT kernel candidates, does not need to be signaled.

[0203] As an example, when a 32x128 matrix is ​​applied as the forward RPT matrix for a 16x8 block (i.e., the r value is 32), and the CG is a 4x4 subblock, then, in terms of scan order, non-zero RPT conversion coefficients may be filled into only two CGs. For example, these RPT conversion coefficients may be filled into the upper-left 4x4 subblock and the 4x4 subblock adjacent to the lower side of the upper-left subblock. The area to be filled with zeros as empty space may be determined to be the remaining area other than those two 4x4 subblocks. The RPT kernel may be determined variably depending on the size and / or shape of the block, and as described, the empty space may be determined differently for 8x8 blocks and 16x8 blocks.

[0204] As an example, when the r value is a multiple of the CG size and the conversion coefficients are scanned in units of CG, if it is detected that a non-zero conversion coefficient exists in a CG belonging to an empty space, the flag and / or index associated with RPT does not need to be signaled. That is, the internal conversion coefficients of each CG can be scanned in a specified order, and the system can move to the next CG according to the scan order for each CG unit and scan the internal conversion coefficients of the CG identically. In conventional image compression techniques, a flag indicating whether or not a non-zero conversion coefficient exists within each CG is signaled in advance, so the application of RPT can be determined based solely on this information, thereby reducing signaling overhead and decreasing the complexity of associated implementation.

[0205] As mentioned above, if a non-zero transformation coefficient is detected in the empty space region that would normally be filled with zeros when RPT is applied, RPT does not need to be applied. In this case, signaling for information related to RPT may be omitted. However, if no non-zero transformation coefficient is detected in the empty space region, it is not possible to determine whether RPT is applied or not. Therefore, after parsing (or signaling) the related transformation coefficients, a flag indicating whether RPT is applied or not can be parsed to make a final decision on whether RPT is applied or not.

[0206] As one embodiment, a forward quadratic transformation can be further applied to the transformation coefficients generated by the application of RPT. Alternatively, a forward quadratic transformation can be further applied to the region in an MxN block where the generated transformation coefficients are located. In this disclosure, such region or a part thereof may be referred to as an ROI from the perspective of the forward quadratic transformation. For the reverse direction, a reverse quadratic transformation can be applied first, followed by a reverse RPT. Specifically, a region or a part thereof where the r transformation coefficients generated by the application of forward RPT are located can be set as an ROI, and a forward quadratic transformation can be applied to it. In this case, when a 16x64 forward RPT transformation matrix is ​​applied to an 8x8 region, the 16 generated transformation coefficients may be located in a 4x4 sub-block at the top left, and this sub-block region can be set as an ROI, and a forward quadratic transformation can be applied to this ROI.

[0207] Furthermore, the coefficient values ​​of the RPT kernel may be adjusted to accommodate operations such as integer arithmetic or fixed-point arithmetic. That is, the RPT kernel can be configured to appropriately scale the kernel coefficients belonging to the kernel that are not theoretical orthogonal or non-orthogonal transformations (where orthogonal and non-orthogonal transformations represent transformations in which the norm of each transformation basis vector is 1) so that the transformation is performed by integer arithmetic (or fixed-point arithmetic) in the actual codec system. The scaling factor applied when applying a deseparated transformation in existing image compression techniques can be similarly reflected when applying RPT. In this case, deseparated or non-deseparated transformations (including RPT) can be performed while maintaining other processes other than the transformation (e.g., quantization, de-quantization).

[0208] By multiplying the transformation basis vector by the scaling value described above, the integerized coefficients of the RPT kernel can be obtained. In practice, multiplying by the scaling value may include applying operations such as rounding, truncation, and rounding up to each kernel coefficient. That is, the integerized RPT kernel obtained by the method described above is defined and may be used in the transformation / inverse transformation process. As described above, by obtaining the scaled integer kernel coefficients through operations such as rounding, truncation, and rounding up, the maximum and minimum values ​​can be found for all kernel coefficients, and from these maximum and minimum values, a number of bits sufficient to represent all kernel coefficients can be obtained. For example, if the maximum value is 127 or less and the minimum value is -128 or greater, all integer kernel coefficients can be represented with 8 bits (especially by two's complement representation, etc.).

[0209] In general, if the maximum value is (2 (N-1) -1) or less, and the minimum value is -2 (N-1) If the above is true, then all integer kernel coefficients can be represented with N bits. For example, if the maximum value is (2 (N-1) -1) greater than or with a minimum value of -2 (N-1)If the range is smaller, it may not be possible to represent all integer kernel coefficients with N bits. In such cases, 1) all kernel coefficients can be further scaled to fit within the N-bit range, or 2) the number of bits required to represent the kernel coefficients can be increased (i.e., N+1 bits or more). For example, if all kernel coefficients are multiplied by a scaling value to represent them with N bits, -p If only multiplication is needed (p>=1), then, to integrate with the existing encoding / decoding process, 2 p It can be compensated by multiplying only by 2. p Multiplying by only this can be implemented by methods such as performing a further shift operation of p bits to the left, or by reducing the amount of right-side shift applied in the quantization or dequantization process by p.

[0210] Using the methods described above, all kernel coefficients can be represented using 8 bits, 9 bits, 10 bits, etc. Of course, the block size and the scaling value of the kernel coefficients can be set differently for each kernel, and the number of bits used to represent the kernel coefficients can also be set differently.

[0211] The aforementioned NSPT may be applied based on at least one of the current block size, tree type, or component type. For example, the decision to apply NSPT may be made based on at least one of the current block size, tree type, or component type. An NSPT index may be signaled based on at least one of the current block size, tree type, or component type. An NSPT set or NSPT kernel may be induced based on at least one of the current block size, tree type, or component type.

[0212] The allowed transform block sizes predefined for the decoding device can be broadly divided into two groups. Either of the two groups (hereinafter referred to as the first group) can represent the set of block sizes to which NSPT can be applied. The first group may consist of any one of the allowed transform block sizes, or it may consist of two or more block sizes from the allowed block sizes. A block size to which NSPT can be applied may be defined as a block size in which at least one of the width and height is less than or equal to a predetermined threshold. Alternatively, a block size to which NSPT can be applied may be defined as a block size in which the product of the width and height is less than or equal to a predetermined threshold. Alternatively, a block size to which NSPT can be applied may be defined as a block size in which the maximum value of the width and height is less than or equal to a predetermined threshold. The threshold may be an integer of 4, 8, 16, 32, 64, 128 or higher.

[0213] The other of the two groups mentioned above (hereinafter referred to as the second group) can represent the set of block sizes to which NSPT does not apply. The aforementioned separable linear transformation may be applied to the block sizes belonging to the second group. In addition, an unseparable quadratic transformation may be applied to all or part of the block sizes belonging to the second group.

[0214] For example, if the current block size belongs to the first group, the inverse NSPT may be applied to the (inversely quantized) transformation coefficients of the current block. If the current block size belongs to the second group, the inverse separated linear transformation may be applied to the (inversely quantized) transformation coefficients of the current block. Alternatively, if the current block size belongs to the second group, the inverse non-separable quadratic transformation (e.g., LFNST (low frequency non-separable transform)) may be applied first to the (inversely quantized) transformation coefficients of the current block, and then the inverse separated linear transformation (e.g., DCT-2) may be applied to the resulting transformation coefficients.

[0215] As an example, the first group, which is the set of block sizes to which NSPT can be applied, may be defined as the sets of 4x4, 4x8, 8x4, and 8x8. Or, the first group may be defined as the sets of 4x8, 8x4, and 8x8. Or, the first group may be defined as the sets of 4x8 and 8x4. Or, the first group may be defined as the sets of 4x4, 4x8, 4x16, 8x4, 8x8, and 16x4. Or, the first group may be defined as the sets of 4x8, 4x16, 8x4, 8x8, and 16x4. Or, the first group may be defined as the sets of 4x8, 4x16, 8x4, and 16x4. Or, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, and 16x16. Alternatively, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8. Alternatively, the first group may be defined as the sets of 4x8, 8x4, 8x8, 8x16, and 16x8. Alternatively, the first group may be defined as the sets of 4x8, 8x4, 8x16, and 16x8. Alternatively, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, 32x16, and 32x32. Alternatively, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 8x4, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 8x4, 8x16, 16x8, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, and 16x4. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x16, 16x4, and 16x8.Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x16, 8x32, 16x4, 16x8, 32x4, and 32x8. Alternatively, the first group may be defined as the set of 4x8, 4x16, 4x32, 8x4, 8x16, 8x32, 16x4, 16x8, 32x4, and 32x8. The first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x16, 32x4, and 32x8. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x32, 32x4, 32x8, and 32x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x16, 16x32, 32x4, 32x8, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 4x16, 4x32, 8x4, 8x16, 8x32, 16x4, 16x8, 16x32, 32x4, 32x8, and 32x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x16, 16x32, 32x4, 32x8, 32x16, and 32x32.

[0216] For block sizes belonging to the first group, an NSPT matrix (or NSPT kernel) with a predetermined dimension may be applied. Here, the NSPT matrix is ​​an inverse transformation matrix and may be expressed as a matrix with dimension PxQ, where PxQ represents a matrix with P and Q rows and columns, respectively.

[0217] For example, a 16x16 NSPT matrix may be applied to a 4x4 block. A 32x20 NSPT matrix may be applied to at least one of a 4x8 or 8x4 block. A 64x24 NSPT matrix may be applied to at least one of a 4x16 or 16x4 block. A 64x32 NSPT matrix may be applied to an 8x8 block. A 128x40 NSPT matrix may be applied to at least one of an 8x16 or 16x8 block. A 256x44 NSPT matrix may be applied to a 16x16 block. A 128x36, 128x38, or 128x40 NSPT matrix may be applied to a 4x32 or 32x4 block. A 256x48 NSPT matrix may be applied to an 8x32 or 32x8 block. For 16x32 or 32x16 blocks, a 512x52 or 512x54 NSPT matrix may be applied.

[0218] For at least one of the 4x32 or 32x4 blocks, an NSPT matrix of 128x36, 128x38, 128x40, or 128x(36-(4*n)) may be applied. Alternatively, for at least one of the 4x32 or 32x4 blocks, an NSPT matrix of 128x(36-(4*n)) may be applied instead of an NSPT matrix of 128x36, 128x38, or 128x40. Here, * represents multiplication, and n may be an integer greater than or equal to 0.

[0219] As an example, at least one of the following matrices may be used as the NSPT matrix for at least one of the 4x32 or 32x4 blocks: a 128x36 matrix, a 128x32 matrix, a 128x28 matrix, a 128x24 matrix, a 128x20 matrix, a 128x16 matrix, a 128x12 matrix, a 128x8 matrix, or a 128x4 matrix.

[0220] Additionally, a 256x48 or 256x(48-(4*m)) NSPT matrix may be applied to at least one of the 8x32 or 32x8 blocks. Alternatively, a 256x(48-(4*m)) NSPT matrix may be applied to at least one of the 8x32 or 32x8 blocks instead of a 256x48 NSPT matrix. Here, * represents multiplication, and m can mean an integer greater than or equal to 0.

[0221] As an example, at least one of the following matrices may be used as the NSPT matrix for at least one of the 8x32 or 32x8 blocks: 256x48, 256x44, 256x40, 256x36, 256x32, 256x28, 256x24, 256x20, 256x16, 256x12, 256x8, or 256x4.

[0222] The combination of NSPT matrix structures for 4x32 and 32x4 blocks and NSPT matrix structures for 8x32 and 32x8 blocks may be constructed using the matrix combinations described above. That is, the combination of NSPT matrix structures for 4x32 and 32x4 blocks and NSPT matrix structures for 8x32 and 32x8 blocks may be defined as a combination of one or more 128x(36-(4*n)) NSPT matrices and one or more 256x(48-(4*m)) NSPT matrices. Here, n may be one or more integers in the range of 0 to 8, and m may be one or more integers in the range of 0 to 11.

[0223] For example, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x24 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x36 matrix.

[0224] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x24 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x32 matrix.

[0225] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x24 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x28 matrix.

[0226] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x24 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x24 matrix.

[0227] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x24 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x20 matrix.

[0228] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x24 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x16 matrix.

[0229] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x20 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x36 matrix.

[0230] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x20 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x32 matrix.

[0231] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x20 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x28 matrix.

[0232] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x20 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x24 matrix.

[0233] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x20 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x20 matrix.

[0234] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x20 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x16 matrix.

[0235] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x16 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x36 matrix.

[0236] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x16 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x32 matrix.

[0237] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x16 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x28 matrix.

[0238] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x16 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x24 matrix.

[0239] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x16 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x20 matrix.

[0240] Alternatively, the NSPT matrix for at least one of the 4x32 or 32x4 blocks may be a 128x16 matrix, and the NSPT matrix for at least one of the 8x32 or 32x8 blocks may be a 256x16 matrix.

[0241] Since the PxQ matrix is ​​an inverse NSPT matrix, the Px1 output vector can be obtained by applying the PxQ matrix to the Qx1 input vector (i.e., (PxQ matrix) x (Qx1 input vector)). Here, the Qx1 input vector may correspond to the inversely quantized transformation coefficients within the current block to which NSPT is applied. In this case, the value of Q can represent the number of transformation coefficients to which NSPT is applied, and may be less than or equal to the product of the width and height of the current block. The value of Q may be variably determined based on the size of the current block from among the block sizes belonging to the first group described above. Alternatively, the value of Q may be set to be the same for all block sizes belonging to the first group. The Px1 output vector may correspond to the residual signal (or the decoded residual sample). The value of P may be equal to the product of the width and height of the current block.

[0242] Conversely, the forward NSPT matrix may be expressed as the QxP matrix, which is the transpose of the PxQ matrix. The Qx1 output vector can be obtained by applying the QxP matrix to the Px1 input vector (i.e., (QxP matrix) x (Px1 input vector)). Here, the Px1 input vector may correspond to the residual sample in the current block to which NSPT is applied. The value of P may be the same as the product of the width and height of the current block. The Qx1 output vector may correspond to the transformation coefficients in the current block induced by NSPT. In this case, the value of Q can represent the number of transformation coefficients output by NSPT, and may be less than or the same as the product of the width and height of the current block. Similarly, the value of Q may be variably determined based on the size of the current block from among the block sizes belonging to the first group described above. Alternatively, the value of Q may be set to be the same for all block sizes belonging to the first group.

[0243] As in the example above, NSPT may be applied to non-square blocks such as MxN and NxM blocks. For example, NSPT may be applied to 4x8 and 8x4 blocks. Alternatively, NSPT may be applied to 4x16 and 16x4 blocks, or to 8x16 and 16x8 blocks, or to 16x32 and 32x16 blocks.

[0244] By applying NSPT to a specific block size belonging to the first group described above, the transformation can be performed with greater accuracy, potentially improving coding performance. When forward LFNST is applied, the linearly transformed transformation coefficients of the remaining region outside the region to which LFNST is applied (i.e., Region-Of-Interest, ROI) may be zeroed out. Furthermore, LFNST may consist of a small number of transform basis vectors. In such cases, applying a separable linear transformation such as DCT-2 and an unseparable quadratic transformation such as LFNST instead of NSPT to the block size may result in performance degradation. Applying NSPT instead of LFNST in such cases omits the zero-out process, potentially improving coding performance compared to applying LFNST. Performance improvement can also be expected depending on the method of applying NSPT. NSPT or LFNST may be applied using symmetry, which will be described later. Here, LFNST uses symmetry only with respect to the ROI region to perform a transpose operation on the input block. In contrast, NSPT performs transpose operations using symmetry across the entire block. Therefore, in NSPT, it becomes possible to train and apply a more precise symmetry method to the NSPT kernel, which is expected to improve performance.

[0245] Furthermore, when applying LFNST instead of NSPT to an 8x8 block, a 32x64 transformation matrix can be applied instead of a 16x64 transformation matrix from the perspective of forward transformation. Here, the 16x64 transformation matrix may be constructed by sampling the top 16 rows of the 32x64 transformation matrix. When applying LFNST based on a 16x64 transformation matrix to an 8x8 block, 16 multiplications are required per sample to apply LFNST, but when using a 32x64 transformation matrix, 32 multiplications are required per sample to apply LFNST. However, when using a 32x64 transformation matrix in this way, an improvement in coding performance can be expected.

[0246] If the current block's tree type is single tree, NSPT can be applied to the rumor component of the current block, but not to the chroma component. If the current block's tree type is dual tree, NSPT can be applied to both the rumor and chroma components of the current block.

[0247] Alternatively, regardless of whether the current block's tree type is single-tree or not, NSPT may be applied to the rumor component of the current block, but not to the chroma component. Alternatively, regardless of whether the current block's tree type is single-tree or not, NSPT may be applied to both the rumor and chroma components of the current block.

[0248] For example, if the current block's tree type is a single tree, NSPT is permitted for the lumern and chroma components, and the current block's size belongs to the first group, then one NSPT index may be signaled, and the lumern and chroma components of the current block can share this NSPT index. Here, the NSPT index may be an index for selecting one of the candidate transformation kernels for NSPT. If the sizes of the lumern and chroma blocks of the current block belong to the first group, then the candidate transformation kernel selected by the same NSPT index may be applied to the lumern and chroma components. If the current block's tree type is a single tree and NSPT is applied only to the lumern component, then LFNST may not be applied to the chroma component of the current block, and a separate transformation may be applied instead. Alternatively, if the current block's tree type is a single tree and NSPT is applied only to the lumern component, then LFNST may be applied to the chroma component of the current block.

[0249] In the case of a single tree, the rumor and chroma components may have a high degree of correlation. In this case, applying NSPT only to the rumor component, or applying a conversion kernel candidate selected by a single NSPT index to both the rumor and chroma components, can reduce unnecessary signaling and improve compression efficiency. On the other hand, in the case of a non-single tree, the rumor and chroma components each have their own partitioning and encoding structures. In this case, signaling NSPT indices for each component can reflect the characteristics of each component and improve compression efficiency.

[0250] The NSPT kernel for the aforementioned NSPT may be induced based on at least one of the symmetries between intra-prediction modes or between block forms. For example, the NSPT kernel may be induced as an NSPT kernel corresponding to at least one of the modes symmetric to the intra-prediction mode of the current block, or a block form symmetric to the block form of the current block. Alternatively, the NSPT kernel may be induced based on an NSPT set containing one or more NSPT kernel candidates, where the NSPT set may be induced as an NSPT set corresponding to at least one of the modes symmetric to the intra-prediction mode of the current block, or a block form symmetric to the block form of the current block. One or more NSPT kernel candidates belonging to the NSPT set may be set as the NSPT kernel for the current block. For this purpose, an NSPT index may be used to identify one or more NSPT kernel candidates belonging to the NSPT set. The NSPT index may be signaled in a bitstream and may be induced based on the aforementioned symmetries.

[0251] Symmetry may exist between at least two of the intra-prediction modes predefined in the decoding device. For the sake of explanation, symmetry centered on the upper left diagonal mode (i.e., mode 34) will be described below. Referring to Figure 5, symmetry exists between the directional modes. All modes except Planar mode (mode 0) and DC mode (mode 1) have a prediction direction. Modes 2 through 6 can be named normal directional modes (can be written as [2,66]), and modes -14 through -1 (can be written as [-14,-1]) and modes 67 through 80 (can be written as [67,80]) can be named wide directional modes. Wide directional modes may include at least one mode with a value less than -14 or a mode with a value greater than 80. Referring to Figure 5, all modes except mode 0 and mode 1 are symmetrical with respect to mode 34. Specifically, with respect to the [2,66] mode, the x-th mode and the (68-x)-th mode are symmetric, and between the [-14,-1] mode and the [67,80] mode, the x-th mode and the (66-x)-th mode are symmetric. A similar symmetry relationship may hold between the [N,-1] mode and the [67,66-N] mode. Here, N may be an integer less than or equal to -14.

[0252] On the other hand, in relation to the symmetry between block configurations, MxN blocks and NxM blocks may be defined as blocks that are symmetrical with each other. Here, M and N may be the same or different. Alternatively, if the ratio of the width to the height of an M1xN1 block (M1 / N1) and the ratio of the height to the width of an M2xN2 block (N2 / M2) are the same, then M1xN1 blocks and M2xN2 blocks may be defined as blocks that are symmetrical with each other. Alternatively, if the ratio of the width to the height of an M1xN1 block (M1 / N1) and the ratio of the width to the height of an M2xN2 block (M2 / N2) are the same, then M1xN1 blocks and M2xN2 blocks may be defined as blocks that are symmetrical with each other.

[0253] In a square block, modes that are symmetric to each other may share at least one of the following: an NSPT set, an NSPT index, or an NSPT kernel. That is, at least one of the NSPT sets, NSPT indexes, or NSPT kernels for one of the symmetric modes may be applied identically to the other of the symmetric modes.

[0254] As an example, modes that are symmetric to each other may share a single NSPT kernel. However, for one of the symmetric modes, the NSPT kernel can be applied to the input data, while for the other, the NSPT kernel can be applied after the input data has been transposed. Specifically, if mode x belongs to the [2,33] mode, for mode x, a one-dimensional vector (1D vector) can be constructed from the input data MxM block in a row-first order, and the NSPT kernel can be applied to this one-dimensional vector. Here, constructing a one-dimensional vector in a row-first order may be done by reading the input data from the input data MxM block column by column to obtain M columns, and then arranging these sequentially to construct a one-dimensional vector. On the other hand, for mode (68-x), which is symmetric to mode x, a one-dimensional vector can be constructed in a column-first order, and the same NSPT kernel can be applied to this one-dimensional vector. Here, the construction of a one-dimensional vector following row-first order may be done by reading the input data row by row from the input data MxM block, obtaining M rows, and sequentially arranging these to construct a one-dimensional vector. When mode x belongs to the [N,-1] mode (N≦-14), a one-dimensional vector can be constructed for the (66-x) mode, which is symmetric to mode x, following a column-first order, and the same NSPT kernel as for mode x can be applied to this one-dimensional vector. For modes 0 and 1, a column-first order or row-first order can be applied, and for modes 34, a column-first order or row-first order can also be applied. Furthermore, for intra-prediction modes belonging to the [2,33] mode, a row-first order can be applied, and for modes symmetric to that intra-prediction mode, a column-first order can be applied. For intra-prediction modes belonging to the [N,-1] mode, a row-first order can be applied, and for modes symmetric to it, a column-first order can be applied.

[0255] In the case of non-square blocks, in addition to the symmetry between intra-predictive modes, symmetry between block shapes may also be considered. A non-square block with widths and heights of M and N, respectively, can be considered to have a symmetric relationship with a non-square block with widths and heights of N and M, respectively. For example, in the [2,66] mode, symmetry may exist between the x-th mode of an MxN block and the (68-x)-th mode of an NxM block. Similarly, if the x-th mode of an MxN block belongs to the [N,-1] mode (N ≤ -14), symmetry may exist between the x-th mode of an MxN block and the (66-x)-th mode of an NxM block.

[0256] The method for constructing a one-dimensional vector from an input data block is as described above. That is, if column-major ordering is applied to mode x, row-major ordering may be applied to the mode symmetric to it. Or, if row-major ordering is applied to mode x, column-major ordering may be applied to the mode symmetric to it. Specifically, when column-major ordering is applied to mode x, input data can be read column by column from the input data MxN block to obtain M columns, and these can be arranged sequentially to construct a one-dimensional vector. Here, each column may have a length of N. For modes symmetric to mode x, input data can be read row by row from the input data MxN block to obtain N rows, and these can be arranged sequentially to construct a one-dimensional vector. Here, each row may have a length of M. Or, when row-major ordering is applied to mode x, input data can be read row by row from the input data MxN block to obtain N rows, and these can be arranged sequentially to construct a one-dimensional vector. Here, each row may have a length of M. For modes symmetric to mode x, we can read the input data column by column from the MxN block of input data to obtain M columns, and then sequentially arrange these to construct a one-dimensional vector. Here, each column may have a length of N.

[0257] The current block is an MxN block having the x-th mode. When the above-described symmetry is utilized for the current block, the NSPT set and / or the NSPT kernel of the current block may be determined based on at least one of an intra prediction mode symmetric to the x-th mode or an NxM block size symmetric to the MxN block size. Here, the NSPT kernel may be set as an NSPT kernel for an NxM block rather than an NSPT kernel for an MxN block. That is, when symmetry is utilized for the current block, the NSPT set and / or the NSPT kernel for a block having symmetry with the current block may be identically utilized. As described above, a one-dimensional vector can be constructed from the input data block according to a predetermined priority order, which may correspond to the input of the NSPT kernel.

[0258] Also, it may be restricted that the symmetry is utilized only when the value of the intra prediction mode of the current block is greater than 34. That is, when the value of the intra prediction mode of the current block is greater than 34, a transposition operation may be applied when constructing a one-dimensional vector from the input data block, and an NSPT set or an NSPT kernel corresponding to a mode and / or a block form having symmetry with the current block may be utilized. Specifically, when the intra prediction mode of the current block belongs to the [N, -1] mode and the [2, 34] mode, symmetry may not be utilized for the current block. On the other hand, when the intra prediction mode of the current block belongs to the [35, 66] mode and the [67, 66 - N] mode, symmetry may be utilized for the current block. Here, N may be an integer less than or equal to -14.

[0259] The derivation of the NSPT set or the NSPT kernel based on the symmetry may be adaptively performed based on the size of the current block. As an example, for a 4x4 block and an 8x8 block, the NSPT set or the NSPT kernel may be derived based on symmetry, and for a 4x8 block and an 8x4 block, the NSPT set or the NSPT kernel may not be derived based on symmetry.

[0260] Depending on whether the symmetry is utilized or not, the number of available NSPT sets may be different from each other. As an example, when the symmetry is utilized, the number of available NSPT sets may be 35, and when the symmetry is not utilized, the number of available NSPT sets may be 67.

[0261] Table 9 below relates to an example in which the NSPT set is determined using symmetry, and shows the mapping relationship between the intra prediction mode and the NSPT set when the number of available NSPT sets is 35.

[0262] [Table 9]

[0263] Referring to Table 9, if the value (X) of the current block's intra-prediction mode is less than 0, the current block's NSPT set may be determined to be the NSPT set with NSPT set index 2 out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than or equal to 0 and less than or equal to 34, the current block's NSPT set may be determined to be the NSPT set with NSPT set index X out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than or equal to 35 and less than or equal to 66, the current block's NSPT set may be determined to be the NSPT set with NSPT set index (68-X) out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than or equal to 35 and less than or equal to 66, the current block's NSPT set may be the same as the NSPT set corresponding to the value (68-X) of the mode symmetric to the current block's intra-prediction mode. Similarly, if the value (X) of the current block's intra-prediction mode is greater than 66, the current block's NSPT set may be determined to be an NSPT set with 2 NSPT set indices out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than 66, the current block's NSPT set may be identical to the NSPT set corresponding to the mode symmetric to the current block's intra-prediction mode.

[0264] Table 10 below shows an example where the NSPT set is determined without using symmetry, illustrating the mapping relationship between the intra-prediction mode and the NSPT set when there are 67 available NSPT sets.

[0265] [Table 10]

[0266] Referring to Table 10, if the value of the current block's intra-prediction mode (X) is less than 0, the current block's NSPT set may be determined to be an NSPT set with 2 NSPT set indices out of 67 NSPT sets. If the value of the current block's intra-prediction mode (X) is greater than or equal to 0 and less than or equal to 66, the current block's NSPT set may be determined to be an NSPT set with X NSPT set indices out of 67 NSPT sets. If the value of the current block's intra-prediction mode (X) is greater than 66, the current block's NSPT set may be determined to be an NSPT set with 66 NSPT set indices out of 67 NSPT sets.

[0267] By utilizing the aforementioned symmetry, it is possible to save the memory size required to store the transformation kernel while maintaining the performance achieved by applying the transformation. For example, by using 35 NSPT sets instead of 67 NSPT sets by utilizing symmetry, the memory size required to store the NSPT kernel can be significantly reduced.

[0268] The number of available NSPT sets and / or the number of NSPT kernel candidates belonging to an NSPT set may differ depending on the block size. For example, there may be 35 available NSPT sets for a 4x4 block, 19 available NSPT sets for a 4x8 and 8x4 block, and 10 available NSPT sets for an 8x8 block. An NSPT set for a 4x4 block may consist of 3 NSPT kernel candidates, an NSPT set for a 4x8 and 8x4 block may consist of 3 or 2 NSPT kernel candidates, and an NSPT set for an 8x8 block may consist of 1 NSPT kernel candidate.

[0269] The size of the conversion kernel may increase as the block size increases. Therefore, by reducing the number of available NSPT sets and / or the number of NSPT kernel candidates belonging to those NSPT sets, the memory size required to store the conversion kernel can be saved. In addition, as the block size increases, the residual signal characteristics within a block tend to become more generalized. Therefore, reducing the number of available NSPT sets and / or the number of NSPT kernel candidates belonging to those NSPT sets helps maintain compression efficiency while reducing the realization complexity, reflecting these statistical characteristics.

[0270] An NSPT kernel may be constructed with 8 bits of precision. The coefficients within the NSPT kernel may be greater than or equal to -128 and less than or equal to -127. If the precision is increased beyond 8 bits, the result obtained from matrix multiplication can be shifted to the right by the increased precision. For example, if the value obtained after matrix multiplication based on an NSPT kernel with 8 bits of precision is shifted to the right by S bits and stored in a buffer, if the kernel coefficients are constructed with N bits of precision, then it can be shifted to the right by (S + (N-8)) bits and stored in the buffer.

[0271] When configuring the NSPT kernel with 8-bit precision, it is possible to prevent an excessive increase in internal precision within the code / decoder performing the conversion, minimizing the decline in compression efficiency while reducing the complexity of implementation in terms of memory requirements and computational complexity.

[0272] When applying an inverse NSPT to a current block of size MxN, the size of the NSPT kernel (or NSPT matrix) can be expressed as MNxr, where MN represents the product of the width and height of the current block, which can represent the output length of the NSPT or the number of residual samples generated by the NSPT. r represents the input length of the NSPT or the number of (inversely quantized) transformation coefficients to which the NSPT is applied. r can be an integer greater than or equal to 0 and less than or equal to MN. The following is an example for an MNxr NSPT matrix with a block size.

[0273] The NSPT matrix for a 4x4 block may be a 16x16 matrix. The NSPT matrices for 4x8 and 8x4 blocks may be a 32x20, 32x16, 32x24, 32x28, or 32x32 matrix. The NSPT matrix for an 8x8 block may be a 64x16, 64x24, 64x32, 64x40, 64x48, 64x56, or 64x64 matrix. The NSPT matrices for 4x16 and 16x4 blocks may be a 64x16, 64x24, 64x32, 64x40, 64x48, 64x56, or 64x64 matrix. The NSPT matrices for 8x16 and 16x8 blocks may consist of 128x96, 128x64, 128x48, or 128x32 matrices. The NSPT matrices for 16x16 blocks may consist of 256x128, 256x96, or 256x64 matrices. The NSPT matrices for 16x32 and 32x16 blocks may consist of 512x256 or 512x128 matrices. The NSPT matrices for 32x32 blocks may consist of 1024x512, 1024x256, or 1024x128 matrices.

[0274] Alternatively, a 16x16 matrix may be applied to the 4xN block and the Nx4 block. Here, N may be an integer greater than or equal to 4. A 64x16 matrix may be applied to the 8x8 block. A 64x32 matrix may be applied to the 8xN block and the Nx8 block. Here, N may be an integer greater than or equal to 16. A 96x32 matrix may be applied to the 16xN block and the Nx16 block. Here, N may be an integer greater than or equal to 16.

[0275] Alternatively, the value of r in the MNxr NSPT matrix can also be determined according to a predetermined criterion. The criterion here may be such that (1) the sum of the computational amount for the primary transformation and the computational amount for the secondary transformation is below a certain level, and (2) the number of multiplications per sample required for the NSPT operation is below a certain number.

[0276] Based on the reverse transformation, when performing a separable primary transformation on the MxN block by matrix multiplication, (M + N) multiplications per sample are required to perform the primary transformation. Also, when applying LFNST to a certain ROI (Region-Of-Interest) region, assuming that the LFNST matrix for the inverse transformation is a PxQ matrix, (P*Q) / (M*N) multiplications per sample are required. Here, the PxQ matrix can mean a matrix with P columns and Q rows.

[0277] When applying NSPT instead of DCT-2 transformation (or a separable transformation such as KLT) and LFNST to the MxN block, the value of r such that the number of multiplications per sample for applying the NSPT is less than or equal to the number of multiplications per sample for applying DCT-2 transformation and LFNST may be determined as follows.

[0278] [Equation 6] [Number]

[0279] If the above equation 6 is satisfied and the value of r is set to its maximum value (i.e., r = M + N + (P*Q) / (M*N)), the value of r in the block-size NSPT matrix may be set as follows. If the value of (P*Q) / (M*N) in equation 6 is not an integer, an integer close to the value of (P*Q) / (M*N) may be used. For example, the floor operation may be applied to the value of (P*Q) / (M*N). In this case, r may be set to (M + N + floor((P*Q) / (M*N))). Here, floor(x) can mean the largest integer not exceeding x. Alternatively, the round operation may be applied to the value of (P*Q) / (M*N). In this case, r may be set to (M + N + round((P*Q) / (M*N))). Here, round(x) can mean the value of x rounded. Alternatively, the ceil operation may be applied to the value of (P*Q) / (M*N). In this case, r may be set to (M+N+ceil((P*Q) / (M*N))). Here, ceil(x) can mean the smallest integer greater than or equal to x. If the floor operation is applied, the inequality in equation 6 above may be satisfied. However, if the round operation or the ceil operation is applied, the inequality in equation 6 may not be satisfied.

[0280] For an NSPT on a 4x4 block, the maximum value of r is 24. However, the value of r must be less than or equal to 16, and therefore, the value of r may be set to 16.

[0281] For NSPTs on 4x8 and 8x4 blocks, the maximum value of r is 20. The value of r may be set to 20.

[0282] For NSPT on an 8x8 block, the maximum value of r is 32. The value of r can be set to 32.

[0283] For NSPTs on 4x16 and 16x4 blocks, the maximum value of r is 24. The value of r may be set to 24.

[0284] For NSPTs on 8x16 and 16x8 blocks, the maximum value of r is 40. The value of r may be set to 40.

[0285] For NSPT on a 16x16 block, the maximum value of r is 44. The value of r may be set to 44.

[0286] For NSPTs on 16x32 and 32x16 blocks, the maximum value of r is 54. The value of r may be set to 54.

[0287] For NSPT on a 32x32 block, the maximum value of r is 67. The value of r can be set to 67.

[0288] For NSPTs on 4x32 and 32x4 blocks, the maximum value of r is 38. The value of r may be set to 38, or it may be set to 20.

[0289] For NSPTs on 8x32 and 32x8 blocks, the maximum value of r is 48. The value of r may be set to 48, or it may be set to 24.

[0290] In the aforementioned NSPT matrices for different block sizes, the value of r may not be a multiple of 4. For ease of implementation, it would be advantageous for r to be a multiple of 4. For example, when implementing parallel processing using SIMD (Single Instruction Multiple Data) instructions, if the inner product of four transformation basis vectors is processed simultaneously during the process of applying forward NSPT (i.e., four transformation coefficients are generated simultaneously), it would be advantageous to set the value of r to be a multiple of 4.

[0291] For example, for NSPTs for 16x32 and 32x16 blocks, the value of r may be set to 52 or 56 instead of 54. For NSPTs for 32x32 blocks, the value of r may be set to 64 or 68 instead of 67. For NSPTs for 4x32 and 32x4 blocks, the value of r may be set to 36 or 40 instead of 38.

[0292] More generally, the value of r may be set to be a multiple of K, where K may be an integer greater than or equal to 1. As an example, the value of r may be set to a multiple of K that satisfies the following inequality, equation 7.

[0293] [Equation 7]

number

[0294] In formula 7 above, func() can be floor, round, or ceil as previously mentioned.

[0295] The value of r according to the predetermined criteria mentioned above is for cases where zero-out is not considered. That is, when forward LFNST is applied, the linearly transformed conversion coefficients of the remaining region outside the region to which LFNST is applied are zeroed out, so the actual computational cost required to apply DCT-2 and LFNST may be less than the aforementioned computational cost. Therefore, when zero-out is considered, the value of r may be set to a value smaller than the value of r according to the predetermined criteria.

[0296] Since zero-out is not performed for 4x4 blocks, the value of r can be set to a value less than or equal to 16.

[0297] For a 4x8 block, a zero-out may be performed on the remaining region excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 8 (((4x4x8)+(4x8x4)) / (4x8)=8), and the number of multiplications per sample required in LFNST is also 8 ((16x16) / 32=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 16, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 16.

[0298] For an 8x4 block, a zero-out may be performed on the remaining region excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 6 ((4x8x4)+(4x4x4) / (8x4)=6), and the number of multiplications per sample required in LFNST is 8 ((16x16) / 32=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 14, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 14.

[0299] For 8x8 blocks, zero-out is not required for the delimited linear transformation, in which case the value of r may be set to a value less than or equal to 32.

[0300] For an 8x16 block, a zero-out may be performed on the remaining region excluding the top-left 8x8 block, based on the forward transformation. For the top-left 8x8 block, a 64x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 16 ((8x8x16)+(8x16x8) / (8x16)=16), and the number of multiplications per sample required in LFNST is also 16 ((64x32) / 128=16). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 32, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. Since the same computational complexity is required when applying the inverse delimited linear transformation and LFNST, the value of r can be set to a value less than or equal to 32.

[0301] For a 16x8 block, a zero-out may be performed on the remaining region excluding the top-left 8x8 block, based on the forward transformation, and a 64x32 matrix, which is the forward LFNST matrix, may be applied to the top-left 8x8 block. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 12 ((8x16x8)+(8x8x8) / (16x8)=12), and the number of multiplications per sample required in LFNST is 16 ((64x32) / 128=16). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 28, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. Since the same computational complexity is required when applying the inverse delimited linear transformation and LFNST, the value of r can be set to a value less than or equal to 28.

[0302] For a 16x16 block, a zero-out may be performed on the remaining region excluding the top-left 12x12 block, based on the forward transformation. For the top-left 12x12 block, a 96x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 21 ((12x16x16)+(12x16x12) / (16x16)=21), and the number of multiplications per sample required in LFNST is 12 ((96x32) / 256=12). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 33, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. Since the same computational complexity is required when applying the inverse delimited linear transformation and LFNST, the value of r can be set to a value less than or equal to 33.

[0303] For a 4x16 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 8 (((4x4x16)+(4x16x4)) / (4x16)=8), and the number of multiplications per sample required in LFNST is 4 ((16x16) / 64=4). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 12, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 12.

[0304] For a 16x4 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 5 (((4x16x4)+(4x4x4)) / (16x4)=5), and the number of multiplications per sample required in LFNST is 4 ((16x16) / 64=4). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 9, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 9.

[0305] For a 4x32 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 8 (((4x4x32)+(4x32x4)) / (4x32)=8), and the number of multiplications per sample required in LFNST is 2 ((16x16) / 128=2). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 10, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 10.

[0306] For a 32x4 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 4.5 (((4x32x4)+(4x4x4)) / (32x4)=4.5), and the number of multiplications per sample required for LFNST is 2 ((16x16) / 128=2). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 6.5. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 6.5. Here, the value of r is a value that constitutes the matrix dimension, so it can be set to an integer of 6 or 7 instead of 6.5.

[0307] For an 8x32 block, a zero-out may be performed on the remaining region, excluding the top-left 8x8 block, based on the forward transformation. For the top-left 8x8 block, a 64x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 16 (((8x8x32)+(8x32x8)) / (8x32)=16), and the number of multiplications per sample required for LFNST is 8 ((64x32) / 256=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 24. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 24.

[0308] For a 32x8 block, a zero-out may be performed on the remaining region, excluding the top-left 8x8 block, based on the forward transformation. For the top-left 8x8 block, a 64x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 10 (((8x32x8)+(8x8x8)) / (32x8)=10), and the number of multiplications per sample required for LFNST is 8 ((64x32) / 256=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 18. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 18.

[0309] For a 16x32 block, a zero-out may be performed on the remaining region, excluding the top-left 12x12 block, based on the forward transformation. For the top-left 12x12 block, a 96x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 21 (((12x16x32)+(12x32x12)) / (16x32)=21), and the number of multiplications per sample required for LFNST is 6 ((96x32) / 512=6). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 27. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 27.

[0310] For a 32x16 block, a zero-out may be performed on the remaining region, excluding the top-left 12x12 block, based on the forward transformation. For the top-left 12x12 block, a 96x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 13.5 (((12x32x12)+(12x16x12)) / (32x16)=13.5), and the number of multiplications per sample required for LFNST is 6 ((96x32) / 512=6). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 19.5. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 19.5. Here, the value of r is a value that constitutes the matrix dimension, so it may be set to an integer of 19 or 20 instead of 19.5.

[0311] In the aforementioned block-size-specific NSPT matrices, there may be cases where the value of r is not a multiple of K. Here, K may be 4. In this case, for ease of implementation, the value of r can be set to be a multiple of K. The value of r that was set in advance by the method described above is r prev In this case, the values ​​of r that are multiples of K may be set as follows:

[0312] [Formula 8]

number

[0313] In formula 8 above, func() can be floor, round, or ceil as previously mentioned.

[0314] As mentioned above, the value of r in the NSPT matrix for an MxN block may be different from the value of r in the NSPT matrix for an NxM block. For example, the inverse NSPT matrix for a 4x8 block may be a 32x16 matrix, and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix. In this case, the NSPT matrix may be determined by utilizing the symmetry between the MxN block and the NxM block.

[0315] Assume the current block is an MxN block having mode x. When applying the aforementioned symmetry to the current block, instead of applying the NSPT matrix corresponding to mode x or the MxN block size, an NSPT matrix corresponding to at least one of the modes symmetric to mode x or the NxM block size symmetric to the MxN block size may be applied. In this case, the NSPT matrix corresponding to the NxM block size may be applied directly to the current block. Alternatively, the NSPT matrix corresponding to the NxM block size may be applied, but the value of r may be the same as the value of r in the NSPT matrix corresponding to the MxN block size.

[0316] For example, the inverse NSPT matrix for a 4x8 block is a 32x16 matrix (i.e., the value of r in the NSPT matrix is ​​16), and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix (i.e., the value of r in the NSPT matrix is ​​14). When the current block is an 8x4 block having mode x, an NSPT matrix for at least one of the modes symmetric to mode x or a 4x8 block symmetric to the 8x4 block may be applied to the current block. In this case, the 32x16 matrix, which is the inverse NSPT matrix for the 4x8 block, may be used as is, or a 32x14 matrix with the value of r in the inverse NSPT matrix for the 8x4 block may be used. Here, the 32x14 matrix may be derived by sampling the 14 rows from the left of the 32x16 matrix. In this way, when applying the 32x14 matrix to the current block having a block size of 8x4, the predetermined criteria mentioned above will be satisfied.

[0317] Conversely, if the current block is a 4x8 block having mode x, then an NSPT matrix for at least one mode symmetric to mode x or an 8x4 block symmetric to the 4x8 block may be applied to the current block. In this case, a 32x14 matrix for the 8x4 block may be applied to the current block instead of a 32x16 matrix for the 4x8 block. This allows NSPT to be performed on the 4x8 block using fewer multiplications than the allowed number of multiplications.

[0318] For NSPTs for MxN and NxM blocks, if the values ​​of r that satisfy the aforementioned predetermined conditions are r1 and r2, respectively, the inverse NSPT matrices for MxN and NxM blocks may be set to MNxmax(r1,r2). Here, max(r1,r2) can be interpreted as selecting the larger or equal value of r1 and r2.

[0319] For example, the inverse NSPT matrix for a 4x8 block is a 32x16 matrix (i.e., the value of r in the NSPT matrix is ​​16), and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix (i.e., the value of r in the NSPT matrix is ​​14). If the current block is a 4x8 block having mode x, then an NSPT matrix for at least one of the modes symmetric to mode x or an 8x4 block symmetric to the 4x8 block may be applied to the current block. In this case, the 32x16 matrix, which is the inverse NSPT matrix for the 8x4 block, may be used. If the inverse NSPT matrix is ​​not constructed as MNxmax(r1,r2), then the 32x14 matrix will be used as the NSPT matrix for the 8x4 block. However, if the NSPT matrix for the 4x8 block and the NSPT matrix for the 8x4 block are constructed as 32xmax(16,14) matrices, then the 32x16 matrix may be fully applied.

[0320] Conversely, if the current block is an 8x4 block having mode x, an NSPT matrix may be applied to the current block for at least one of the modes symmetric to mode x or a 4x8 block symmetric to the 8x4 block. In this case, a 32x16 matrix, which is the inverse NSPT matrix for the 4x8 block, may be used, or a 32x14 matrix may be used. Here, the 32x14 matrix may be derived by sampling the 14 rows from the left of the 32x16 matrix.

[0321] As mentioned above, when constructing an NSPT matrix, it is possible to apply a transformation composed of the maximum number of transformation basis vectors while satisfying the predetermined conditions, thereby maximizing coding performance.

[0322] In the embodiments described above, the value of r may be set to be a multiple of 16. For example, in the case of reverse NSPT for 4x8 and 8x4 blocks, a 32x16 matrix may be applied instead of a 32x20 matrix. The transformation coefficients of the transformation block may be encoded in units of a predetermined coefficient group (Coefficient Group, CG). Here, CG may be defined as a group of 16 transformation coefficients, and for example, CG may be subblocks of sizes such as 4x4, 2x8, and 8x2. There may be cases where no non-zero transformation coefficients exist within a single CG, in which case the encoding process of the transformation coefficients may be skipped for that CG. Therefore, setting the value of r to a multiple of 16 has the advantage of reducing the complexity of implementation.

[0323] A forward NSPT can be applied to a resistive sample of an MxN block to induce transformation coefficients. In this case, the number of induced transformation coefficients may be less than or equal to the value of (M*N) by zero-out. That is, the forward NSPT matrix may be defined as an rx(M*N) matrix, where r represents the output length of the NSPT or the number of transformation coefficients induced by the NSPT, and (M*N) represents the input length of the NSPT or the number of resistive samples to which the NSPT is applied.

[0324] The derived conversion coefficients may be arranged within an MxN block according to a predetermined scan order, and any region where the conversion coefficients are not filled may be filled with zeros (i.e., zero-out). Therefore, if, during the process of scanning the conversion coefficients with the decoding device, a non-zero conversion coefficient is found in a region that would have been filled with zeros if NSPT had been applied (or if the scan position of the last effective coefficient in the MxN block is greater than or equal to r), it is considered that NSPT has not been applied to that MxN block, and the NSPT index does not need to be signaled. The upper left coefficient in the MxN block (i.e., the DC component coefficient) may be assigned an index indicating the scan position of zero, and the remaining coefficients in the MxN block may be assigned indices increasing by 1 according to a predetermined order.

[0325] One or more values ​​of r may be defined for the block sizes to which NSPT can be applied. For example, one or more values ​​of r may be defined for each of the block sizes to which NSPT can be applied. Alternatively, one value of r may be defined for each block size to which NSPT can be applied, and the value of r for one of the block sizes to which NSPT can be applied may be different from the value of r for any other size. Alternatively, one value of r may be defined for some of the block sizes to which NSPT can be applied, and two or more values ​​of r may be defined for the rest.

[0326] If multiple values ​​for r are available, an index that identifies any one of the multiple values ​​of r, or r itself, may be signaled. This index may be signaled using high-level syntax (HLS) such as VPS, SPS, PPS, PH, SH, or at the block level such as CTU, CU, TU. If the value of r belongs to a specific range, the bits that may include that range can be allocated and signaled. For example, if the value of r belongs to the range of 1 to 256, 8 bits can be specified as fixed-length and signaled.

[0327] The current block's translation kernel may be determined based on any one of the aforementioned Examples 1 to 3. Alternatively, the current block's translation kernel may be determined based on a combination of at least two of Examples 1 to 3, provided that the inventions described in Examples 1 to 3 do not conflict with each other.

[0328] A transformation index for the inverse transformation of the current block may be signaled. Here, the transformation index can identify one of one or more transformation kernels (or transformation matrices) belonging to the transformation set. Here, the transformation index can mean an NSPT index that identifies one of one or more NSPT kernels belonging to the NSPT set. Or, the transformation index can mean an LFNST index that identifies one of one or more LFNST kernels belonging to the LFNST set.

[0329] Whether the aforementioned conversion index corresponds to an NSPT index may be determined based on whether the current block size is one of the block sizes belonging to the first group described above. This assumes that the block sizes to which NSPT can be applied and the block sizes to which LFNST can be applied are distinct from each other. In this case, if the current block size belongs to the first group, the conversion index signaled to the current block corresponds to an NSPT index, and an NSPT kernel may be determined from the NSPT set based on that conversion index. On the other hand, if the current block size does not belong to the first group, the conversion index signaled to the current block corresponds to an LFNST index, and an LFNST kernel may be determined from the LFNST set based on that conversion index. If the current block size does not belong to the first group, it may mean that the current block size belongs to the second group described above. Alternatively, if the current block size does not belong to the first group, it may mean that the current block size corresponds to a block size among the block sizes belonging to the second group to which LFNST can be applied. Thus, NSPT indexes and LFNST indexes do not need to be composed of separate syntaxes, but rather can be composed of a single, integrated syntax.

[0330] As an example, let's assume that the block sizes to which NSPT can be applied that belong to the first group are 4x4, 4x8, 8x4, and 8x8. For the block sizes belonging to the first group, NSPT may be applied instead of LFNST. Specifically, NSPT may be applied instead of the combination of a separate linear transform (e.g., DCT-2, separate KLT) and LFNST. An NSPT index may be signaled for the four block sizes belonging to the first group, and an LFNST index may be signaled for the remaining block sizes (to which LFNST is permitted).

[0331] Thus, signaling with a single syntax that integrates NSPT and LFNST indices can reduce the amount of information encoded. Furthermore, the complexity of implementation can be reduced by applying at least one of the following to NSPT / LFNST indices identically: binary code coding, CABAC context, or initial value.

[0332] Alternatively, the NSPT index and LFNST index can be signaled as separate syntaxes. In this case, the complexity of implementation may increase to some extent, but compression performance can be improved by performing optimized entropy coding for each index.

[0333] If the number of LFNST kernel candidates belonging to the LFNST set is the same as the number of NSPT kernel candidates belonging to the NSPT set, the same binarization may be applied to the LFNST index and the NSPT index. The same CABAC context (or CABAC context increment) may be assigned to the bins of the LFNST index and the NSPT index.

[0334] Different binary evolutions and / or CABAC contexts may be used for LFNST and NSPT indexes. Different CABAC initial values ​​may be assigned to LFNST and NSPT indexes. For example, one of the LFNST and NSPT indexes may be binary-evolved based on fixed-length binarization, and the other may be binary-evolved based on truncated unary binarization. Even if the binary evolutions of the LFNST and NSPT indexes are the same, different CABAC contexts and / or CABAC initial values ​​may be assigned to them. Different binary evolutions and / or CABAC contexts may be used for LFNST and NSPT indexes when the number of LFNST kernel candidates belonging to the LFNST set and the number of NSPT kernel candidates belonging to the NSPT set are different from each other.

[0335] The number of NSPT kernel candidates belonging to an NSPT set may be set to differ from one another for each block size. Alternatively, the block sizes belonging to the first group may be divided into multiple subgroups. In this case, the number of NSPT kernel candidates belonging to an NSPT set may be set to differ from one another for each of the multiple subgroups. Here, at least one of the multiple subgroups may contain multiple different block sizes.

[0336] Depending on the number of NSPT kernel candidates belonging to the NSPT set, the binary evolution applied to the NSPT index may differ from one another.

[0337] For example, if there are three NSPT kernel candidates in the NSPT set for a given block size, the NSPT index may have one value between 0 and 3. If the NSPT index value is 0, this can indicate that NSPT is not currently applied to the block. If the NSPT index value is not 0, this can indicate the NSPT kernel candidate corresponding to that NSPT index among the three NSPT kernel candidates. One bin can be assigned to distinguish between cases where NSPT is applied and cases where it is not. If the value of the bin is 0, this may correspond to the case where the NSPT index value is 0. On the other hand, if the value of the bin is 1, this may correspond to the case where the NSPT index value is 1, 2, or 3. In this case, truncated unative binary evolution may be applied to distinguish between the three NSPT kernel candidates. That is, two bins can be assigned to distinguish between 0, 10, 11 and the three NSPT kernel candidates.

[0338] When there are two NSPT kernel candidates in the NSPT set for a given block size, the NSPT index may have a value between 0 and 2. A value of 0 in the NSPT index indicates that NSPT is not currently applied to the block. A non-zero value in the NSPT index indicates the NSPT kernel candidate corresponding to that index among the two candidates. A bin can be assigned to distinguish between cases where NSPT is applied and cases where it is not. A bin can be assigned to indicate one of the two NSPT kernel candidates, thus distinguishing between the two NSPT kernel candidates.

[0339] When there is only one candidate NSPT kernel in the NSPT set for a given block size, the NSPT index may have a value of either 0 or 1. A value of 0 in the NSPT index indicates that NSPT is not currently applied to the block. A value of 1 indicates that there is one candidate NSPT kernel. In such cases, the presence or absence of NSPT application and the candidate NSPT kernel can be determined from a single bin.

[0340] The inverse transform of the current block may be a separated linear transform and / or an LFNST-based inverse transform. That is, a reverse LFNST can be applied to all or some of the (inversely quantized) transformation coefficients of the current block, and then a reverse separated linear transform can be applied to the transformation coefficients induced by the LFNST to induce a residual sample. As an example, a reverse LFNST may be applied to the (inversely quantized) transformation coefficients belonging to a portion of the current block. Here, the portion of the block means the region to which a forward LFNST has been applied, and will be hereinafter referred to as the ROI (region of interest) region. The transformation coefficients induced by the LFNST may be arranged in the ROI region according to a predetermined scan order. The predetermined scan order may be row-first order or column-first order. A reverse separated linear transform may be applied to the transformation coefficients induced by the LFNST and the transformation coefficients belonging to the remaining region of the current block excluding the ROI region. Alternatively, if zero-out is performed on the remaining region of the current block excluding the ROI region during the forward transformation process (i.e., the transformation coefficients in the remaining region are set to 0), a reverse separation linear transformation may be applied to the transformation coefficients induced by LFNST. The size of the ROI region may be determined based on at least one of the width or height of the current block. Here, the size of the ROI region may mean at least one of the width or height of the ROI region, or it may mean the number of sample positions belonging to the ROI region.

[0341] For the current block, either LFNST defined for a 4x8 block (hereinafter referred to as LFNST4x8) or LFNST defined for an 8x4 block (hereinafter referred to as LFNST8x4) may be applied. When LFNST4x8 is applied to the current block, the ROI region of the current block may be a 4x8 region within the current block. Here, the 4x8 region is the region containing the top-left sample of the current block, and can mean a block with a width of 4 and a height of 8, respectively. When LFNST8x4 is applied to the current block, the ROI region of the current block may be an 8x4 region within the current block. Here, the 8x4 region is the region containing the top-left sample of the current block, and can mean a block with a width of 8 and a height of 4, respectively.

[0342] The LFNST4x8 may be applied only when the current block is a 4x8 block. Alternatively, the LFNST4x8 may be applied when the current block is 4xN and N is greater than or equal to 8 (for example, when the current block is a 4x8, 4x16, 4x32, or larger block). Alternatively, the LFNST4x8 may be applied when the current block is 4xN and N is greater than or equal to 8 and less than or equal to a predetermined threshold. For example, when the threshold is 16, the LFNST4x8 may be applied when the current block is a 4x8 or 4x16 block.

[0343] The LFNST8x4 may be applied only when the current block is an 8x4 block. Alternatively, the LFNST8x4 may be applied when the current block is Nx4 and N is greater than or equal to 8 (for example, when the current block is an 8x4, 16x4, 32x4, or larger block). Alternatively, the LFNST8x4 may be applied when the current block is Nx4 and N is greater than or equal to 8 and less than or equal to a predetermined threshold. For example, when the threshold is 16, the LFNST8x4 may be applied when the current block is an 8x4 or 16x4 block.

[0344] Specifically, when LFNST4x8 or LFNST8x4 is applied to the current block, an LFNST (hereinafter referred to as LFNST-32) with an input length less than or equal to 32 and an output length of 32 may be used. The input length can mean the number of transformation coefficients input to the LFNST. Here, the transformation coefficients input to the LFNST can mean all of the transformation coefficients derived based on the current block's residual information, or it can mean the transformation coefficients to which the forward LFNST is applied among the derived transformation coefficients. The output length can mean the number of transformation coefficients derived by the LFNST. The ROI region of the current block may consist of 32 sample positions. In this case, the dimension (or size) of the transformation matrix for the reverse LFNST4x8 or LFNST8x4 may be 32xN, where N may be 16, 20, or 24. In other words, the input length of the LFNST-32 is N, and the output length of the LFNST-32 may be 32.

[0345] For example, if the current block size is 4xN, LFNST-32 may be applied to all or part of the transformation coefficients belonging to a 4x8 region, which is the ROI area within the current block. The transformation coefficients induced by LFNST-32 may be arranged in the 4x8 region within the current block according to a predetermined scan order. Alternatively, if the current block size is Nx4, LFNST-32 may be applied to all or part of the transformation coefficients belonging to an 8x4 region, which is the ROI area within the current block. The transformation coefficients induced by LFNST-32 may be arranged in the 8x4 region within the current block according to a predetermined scan order.

[0346] The translation kernel for LFNST-32 may be determined to be one or more translation kernels (or candidate LFNST kernels) belonging to the translation set. For this purpose, an index may be used to identify one of the multiple translation kernels belonging to the translation set. Here, the index may be signaled by a bitstream. The translation set of the current block may be determined to be one of a plurality of predefined translation sets. For example, the translation set of the current block may be determined to be one of 35 translation sets, each translation set may contain 3 translation kernels. The translation set of the current block may be determined based on the intra-prediction mode of the current block, as described above. The plurality of translation sets for LFNST-32 may be defined for LFNST4x8 and LFNST8x4, respectively.

[0347] However, as mentioned above, LFNST4x8 and LFNST8x4 may be applied adaptively based on the size of the current block. If LFNST4x8 and LFNST8x4 are not applicable to the current block, an LFNST (hereinafter referred to as LFNST-16) with an input length less than or equal to 16 and an output length of 16 may be used. LFNST-16 may be applied to all or part of the transformation coefficients belonging to a 4x4 region within the current block. The 4x4 region within the current block is the region containing the top-left sample of the current block, and can represent a block with a width and height of 4 and 4, respectively. The transformation coefficients induced by LFNST-16 may be arranged in the 4x4 region within the current block according to a predetermined scan order.

[0348] From the perspective of forward transformation, transformation coefficients can be derived by applying a separated linear transformation to the residual samples of the current block. Forward LFNST may be applied only to all or some of the derived transformation coefficients that belong to a region within the current block (i.e., a region of interest (ROI)). The transformation coefficients derived by LFNST may be arranged in the ROI region within the current block according to a predetermined scan order. If the number of transformation coefficients output from LFNST is less than the number of transformation coefficients input to LFNST, there may be regions within the ROI region that are not filled with transformation coefficients derived by LFNST. These regions may be filled with transformation coefficients that are 0. In addition, the remaining regions in the current block excluding the ROI region (or regions to which LFNST is not applied) may be filled with transformation coefficients derived by a separated linear transformation, or they may be filled with transformation coefficients that are 0 by zero-out.

[0349] Specifically, when an LFNST4x8 or LFNST8x4 is applied to the current block or the ROI region of the current block, an LFNST with an input length of 32 and an output length less than or equal to 32 (hereinafter referred to as LFNST-32) may be used. Here, the input length can mean the number of transformation coefficients input to the LFNST. The input transformation coefficients may be coefficients induced by a separated linear transformation. The input length can also mean the number of transformation coefficients belonging to the ROI region of the current block. The output length can mean the number of transformation coefficients induced by the LFNST. The ROI region of the current block may consist of 32 sample positions. In this case, the dimension of the transformation matrix for the forward LFNST4x8 or LFNST8x4 may be Nx32, where N may be 16, 20, or 24. That is, the input length of LFNST-32 is 32, and the output length of LFNST-32 may be N.

[0350] For example, if the current block size is 4xN, LFNST-32 may be applied to the transformation coefficients belonging to a 4x8 region, which is the ROI area within the current block. The transformation coefficients induced by LFNST-32 may be arranged in the 4x8 region within the current block according to a predetermined scan order. Alternatively, if the current block size is Nx4, LFNST-32 may be applied to the transformation coefficients belonging to an 8x4 region, which is the ROI area within the current block. The transformation coefficients induced by LFNST-32 may be arranged in the 8x4 region within the current block according to a predetermined scan order.

[0351] However, as mentioned above, LFNST4x8 and LFNST8x4 may be applied adaptively based on the size of the current block. If LFNST4x8 and LFNST8x4 are not applicable to the current block, an LFNST with input and output lengths of 16 and 16 respectively (hereinafter referred to as LFNST-16) may be used. LFNST-16 may be applied to conversion coefficients belonging to a 4x4 region within the current block. The 4x4 region within the current block is the region containing the top-left sample of the current block, and can represent a block with a width and height of 4 and 4, respectively. The conversion coefficients induced by LFNST-16 may be arranged in the 4x4 region within the current block according to a predetermined scan order.

[0352] When applying LFNST4x8 or LFNST8x4 to a current block, the aforementioned symmetries between block configurations may be utilized. For example, the symmetry between a 4xN block and an Nx4 block may determine the LFNST set and / or LFNST kernel for the current block's LFNST. A 4xN block (or an Nx4 block) can use the same LFNST set and / or LFNST kernel as an Nx4 block (or a 4xN block) that has symmetry with it. For blocks where N is 8, the aforementioned symmetries may be used directly, and for blocks where N is greater than or equal to 16, LFNST may be applied to the ROI region (i.e., the 4x8 region or 8x4 region) within the current block using the aforementioned symmetries.

[0353] Furthermore, LFNST for 8x16 blocks (hereinafter referred to as LFNST8x16) and LFNST for 16x8 blocks (hereinafter referred to as LFNST16x8) may be defined. When LFNST8x16 is applied to the current block, the ROI region of the current block may be a 4x8, 4x16, 8x8, or 8x16 region within the current block. Here, the ROI region is the region containing the upper left corner sample of the current block and can mean a block having that width and height. When LFNST16x8 is applied to the current block, the ROI region of the current block may be an 8x4, 16x4, 8x8, or 16x8 region within the current block. Here, the ROI region is the region containing the upper left corner sample of the current block and can mean a block having that width and height. The ROI region may be pre-set identically for the encoding device and the decoding device for LFNST8x16 and LFNST16x8, respectively. Blocks with widths of 8 and N, respectively, may be assigned the same ROI region. Alternatively, at least one block among those with widths of 8 and N, respectively, may have a different ROI region from the other blocks. Similarly, blocks with widths of N and 8, respectively, may be assigned the same ROI region. Alternatively, at least one block among those with widths of N and 8, respectively, may have a different ROI region from the other blocks.

[0354] The LFNST8x16 may be applied only when the current block is an 8x16 block. Alternatively, the LFNST8x16 may be applied when the current block is 8xN and N is greater than or equal to 16 (for example, when the current block is an 8x16, 8x32, 8x64, or larger block). Alternatively, the LFNST8x16 may be applied when the current block is 8xN and N is greater than or equal to 16 and less than or equal to a predetermined threshold. For example, if the threshold is 32, the LFNST8x16 may be applied when the current block is an 8x16 or 8x32 block.

[0355] The LFNST16x8 may be applied only when the current block is a 16x8 block. Alternatively, the LFNST16x8 may be applied when the current block is Nx8 and N is greater than or equal to 16 (for example, when the current block is a 16x8, 32x8, 64x8, or larger block). Alternatively, the LFNST16x8 may be applied when the current block is Nx8 and N is greater than or equal to 16 and less than or equal to a predetermined threshold. For example, if the threshold is 32, the LFNST16x8 may be applied when the current block is a 16x8 or 32x8 block.

[0356] Specifically, when LFNST8x16 or LFNST16x8 is applied to the current block, an LFNST (hereinafter referred to as LFNST-K) with an input length less than or equal to K and an output length of K may be used. As mentioned above, the input length can mean the number of conversion coefficients input to the LFNST. Here, the conversion coefficients input to the LFNST can mean all of the conversion coefficients induced based on the current block's residual information, or it can mean the conversion coefficients to which a forward LFNST is applied among the induced conversion coefficients. The output length can mean the number of conversion coefficients induced by the LFNST.

[0357] The aforementioned K may be determined based on the size of the ROI region of the current block. For example, if the ROI region is 4x8 or 8x4, an LFNST with an input length less than or equal to 32 and an output length of 32 (i.e., LFNST-32) may be used. If the ROI region is 4x16, 16x4, or 8x8, an LFNST with an input length less than or equal to 64 and an output length of 64 (i.e., LFNST-64) may be used. If the ROI region is 8x16 or 16x8, an LFNST with an input length less than or equal to 128 and an output length of 128 (i.e., LFNST-128) may be used. In other words, when LFNST8x16 or LFNST16x8 is applied to an 8xN or Nx8 current block, LFNST-K may be applied to the ROI region within the current block. Here, K may be 32, 64, or 128, but these are just examples, and K may be an integer greater than 128. The ROI region of the current block may consist of 128 sample positions. In this case, the dimension (or size) of the transformation matrix for the inverse LFNST8x16 or LFNST16x8 may be 128xN, where N may be 32, 40, or 48. That is, the input length of LFNST-128 may be N, and the output length of LFNST-128 may be 128.

[0358] If the current block size is 8xN, LFNST-K may be applied to all or part of the transformation coefficients belonging to the ROI region within the current block. The transformation coefficients induced by LFNST-K may be arranged in the ROI region within the current block according to a predetermined scan order. Alternatively, if the current block size is Nx8, LFNST-K may be applied to all or part of the transformation coefficients belonging to the ROI region within the current block. The transformation coefficients induced by LFNST-K may be arranged in the ROI region within the current block according to a predetermined scan order.

[0359] The translation kernel for LFNST-128 may be determined to be one or more translation kernels (or LFNST kernel candidates) belonging to the translation set. For this purpose, an index may be used to identify one of the multiple translation kernels belonging to the translation set. Here, the index may be signaled by a bitstream. The translation set of the current block may be determined to be one of a plurality of predefined translation sets. For example, the translation set of the current block may be determined to be one of 35 translation sets, each translation set may contain 3 translation kernels. The translation set of the current block may be determined based on the intra-prediction mode of the current block, as described above. The plurality of translation sets for LFNST-128 may be defined for LFNST8x16 and LFNST16x8, respectively.

[0360] However, as mentioned above, LFNST8x16 and LFNST16x8 may be applied adaptively based on the size of the current block. If LFNST8x16 and LFNST16x8 are not applicable to the current block, an LFNST with an input length of 16 and an output length of 48 (hereinafter referred to as LFNST-48) may be used. LFNST-48 may be applied to conversion coefficients belonging to a 4x4 region within the current block. The 4x4 region within the current block is the region containing the upper left corner sample of the current block, and can represent a block with a width and height of 4 and 4, respectively. The conversion coefficients induced by LFNST-48 may be arranged in three 4x4 regions within the current block according to a predetermined scan order. The three 4x4 regions may consist of a 14x4 region containing the upper left corner sample of the current block, a 24x4 region adjacent to the right side of the 14x4 region, and a 34x4 region adjacent to the bottom edge of the 14x4 region.

[0361] From the perspective of forward transformation, transformation coefficients can be derived by applying a separated linear transformation to the residual samples of the current block. Forward LFNST may be applied only to all or some of the derived transformation coefficients that belong to a region within the current block (i.e., a region of interest (ROI)). The transformation coefficients derived by LFNST may be arranged in the ROI region within the current block according to a predetermined scan order. If the number of transformation coefficients output from LFNST is less than the number of transformation coefficients input to LFNST, there may be regions within the ROI region that are not filled with transformation coefficients derived by LFNST. These regions may be filled with transformation coefficients of 0. In addition, the remaining regions in the current block excluding the ROI region (or regions to which LFNST is not applied) may be filled with transformation coefficients derived by a separated linear transformation, or they may be filled with transformation coefficients of 0 by zero-out.

[0362] Specifically, when LFNST8x16 or LFNST16x8 is applied to the current block or the ROI region of the current block, an LFNST with an input length of K and an output length less than or equal to K (i.e., LFNST-K) may be used. Here, the input length can mean the number of transformation coefficients input to the LFNST. The input transformation coefficients may be coefficients induced by a separate linear transformation. The input length can mean the number of transformation coefficients belonging to the ROI region of the current block. The output length can mean the number of transformation coefficients induced by the LFNST. The ROI region of the current block may consist of 128 sample positions. In this case, the dimension of the transformation matrix for the forward LFNST8x16 or LFNST16x8 may be Nx128, where N may be 32, 40, or 48. That is, the input length of LFNST-128 may be 128, and the output length of LFNST-128 may be N.

[0363] For example, if the current block size is 8xN and N is greater than or equal to 16, LFNST-K may be applied to the transformation coefficients belonging to the ROI region within the current block. The transformation coefficients induced by LFNST-K may be arranged in the ROI region within the current block according to a predetermined scan order. Alternatively, if the current block size is Nx8 and N is greater than or equal to 16, LFNST-K may be applied to the transformation coefficients belonging to the ROI region within the current block. The transformation coefficients induced by LFNST-K may be arranged in the ROI region within the current block according to a predetermined scan order.

[0364] However, as mentioned above, LFNST8x16 and LFNST16x8 may be applied adaptively based on the size of the current block. If LFNST8x16 and LFNST16x8 are not applicable to the current block, an LFNST with input and output lengths of 48 and 16, respectively (i.e., LFNST-48) may be used. LFNST-48 may be applied to transformation coefficients belonging to three 4x4 regions within the current block. The three 4x4 regions may consist of a 14x4 region containing the upper left sample of the current block, a 24x4 region adjacent to the right of the 14x4 region, and a 34x4 region adjacent to the bottom of the 14x4 region. The transformation coefficients induced by LFNST-48 may be arranged in the 4x4 regions within the current block according to a predetermined scan order.

[0365] When applying LFNST8x16 or LFNST16x8 to a current block, the aforementioned symmetries between block configurations may be utilized. For example, the symmetry between an 8xN block and an Nx8 block may determine the LFNST set and / or LFNST kernel for LFNST-K of the current block. An 8xN block (or an Nx8 block) can use the same LNFST set and / or LFNST kernel as an Nx8 block (or an 8xN block) that has symmetry with it. For blocks where N is 16, the aforementioned symmetries may be used directly, and for blocks where N is greater than or equal to 32, LFNST may be applied to the ROI region (i.e., the 8x16 region or 16x8 region) within the current block using the aforementioned symmetries.

[0366] As mentioned above, LFNSTMx2M may be defined for Mx2M blocks and LFNST2MxM may be defined for 2MxM blocks. LFNSTMx2M may be applied to the current MxN block, and LFNST2MxM may be applied to the current NxM block. Here, N may be greater than or equal to (2*M). For example, M may be 16 and N may be greater than or equal to 32.

[0367] ROI regions may be defined for LFNSTMx2M and LFNST2MxM respectively, and LFNSTs with predetermined input and output lengths may be defined / used. The size of the ROI region may be determined based on the minimum value (i.e., M) of the width and height of the current block. The sizes of the ROI regions may differ depending on the value of M. For example, the ROI region of an MxN block may be an Mx2M region within the MxN block. Here, the Mx2M region is the region containing the top-left sample of the current block, and can represent a block with widths and heights of M and 2M, respectively. The ROI region of an NxM block may be a 2MxM region within the NxM block. Here, the 2MxM region is the region containing the top-left sample of the current block, and can represent a block with widths and heights of 2M and M, respectively.

[0368] An LFNSTMx2M or LFNST2MxM may be applied to the current block (or the ROI region of the current block). In this case, at least one of the input length and output length of the LFNST for the LFNSTMx2M or LFNST2MxM may be determined based on at least one of the minimum width and height of the current block or the size of the ROI region. For example, the input length of the LFNST may be less than or equal to (2*M*M), and the output length of the LFNST may be (2*M*M). The input length can mean the number of transformation coefficients input to the LFNST. Here, the transformation coefficients input to the LFNST may mean all of the transformation coefficients derived based on the residual information of the current block, or it may mean the transformation coefficients to which a forward LFNST has been applied among the derived transformation coefficients. The output length can mean the number of transformation coefficients derived by the LFNST. The ROI region of the current block may consist of (2*M*M) sample positions. In this case, the dimension (or size) of the transformation matrix for the reverse LFNSTMx2M or LFNST2MxM may be (2*M*M)xL. That is, the input length of LFNST may be L, and the output length of LFNST may be (2*M*M). Here, the value of L may be set to be less than or equal to (2*M*M).

[0369] If the current block size is MxN, LFNST may be applied to all or part of the transformation coefficients belonging to the Mx2M region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the Mx2M region within the current block according to a predetermined scan order. Alternatively, if the current block size is NxM, LFNST may be applied to all or part of the transformation coefficients belonging to the 2MxM region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the 2MxM region within the current block according to a predetermined scan order.

[0370] The translation kernel for the LFNST may be determined to be one of one or more translation kernels (or LFNST kernel candidates) belonging to the translation set. For this purpose, an index may be used to identify one of the multiple translation kernels belonging to the translation set. Here, the index may be signaled by a bitstream. The translation set of the current block may be determined to be one of a plurality of predefined translation sets. For example, the translation set of the current block may be determined to be one of 35 translation sets, each translation set may contain 3 translation kernels. The translation set of the current block may be determined based on the intra-prediction mode of the current block, as described above. The plurality of translation sets for the LFNST may be defined for LFNSTMx2M and LFNST2MxM, respectively.

[0371] From the perspective of forward transformation, transformation coefficients can be derived by applying a separated linear transformation to the residual samples of the current block. Forward LFNST may be applied only to all or some of the derived transformation coefficients that belong to a region within the current block (i.e., a region of interest (ROI)). The transformation coefficients derived by LFNST may be arranged in the ROI region within the current block according to a predetermined scan order. If the number of transformation coefficients output from LFNST is less than the number of transformation coefficients input to LFNST, there may be regions within the ROI region that are not filled with transformation coefficients derived by LFNST. These regions may be filled with transformation coefficients that are 0. In addition, the remaining regions in the current block excluding the ROI region (or regions to which LFNST is not applied) may be filled with transformation coefficients derived by a separated linear transformation, or they may be filled with transformation coefficients that are 0 due to zero-out.

[0372] Specifically, when LFNSTMx2M or LFNST2MxM is applied to the current block or the ROI region of the current block, an LFNST with an input length of (2*M*M) and an output length less than or equal to (2*M*M) may be used. Here, the input length can mean the number of transformation coefficients input to the LFNST. The input transformation coefficients may be coefficients induced by a separate linear transformation. The input length can also mean the number of transformation coefficients belonging to the ROI region of the current block. The output length can mean the number of transformation coefficients induced by the LFNST. The ROI region of the current block may consist of (2*M*M) sample positions. In this case, the dimension of the transformation matrix for the forward LFNSTMx2M or LFNST2MxM may be Lx(2*M*M). That is, the input length of the LFNST may be (2*M*M), and the output length of the LFNST may be L. Here, the value of L may be set to be less than or equal to (2*M*M).

[0373] If the current block size is MxN, LFNST may be applied to the transformation coefficients belonging to the Mx2M region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the Mx2M region within the current block according to a predetermined scan order. Alternatively, if the current block size is NxM, LFNST may be applied to the transformation coefficients belonging to the 2MxM region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the 2MxM region within the current block according to a predetermined scan order.

[0374] When applying LFNSTMx2M or LFNST2MxM to a current block, the aforementioned symmetries between block configurations may be utilized. For example, the LFNST set and / or LFNST kernel for the current block's LFNST may be determined by the symmetry between an MxN block and an NxM block. An MxN block (or an NxM block) can use the same LNFST set and / or LFNST kernel as an NxM block (or an MxN block) that has symmetry with it. For blocks where N is 2M, the aforementioned symmetries may be used as is, and for blocks where N is greater than 2M, LFNST may be applied to the ROI region (i.e., the Mx2M region or 2MxM region) within the current block using the aforementioned symmetries.

[0375] LFNSTMxrM may be defined for MxrM blocks and LFNSTrMxM may be defined for rMxM blocks, where the value of r may be an integer greater than or equal to 2. rM represents (r*M) and may be considered to have the same meaning hereafter. LFNSTMxrM may be applied to current MxN blocks and LFNSTrMxM may be applied to current NxM blocks, where N may be greater than or equal to (r*M).

[0376] ROI regions may be defined for LFNSTMxrM and LFNSTrMxM respectively, and LFNSTs with predetermined input and output lengths may be defined / used. The size of the ROI region may be determined based on the minimum value (i.e., M) of the width and height of the current block. The sizes of the ROI regions may differ from each other depending on the value of M. For example, the ROI region of an MxN block may be the MxrM region within the MxN block. Here, the MxrM region is the region containing the top-left sample of the current block, and can represent a block with widths and heights of M and rM, respectively. The ROI region of an NxM block may be the rMxM region within the NxM block. Here, the rMxM region is the region containing the top-left sample of the current block, and can represent a block with widths and heights of rM and M, respectively.

[0377] An LFNSTMxrM or LFNSTrMxM may be applied to the current block (or the ROI region of the current block). In this case, at least one of the input length and output length of the LFNST for the LFNSTMxrM or LFNSTrMxM may be determined based on at least one of the minimum width and height of the current block or the size of the ROI region. For example, the input length of the LFNST may be less than or equal to (r*M*M), and the output length of the LFNST may be (r*M*M). The input length can mean the number of transformation coefficients input to the LFNST. Here, the transformation coefficients input to the LFNST may mean all of the transformation coefficients derived based on the current block's residual information, or it may mean the transformation coefficients to which a forward LFNST has been applied among the derived transformation coefficients. The output length can mean the number of transformation coefficients induced by the LFNST. The ROI region of the current block may consist of (r*M*M) sample positions. In this case, the dimension (or size) of the transformation matrix for the reverse LFNSTMxrM or LFNSTrMxM may be (r*M*M)xL. That is, the input length of LFNST may be L, and the output length of LFNST may be (r*M*M). Here, the value of L may be set to be less than or equal to (r*M*M).

[0378] If the current block size is MxN, LFNST may be applied to all or part of the transformation coefficients belonging to the MxrM region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the MxrM region within the current block according to a predetermined scan order. Alternatively, if the current block size is NxM, LFNST may be applied to all or part of the transformation coefficients belonging to the rMxM region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the rMxM region within the current block according to a predetermined scan order.

[0379] The translation kernel for the LFNST may be determined to be one of one or more translation kernels (or LFNST kernel candidates) belonging to the translation set. For this purpose, an index may be used to identify one of the multiple translation kernels belonging to the translation set. Here, the index may be signaled by a bitstream. The translation set of the current block may be determined to be one of a plurality of predefined translation sets. For example, the translation set of the current block may be determined to be one of 35 translation sets, each translation set may contain 3 translation kernels. The translation set of the current block may be determined based on the intra-prediction mode of the current block, as described above. The plurality of translation sets for the LFNST may be defined for LFNSTMxrM and LFNSTrMxM, respectively.

[0380] From the perspective of forward transformation, transformation coefficients can be derived by applying a separated linear transformation to the residual samples of the current block. Forward LFNST may be applied only to all or some of the derived transformation coefficients that belong to a region within the current block (i.e., a region of interest (ROI)). The transformation coefficients derived by LFNST may be arranged in the ROI region within the current block according to a predetermined scan order. If the number of transformation coefficients output from LFNST is less than the number of transformation coefficients input to LFNST, there may be regions within the ROI region that are not filled with transformation coefficients derived by LFNST. These regions may be filled with transformation coefficients that are 0. In addition, the remaining regions in the current block excluding the ROI region (or regions to which LFNST is not applied) may be filled with transformation coefficients derived by a separated linear transformation, or they may be filled with transformation coefficients that are 0 due to zero-out.

[0381] Specifically, when LFNSTMxrM or LFNSTrMxM is applied to the current block or the ROI region of the current block, an LFNST may be used with an input length of (r*M*M) and an output length less than or equal to (r*M*M). Here, the input length can mean the number of transformation coefficients input to the LFNST. The input transformation coefficients may be coefficients induced by a separate linear transformation. The input length can also mean the number of transformation coefficients belonging to the ROI region of the current block. The output length can mean the number of transformation coefficients induced by the LFNST. The ROI region of the current block may consist of (r*M*M) sample positions. In this case, the dimension of the transformation matrix for the forward LFNSTMxrM or LFNSTrMxM may be Lx(r*M*M). That is, the input length of the LFNST may be (r*M*M), and the output length of the LFNST may be L. Here, the value of L may be set to be less than or equal to (r*M*M).

[0382] If the current block size is MxN, LFNST may be applied to the transformation coefficients belonging to the MxrM region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the MxrM region within the current block according to a predetermined scan order. Alternatively, if the current block size is NxM, LFNST may be applied to the transformation coefficients belonging to the rMxM region, which is the ROI region within the current block. The transformation coefficients induced by LFNST may be arranged in the rMxM region within the current block according to a predetermined scan order.

[0383] When applying LFNSTMxrM or LFNSTrMxM to a block, the aforementioned symmetries between block configurations may be utilized. For example, the symmetry between an MxN block and an NxM block may determine the LFNST set and / or LFNST kernel for the current block's LFNST. An MxN block (or an NxM block) can use the same LNFST set and / or LFNST kernel as an NxM block (or an MxN block) that has symmetry with it. For blocks where N is rM, the aforementioned symmetries may be used directly, and for blocks where N is greater than rM, LFNST may be applied to the ROI region (i.e., the MxrM region or rMxM region) within the current block using the aforementioned symmetries.

[0384] For example, if the value of M is 4 and the value of r is 4, then LFNST4x16 and LFNST16x4 may be defined. LFNST4x16 may be applied to a 4xN current block, and LFNST16x4 may be applied to an Nx4 current block. Here, N may be greater than or equal to 16. If the current block is a 4xN block, the ROI region of the current block may be a 4x16 region within the current block. Here, the 4x16 region is the region containing the top-left sample of the current block, and can represent a block with a width and height of 4 and 16, respectively. LFNST4x16 may be applied to the 4x16 region that is the ROI region within the current block. Alternatively, if the current block is an Nx4 block, the ROI region of the current block may be a 16x4 region within the current block. Here, the 16x4 region is the region containing the top-left sample of the current block, and can represent a block with a width and height of 16 and 4, respectively. LFNST16x4 can now be applied to the 16x4 region, which is currently the ROI area within the block.

[0385] Based on the current block size, the aforementioned LFNSTMxrM (or LFNSTrMxM) may be adaptively applied to the current block.

[0386] Specifically, if either the width or height of the current block is smaller than a predetermined threshold size, LFNSTMxrM (or LFNSTrMxM) may be applied to the current block. On the other hand, if either the width or height of the current block is greater than or equal to the predetermined threshold size, LFNSTMxrM (or LFNSTrMxM) may not be applied to the current block. If either the width or height of the current block is greater than or equal to the predetermined threshold size, NSPT may be applied to the current block. As an example, either the width or height of the current block may be defined as the minimum value of the width or height of the current block. The threshold size may be defined as 8. In this case, if the current block is a 4xN block, LFNST4x8 may be applied to the current block. Or, if the current block is an Nx4 block, LFNST8x4 may be applied to the current block. Here, N may be an integer greater than or equal to 8. On the other hand, if the current block is an 8x16 block (or a 16x8 block), LFNST8x16 (or LFNST16x8) does not need to be applied to the current block. If the current block is an 8x16 block (or a 16x8 block), NSPT may be applied to the current block.

[0387] Alternatively, if either the width or height of the current block is greater than or equal to a predetermined threshold size, LFNSTMxrM (or LFNSTrMxM) may be applied to the current block. On the other hand, if either the width or height of the current block is smaller than the predetermined threshold size, LFNSTMxrM (or LFNSTrMxM) does not need to be applied to the current block. If either the width or height of the current block is smaller than the predetermined threshold size, NSPT may be applied to the current block. As an example, either the width or height of the current block may be defined as the maximum value of the width or height of the current block. The threshold size may be defined as 16. In this case, if the current block is a 4xL block, LFNST4x16 may be applied to the current block. Alternatively, if the current block is an Lx4 block, LFNST16x4 may be applied to the current block. Here, L may be an integer greater than or equal to 16. On the other hand, if the current block is a 4x8 block (or an 8x4 block), LFNST4x8 (or LFNST8x4) does not need to be applied to the current block. If the current block is a 4x8 block (or an 8x4 block), NSPT may be applied to the current block.

[0388] Alternatively, if either the width or height of the current block is greater than or equal to a predetermined threshold size, LFNSTMxrM (or LFNSTrMxM) may be applied to the current block. On the other hand, if either the width or height of the current block is smaller than the predetermined threshold size, LFNSTMxrM (or LFNSTrMxM) does not need to be applied to the current block. If either the width or height of the current block is smaller than the predetermined threshold size, NSPT may be applied to the current block. As an example, either the width or height of the current block may be defined as the minimum value of the width or height of the current block. The threshold size may be defined as 8. In this case, if the current block is an 8xN block, LFNST8x16 may be applied to the current block. Alternatively, if the current block is an Nx8 block, LFNST16x8 may be applied to the current block. Here, N may be an integer greater than or equal to 16. On the other hand, if the current block is a 4x8 block, 8x4 block, 4x16 block, or 16x4 block, NSPT may be applied to the current block.

[0389] The ROI regions of LFNST4x8 and LFNST8x4 may be configured identically to those of LFNST4. Here, LFNST4 refers to LFNST applied to a 4x4 block, and the ROI region of LFNST4 may be a 4x4 block. That is, when LFNST is applied to a 4xN or Nx4 block to which LFNST4x8 or LFNST8x4 can be applied, 16 conversion coefficients are output, and these 16 output conversion coefficients may be arranged in a 4x4 region within the 4xN or Nx4 block. Here, N may be an integer greater than or equal to 8. For example, N may be an integer of 8, 16, 32, or more. The aforementioned 4x4 region is the region containing the top-left sample within the 4xN or Nx4 block, and may be a region with a width and height of 4 and 4, respectively.

[0390] The dimensions (or size) of the transformation matrices for LFNST4x8 and LFNST8x4 may be configured to be the same as those for LFNST4. For example, the dimensions of the transformation matrices for LFNST4x8 and LFNST8x4 may be 16x16 matrices. That is, when LFNST is applied to a 4xN or Nx4 block, 16 transformation coefficients belonging to the ROI region (i.e., the 4x4 region) within the 4xN or Nx4 block may be input to LFNST, and 16 transformation coefficients may be output by LFNST. However, the dimensions of the transformation matrices for LFNST4x8 and LFNST8x4 may be configured to be different from those for LFNST4. For example, the input length of LFNST for a 4x4 block may be different from the input length of LFNST for a 4xN or Nx4 block, and the output length of LFNST for a 4x4 block may be the same as the output length of LFNST for a 4xN or Nx4 block. As mentioned above, the input or output length of an LFNST can mean the number of conversion coefficients input to or output by the LFNST.

[0391] From the perspective of forward transformation, when LFNST is applied to a 4xN or Nx4 block to which LFNST4x8 or LFNST8x4 can be applied, 16 transformation coefficients belonging to a 4x4 region within the 4xN or Nx4 block may be input to LFNST. Here, N is an integer greater than or equal to 8, as mentioned above. For example, N may be an integer of 8, 16, 32 or more. The aforementioned 4x4 region is the region containing the top-left sample within the 4xN or Nx4 block, and may be a region with a width and height of 4 and 4, respectively.

[0392] The dimensions (or size) of the transformation matrices for LFNST4x8 and LFNST8x4 may be the same as those for LFNST4. For example, the dimensions of the transformation matrices for LFNST4x8 and LFNST8x4 may be 16x16 matrices. That is, when LFNST is applied to a 4xN or Nx4 block, 16 transformation coefficients belonging to the ROI region (i.e., the 4x4 region) within the 4xN or Nx4 block may be input to LFNST, and 16 transformation coefficients may be output by LFNST. However, this is not limited to this, and the dimensions of the transformation matrices for LFNST4x8 and LFNST8x4 may be configured to be different from those for LFNST4. For example, the input length of LFNST for a 4x4 block may be the same as the input length of LFNST for a 4xN or Nx4 block, and the output length of LFNST for a 4x4 block may be different from the output length of LFNST for a 4xN or Nx4 block.

[0393] At least one of the following—the conversion sets for LFNST4x8 and LFNST8x4, the number of conversion kernels belonging to each conversion set, or the determination of the conversion sets—may be configured identically to that of LFNST4. In other words, the conversion sets for 4x4 blocks, the number of conversion kernels belonging to each conversion set, or the method for determining the conversion sets may be applied identically to the aforementioned 4xN or Nx4 blocks.

[0394] The ROI regions of LFNST8x16 and LFNST16x8 may be configured identically to those of LFNST8. Here, LFNST8 refers to LFNST applied to an 8x8 block, and the ROI region of LFNST8 may be an 8x8 block. That is, when LFNST is applied to an 8xN or Nx8 block to which LFNST8x16 or LFNST16x8 can be applied, 64 conversion coefficients are output, and these 64 output conversion coefficients may be arranged in an 8x8 region within the 8xN or Nx8 block. Here, N is an integer greater than or equal to 16, as mentioned above. For example, N may be an integer of 16, 32, 64, or more. The 8x8 region is the region containing the top-left sample within the 8xN or Nx8 block, and may be a region with a width and height of 8 and 8, respectively.

[0395] The dimensions (or size) of the transformation matrices for LFNST8x16 and LFNST16x8 may be the same as those for LFNST8. For example, the dimensions of the transformation matrices for LFNST8x16 and LFNST16x8 may be 64x32 matrices. That is, when LFNST is applied to an 8xN or Nx8 block, 32 transformation coefficients belonging to the ROI region (i.e., the 8x8 region) within the 8xN or Nx8 block may be input to LFNST, and 64 transformation coefficients may be output by LFNST. However, this is not limited to this, and the dimensions of the transformation matrices for LFNST8x16 and LFNST16x8 may be different from those for LFNST8. For example, the input length of LFNST for an 8x8 block may be different from the input length of LFNST for an 8xN or Nx8 block, and the output length of LFNST for an 8x8 block may be the same as the output length of LFNST for an 8xN or Nx8 block. The dimension of the transformation matrix for LFNST8 is 64x16, and the dimension of the transformation matrix for LFNST8x16 and LFNST16x8 may be a 64x32 matrix. That is, the input length and output length of LFNST for an 8x8 block may be 16 and 64, respectively. On the other hand, the input length and output length of LFNST for an 8xN or Nx8 block may be 32 and 64, respectively. This means that 32 transformation coefficients belonging to the ROI region (i.e., the 8x8 region) within the 8xN or Nx8 block may be input to LFNST, and 64 transformation coefficients may be output by LFNST.

[0396] From the perspective of forward transformation, when LFNST is applied to an 8xN or Nx8 block to which LFNST8x16 or LFNST16x8 can be applied, 64 transformation coefficients belonging to the 8x8 region within the 8xN or Nx8 block may be input to LFNST. Here, N is an integer greater than or equal to 16, as mentioned above. For example, N may be an integer of 16, 32, 64, or more. The 8x8 region is the region containing the top-left sample within the 8xN or Nx8 block, and may be a region with a width and height of 8 and 8, respectively.

[0397] The dimensions (or size) of the transformation matrices for LFNST8x16 and LFNST16x8 may be the same as those for LFNST8. For example, the dimensions of the transformation matrices for LFNST8x16 and LFNST16x8 may be 32x64 matrices. That is, when LFNST is applied to an 8xN or Nx8 block, 64 transformation coefficients belonging to the ROI region (i.e., the 8x8 region) within the 8xN or Nx8 block may be input to LFNST, and 32 transformation coefficients may be output by LFNST. However, this is not limited to this, and the dimensions of the transformation matrices for LFNST8x16 and LFNST16x8 may be configured to be different from those for LFNST8. For example, the input length of LFNST for an 8x8 block may be the same as the input length of LFNST for an 8xN or Nx8 block, and the output length of LFNST for an 8x8 block may be different from the output length of LFNST for an 8xN or Nx8 block. The dimension of the transformation matrix for LFNST8 is 16x64, and the dimension of the transformation matrix for LFNST8x16 and LFNST16x8 may be a 32x64 matrix. That is, the input length and output length of LFNST for an 8x8 block may be 64 and 16, respectively. On the other hand, the input length and output length of LFNST for an 8xN or Nx8 block may be 64 and 32, respectively. This means that 64 transformation coefficients belonging to the ROI region (i.e., the 8x8 region) within the 8xN or Nx8 block may be input to LFNST, and 32 transformation coefficients may be output by LFNST.

[0398] At least one of the following—the conversion sets for LFNST8x16 and LFNST16x8, the number of conversion kernels belonging to each conversion set, or the determination of the conversion sets—may be configured identically to that of LFNST8. In other words, the conversion sets for 8x8 blocks, the number of conversion kernels belonging to each conversion set, or the method for determining the conversion sets may be applied identically to the aforementioned 8xN or Nx8 blocks.

[0399] The ROI areas of LFNST16x32 and LFNST32x16 may be configured identically to those of LFNST16. Here, LFNST16 refers to an LFNST applied to a 16x16 block, and the ROI area of ​​LFNST16 may consist of one or more 4x4 blocks. For example, the ROI area of ​​LFNST16 may consist of six 4x4 blocks within a 16x16 block. Here, the six 4x4 blocks can refer to the first to sixth 4x4 blocks according to a predetermined scan order. That is, the ROI area of ​​LFNST16 may consist of 96 sample positions. The scan order refers to a diagonal scan order, but is not limited to this; vertical scans, horizontal scans, etc., may also be used.

[0400] In other words, when LFNST is applied to a 16xN or Nx16 block, which is a block to which LFNST16x32 or LFNST32x16 can be applied, 96 conversion coefficients are output, and these 96 output conversion coefficients may be arranged in the ROI area within the 16xN or Nx16 block. Here, as mentioned above, N is an integer greater than or equal to 32. For example, N may be 32, 64, or any integer greater than or equal to 32.

[0401] The dimensions (or size) of the transformation matrices for LFNST16x32 and LFNST32x16 may be the same as those for LFNST16. For example, the dimensions of the transformation matrices for LFNST16x32 and LFNST32x16 may be 96x32 matrices. That is, when LFNST is applied to a 16xN or Nx16 block, 32 transformation coefficients belonging to the ROI region within the 16xN or Nx16 block may be input to LFNST, and 96 transformation coefficients may be output by LFNST. However, the dimensions of the transformation matrices for LFNST16x32 and LFNST32x16 may be different from those for LFNST16. For example, the input length of the LFNST for a 16x16 block may be different from the input length of the LFNST for a 16xN or Nx16 block, and the output length of the LFNST for a 16x16 block may be the same as the output length of the LFNST for a 16xN or Nx16 block.

[0402] From the perspective of forward transformation, when LFNST is applied to a 16xN or Nx16 block to which LFNST16x32 or LFNST32x16 can be applied, 96 transformation coefficients belonging to the ROI region within the 16xN or Nx16 block may be input to LFNST. Here, as mentioned above, N is an integer greater than or equal to 32. For example, N may be an integer of 32, 64, or any greater.

[0403] The dimensions (or size) of the transformation matrices for LFNST16x32 and LFNST32x16 may be configured to be the same as those for LFNST16. For example, the dimensions of the transformation matrices for LFNST16x32 and LFNST32x16 may be 32x96 matrices. That is, when LFNST is applied to a 16xN or Nx16 block, 96 transformation coefficients belonging to the ROI region within the 16xN or Nx16 block may be input to LFNST, and 32 transformation coefficients may be output by LFNST. However, this is not limited to this, and the dimensions of the transformation matrices for LFNST16x32 and LFNST32x16 may be configured to be different from those for LFNST16. For example, the input length of the LFNST for a 16x16 block may be the same as the input length of the LFNST for a 16xN or Nx16 block, while the output length of the LFNST for a 16x16 block may be different from the output length of the LFNST for a 16xN or Nx16 block.

[0404] At least one of the following—the conversion sets for LFNST16x32 and LFNST32x16, the number of conversion kernels belonging to each conversion set, or the determination of the conversion sets—may be configured identically to that of LFNST16. In other words, the conversion sets for 16x16 blocks, the number of conversion kernels belonging to each conversion set, or the method for determining the conversion sets may be applied identically to the aforementioned 16xN or Nx16 blocks.

[0405] For MxN blocks, an LFNST for square blocks or an LFNST for non-square blocks may be applied. Here, M and N may each be integers greater than or equal to 32. M and N may be the same or different from each other. The LFNST for square blocks may mean the LFNST16 described above. However, it is not limited to this, and the LFNST for square blocks may also mean LFNST4 or LFNST8. The LFNST for non-square blocks may mean the LFNST16x32 (or LFNST32x16) described above. However, it is not limited to this. As an example, the LFNST for non-square blocks may also mean the LFNST8x16 (or LFNST16x8) described above, or LFNST4x8 (or LFNST8x4). The LFNST for square blocks may be defined as an LFNST having the same output length as the LFNST for non-square blocks and a different input length than the LFNST for non-square blocks. Alternatively, the LFNST for the square block may be defined as an LFNST having different input and output lengths than the LFNST for the non-square block. From the viewpoint of forward conversion, the LFNST for the square block may be defined as an LFNST having the same input length as the LFNST for the non-square block and different output lengths than the LFNST for the non-square block. Alternatively, the LFNST for the square block may be defined as an LFNST having different input and output lengths than the LFNST for the non-square block.

[0406] Specifically, LFNST for square blocks may be applied to all blocks where M and N are greater than or equal to 32. For example, LFNST16 may be applied to all blocks where M and N are greater than or equal to 32. Alternatively, LFNST8 may be applied to all blocks where M and N are greater than or equal to 32. Alternatively, LFNST4 may be applied to all blocks where M and N are greater than or equal to 32.

[0407] Alternatively, LFNST for square blocks may be applied to blocks where M and N are greater than or equal to 32, and M and N are the same. On the other hand, LFNST for non-square blocks may be applied to blocks where M and N are greater than or equal to 32, and M and N are different. For example, LFNST16 may be applied to 32x32, 64x64, or 128x128 blocks. Alternatively, LFNST8 may be applied to 32x32, 64x64, or 128x128 blocks. Alternatively, LFNST4 may be applied to 32x32, 64x64, or 128x128 blocks. LFNST16x32 may be applied to 32x64, 32x128, or 64x128 blocks. Alternatively, LFNST8x16 may be applied to 32x64, 32x128, or 64x128 blocks. Alternatively, LFNST4x8 may be applied to 32x64 blocks, 32x128 blocks, or 64x128 blocks. LFNST32x16 may be applied to 64x32 blocks, 128x32 blocks, or 128x64 blocks. Alternatively, LFNST16x8 may be applied to 64x32 blocks, 128x32 blocks, or 128x64 blocks. Alternatively, LFNST8x4 may be applied to 64x32 blocks, 128x32 blocks, or 128x64 blocks.

[0408] The allowed transform block sizes predefined for a decoding device may be distinguished into multiple groups. The multiple groups relating to this disclosure may include at least two of the following: Group_4xN_Nx4, Group_8xN_Nx8, Group_16xN_Nx16, Group_GTE16x16, Group_GTE32x32, or Group_GTE8x8. Group_4xN_Nx4 may be a group containing at least one 4xN block or an Nx4 block, where N may be an integer greater than or equal to 4. Group_4xN_Nx4 may be a group containing block sizes where the smaller of the width and height (i.e., the minimum value) is 4. Group_8xN_Nx8 may be a group containing at least one 8xN block or an Nx8 block, where N may be an integer greater than or equal to 8. Group_8xN_Nx8 may be a group containing block sizes where the smaller of the width and height (i.e., the minimum value) is 8. Group_16xN_Nx16 may be a group containing at least one 16xN block or an Nx16 block, where N may be an integer greater than or equal to 16. Group_16xN_Nx16 may be a group containing block sizes where the smaller of the width and height (i.e., the minimum value) is 16. Group_GTE16x16 may be a group containing MxN blocks, where M and N may each be integers greater than or equal to 16. Group_GTE16x16 may be a group containing block sizes greater than or equal to 16. Group_GTE32x32 may be a group containing MxN blocks, where M and N may each be integers greater than or equal to 32. Group_GTE32x32 may be a group containing block sizes greater than or equal to 32x32. Group_GTE8x8 may be a group containing MxN blocks, where M and N may each be integers greater than or equal to 8. Group_GTE8x8 may be a group containing blocks with a size greater than or equal to 8x8.

[0409] For example, the permissible conversion block sizes may be distinguished as combinations of Group_4xN_Nx4, Group_8xN_Nx8, and Group_GTE16x16. Alternatively, the permissible conversion block sizes may be distinguished as combinations of Group_4xN_Nx4, Group_8xN_Nx8, Group_16xN_Nx16, and Group_GTE32x32. Alternatively, the permissible conversion block sizes may be distinguished as combinations of Group_4xN_Nx4 and Group_GTE8x8.

[0410] Each group can be considered a set that has one or more block sizes belonging to that group as elements. For each block size belonging to that group, at least one of NSPT or LFNST may be applied, or neither NSPT nor LFNST may be applied. The conversion kernel for block sizes to which neither NSPT nor LFNST is applied may be determined by DCT-2 or may be determined based on the MTS described above. When LFNST is applied, LFNST based on block size may be applied. For example, LFNST4, LFNST8, or LFNST16 described above may be applied, or LFNST4x8, LFNST8x4, LFNST8x16, or LFNST16x8 may be applied. With the above settings, each group may be defined as one of the following:

[0411] a) A set of one or more block sizes to which NSPT can be applied.

[0412] b) A set of one or more block sizes to which NSPT can be applied, and one or more block sizes to which NSPT and LFNST cannot be applied.

[0413] c) A set of one or more block sizes to which NSPT can be applied, and one or more block sizes to which LFNST can be applied.

[0414] d) A set of one or more block sizes to which NSPT can be applied, one or more block sizes to which LFNST can be applied, and one or more block sizes to which neither NSPT nor LFNST can be applied.

[0415] e) A set of one or more block sizes to which LFNST can be applied.

[0416] f) A set of one or more block sizes to which LFNST can be applied, and one or more block sizes to which NSPT and LFNST cannot be applied.

[0417] g) A set of one or more block sizes to which NSPT and LFNST do not apply.

[0418] In the case of a group defined in a) above, the group may include one or more block sizes for which NSPT is permitted. For example, NSPT may be permitted for block sizes belonging to the group, but LFNST may not be permitted. In this case, LFNST may not be required for the group defined in a).

[0419] In the case of a group defined in (b) above, the group may include one or more block sizes for which NSPT is permitted, and one or more block sizes for which neither NSPT nor LFNST is permitted. For example, NSPT may be permitted for some block sizes belonging to the group, while LFNST may not be permitted. Also, both NSPT and LFNST may not be permitted for some other block sizes belonging to the group. In this case, LFNST may not be required for the group defined in (b).

[0420] In the case of a group defined in c) above, the group may include one or more block sizes for which at least one of NSPT or LFNST is permitted. For example, NSPT may be permitted for some block sizes belonging to the group, while LFNST may not be permitted. LFNST may be permitted for other block sizes belonging to the group, while NSPT may not be permitted.

[0421] In the case of a group defined in d) above, the group may include one or more block sizes in which at least one of NSPT or LFNST is permitted, and one or more block sizes in which neither NSPT nor LFNST is permitted. For example, NSPT may be permitted and LFNST may not be permitted for some block sizes belonging to the group. LFNST may be permitted and NSPT may not be permitted for other block sizes belonging to the group. Neither NSPT nor LFNST may be permitted for yet another set of block sizes belonging to the group.

[0422] In the case of a group defined in e) above, the group may include one or more block sizes for which LFNST is permitted. LFNST may be permitted for block sizes belonging to the group, but NSPT may not be permitted. In this case, NSPT may not be required for the group defined in e).

[0423] In the case of a group defined in f) above, the group may include one or more block sizes for which LFNST is permitted, and one or more block sizes for which neither NSPT nor LFNST is permitted. For example, LFNST may be permitted for some block sizes belonging to the group, while NSPT may not be permitted. Also, neither NSPT nor LFNST may be permitted for some other block sizes belonging to the group. In this case, NSPT may not be required for the group defined in f).

[0424] In the case of a group defined in g) above, the group may include one or more block sizes for which NSPT and LFNST are not permitted. For example, NSPT and LFNST may not be permitted for all block sizes belonging to the group. In this case, both NSPT and LFNST may not be required for the group defined in g).

[0425] Removing the unnecessary conversions (or conversion kernels, conversion kernel data) described above reduces the memory space required to store the conversions. Furthermore, since hardware circuits and software code for performing the conversions can be removed, the complexity of implementation can be significantly reduced. Because the encoding device does not attempt the conversions, the time required for encoding can be significantly reduced. The removal of conversions according to this disclosure means that the conversion kernels (or conversion kernel data) for those conversions are not stored or are unavailable. Various embodiments of removing unnecessary conversions are described below.

[0426] 1. In Group_4xN_Nx4, NSPT is applied / allowed for 4xL blocks and Lx4 blocks (4≦L≦32), while neither NSPT nor LFNST is applied / allowed for 4xM blocks and Mx4 blocks (M>32). In this case, LFNST4, LFNST4x8, and / or LFNST8x4 may be removed. In other words, the translation kernels for LFNST4, LFNST4x8, and / or LFNST8x4 may not be saved or available. The translation kernels for LFNST4, LFNST4x8, and / or LFNST8x4 may not be saved or available if LFNST4, LFNST4x8, and / or LFNST8x4 are not applied / allowed in other groups other than Group_4xN_Nx4.

[0427] 2. NSPT does not need to be applied to / allowed for Group_4xN_Nx4. That is, for block sizes belonging to Group_4xN_Nx4, LFNST such as LFNST4, LFNST4x8, or LFNST8x4 may be applied to / allowed, or neither NSPT nor LFNST may be applied to / allowed. In this case, NSPT for Group_4xN_Nx4 may be removed. In other words, the NSPT conversion kernel for Group_4xN_Nx4 does not need to be saved or available.

[0428] 3. In Group_4xN_Nx4, NSPT may be applied / allowed for 4x4 blocks, and LFNST4x8 and LFNST8x4 may be applied / allowed for 4xL blocks and Lx4 blocks (L≧8), or neither NSPT nor LFNST may be applied / allowed. In this case, LFNST4 may be removed. In other words, the LFNST4 translation kernel may not be saved or available. The LFNST4 translation kernel may not be saved or available if LFNST4 is not applied / allowed in any other group besides Group_4xN_Nx4.

[0429] 4. In Group_8xN_Nx8, NSPT is applied / allowed for 8xL blocks and Lx8 blocks (8≦L≦32), while neither NSPT nor LFNST is applied / allowed for 8xM blocks and Mx8 blocks (M>32). In this case, LFNST8, LFNST8x16, and / or LFNST16x8 may be removed. In other words, the translation kernels for LFNST8, LFNST8x16, and / or LFNST16x8 may not be saved or available. The translation kernels for LFNST8, LFNST8x16, and / or LFNST16x8 may not be saved or available if LFNST8, LFNST8x16, and / or LFNST16x8 are not applied / allowed in other groups other than Group_8xN_Nx8.

[0430] 5. NSPT does not need to be applied to / allowed for Group_8xN_Nx8. That is, for block sizes belonging to Group_8xN_Nx8, LFNST such as LFNST8, LFNST8x16, and LFNST16x8 may be applied to / allowed, or neither NSPT nor LFNST may be applied to / allowed. In this case, NSPT for Group_8xN_Nx8 may be removed. In other words, the NSPT conversion kernel for Group_8xN_Nx8 does not need to be saved or available.

[0431] 6. In Group_8xN_Nx8, NSPT may be applied / allowed for 8x8 blocks, and LFNST8x16 and LFNST16x8 may be applied / allowed for 8xL blocks and Lx8 blocks (L≧16), or neither NSPT nor LFNST may be applied / allowed. In this case, LFNST8 may be removed. In other words, the LFNST8 translation kernel may not be saved or available. The LFNST8 translation kernel may not be saved or available if LFNST8 is not applied / allowed in any other group besides Group_8xN_Nx8.

[0432] 7. NSPT does not need to be applied to / allowed for Group_16xN_Nx16. That is, for block sizes belonging to Group_16xN_Nx16, LFNST such as LFNST16, LFNST16x32, and LFNST32x16 may be applied to / allowed, or neither NSPT nor LFNST may be applied to / allowed. In this case, NSPT for Group_16xN_Nx16 may be removed. In other words, the NSPT conversion kernel for Group_16xN_Nx16 does not need to be saved or available.

[0433] 8. NSPT does not need to be applied to / allowed for Group_GTE16x16. That is, for block sizes belonging to Group_GTE16x16, LFNST such as LFNST8, LFNST16, LFNST16x32, and LFNST32x16 may be applied to / allowed, or neither NSPT nor LFNST may be applied to / allowed. In this case, NSPT for Group_GTE16x16 may be removed. In other words, the NSPT translation kernel for Group_GTE16x16 does not need to be saved or available.

[0434] 9. NSPT does not need to be applied to / allowed for Group_GTE32x32. That is, LFNST such as LFNST8 and LFNST16 may be applied to / allowed for block sizes belonging to Group_GTE32x32, or neither NSPT nor LFNST may be applied / allowed. In this case, NSPT for Group_GTE32x32 may be removed. In other words, the NSPT translation kernel for Group_GTE32x32 does not need to be saved or available.

[0435] 10. NSPT does not need to be applied to / allowed for Group_GTE8x8. That is, for block sizes belonging to Group_GTE8x8, LFNST such as LFNST8, LFNST8x16, and LFNST16x8 may be applied to / allowed, or neither NSPT nor LFNST may be applied to / allowed. In this case, NSPT for Group_GTE8x8 may be removed. In other words, the NSPT translation kernel for Group_GTE8x8 does not need to be saved or available.

[0436] 11. In Group_4xN_Nx4, at least one of NSPT or LFNST4 may be applied to / allowed for 4x4 blocks, or neither NSPT nor LFNST may be applied to / allowed. On the other hand, in Group_4xN_Nx4, LFNST4x8 and LFNST8x4 may be applied to / allowed for 4xL blocks and Lx4 blocks (L≧8), or LFNST4 may be applied to / allowed. Alternatively, neither NSPT nor LFNST may be applied to / allowed for 4xL blocks and Lx4 blocks (L≧8).

[0437] 12. In Group_4xN_Nx4, at least one of NSPT or LFNST4 may be applied to / allowed for 4x4 blocks, or neither NSPT nor LFNST may be applied to / allowed. In Group_4xN_Nx4, NSPT may be applied to / allowed for 4x8 blocks and 8x4 blocks, or LFNST4x8 and LFNST8x4 may be applied to / allowed. Alternatively, neither NSPT nor LFNST may be applied to / allowed for 4x8 blocks and 8x4 blocks. In Group_4xN_Nx4, LFNST4x16 and LFNST16x4 may be applied to / allowed for 4xL blocks and Lx4 blocks (L≧16), or LFNST4 may be applied to / allowed. Alternatively, neither NSPT nor LFNST may be applied to / allowed for 4xL blocks and Lx4 blocks (L≧16).

[0438] 13. In Group_8xN_Nx8, at least one of NSPT or LFNST8 may be applied to / allowed for 8x8 blocks, or neither NSPT nor LFNST may be applied to / allowed. In Group_8xN_Nx8, LFNST8x16 and LFNST16x8 may be applied to / allowed for 8xL blocks and Lx8 blocks (L≧16), or LFNST8 may be applied to / allowed. Alternatively, neither NSPT nor LFNST may be applied to / allowed for 8xL blocks and Lx8 blocks (L≧16).

[0439] According to the combinations of 1, 4, and 8 mentioned above, LFNST4 and LFNST8 can be eliminated. In other words, the conversion kernels for LFNST4 and LFNST8 do not need to be saved or available.

[0440] Specifically, NSPT may be applied / allowed for 4x4, 4x8, 4x16, 4x32, 8x4, 16x4, and 32x4 blocks, while neither NSPT nor LFNST may be applied / allowed for 4xL and Lx4 blocks (L≧64). NSPT may be applied / allowed for 8x8, 8x16, 8x32, 16x8, and 32x8 blocks, while neither NSPT nor LFNST may be applied / allowed for 8xM and Mx8 blocks (M≧64). LFNST8 or LFNST16 may be applied / allowed for PxQ blocks (P≧16, Q≧16). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), the system may be configured so that neither NSPT nor LFNST applies to or is permitted for that block size.

[0441] According to the combinations of numbers 3, 4, and 8 mentioned above, LFNST4 and LFNST8 may be removed, and LFNST4x8 and LFNST8x4 may be included. That is, the conversion kernels for LFNST4 and LFNST8 may not be saved or available, while the conversion kernels for LFNST4x8 and LFNST8x4 may be saved or available.

[0442] Specifically, NSPT may be applied / allowed for 4x4 blocks, and LFNST4x8 and LFNST8x4 may be applied / allowed for 4xL and Lx4 blocks (8≦L). NSPT may be applied / allowed for 8x8, 8x16, 8x32, 16x8, and 32x8 blocks, while neither NSPT nor LFNST may be applied / allowed for 8xM and Mx8 blocks (M≧64). LFNST8 or LFNST16 may be applied / allowed for PxQ blocks (P≧16, Q≧16). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0443] According to the combinations of points 1, 5, and 8 mentioned above, LFNST4 can be eliminated. That is, the LFNST4 conversion kernel does not need to be saved or available.

[0444] Specifically, NSPT may be applied / allowed for 4x4, 4x8, 4x16, 4x32, 8x4, 16x4, and 32x4 blocks, while neither NSPT nor LFNST may be applied / allowed for 4xL and Lx4 blocks (L≧64). LFNST8 may be applied / allowed for 8xM and Mx8 blocks (M≧8). LFNST8 or LFNST16 may be applied / allowed for PxQ blocks (P≧16, Q≧16). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0445] According to the combinations of numbers 3, 5, and 8 mentioned above, LFNST4 may be removed, and LFNST4x8 and LFNST8x4 may be included. That is, the conversion kernel for LFNST4 may not be saved or available, while the conversion kernels for LFNST4x8 and LFNST8x4 may be saved or available.

[0446] Specifically, NSPT may be applied / allowed for 4x4 blocks, LFNST4x8 and LFNST8x4 may be applied / allowed for 4xL and Lx4 blocks (8≦L), respectively. LFNST8 may be applied / allowed for 8xM and Mx8 blocks (M≧8). LFNST8 or LFNST16 may be applied / allowed for PxQ blocks (P≧16, Q≧16). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0447] According to the combinations of numbers 1, 6, and 8 mentioned above, LFNST4 and LFNST8 may be removed, and LFNST8x16 and LFNST16x8 may be included. That is, the conversion kernels for LFNST4 and LFNST8 may not be saved or available, while the conversion kernels for LFNST8x16 and LFNST16x8 may be saved or available.

[0448] Specifically, NSPT may be applied / allowed for 4x4, 4x8, 4x16, 4x32, 8x4, 16x4, and 32x4 blocks, while neither NSPT nor LFNST may be applied / allowed for 4xL and Lx4 blocks (L≧64). NSPT may be applied / allowed for 8x8 blocks, while LFNST8x16 and LFNST16x8 may be applied / allowed for 8xM and Mx8 blocks (M≧16). LFNST8 or LFNST16 may be applied to PxQ blocks (P≧16, Q≧16). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0449] According to the combinations of numbers 3, 6, and 8 mentioned above, LFNST4 and LFNST8 may be removed, and LFNST4x8, LFNST8x4, LFNST8x16, and LFNST16x8 may be included. That is, the conversion kernels for LFNST4 and LFNST8 may not be saved or available, while the conversion kernels for LFNST4x8, LFNST8x4, LFNST8x16, and LFNST16x8 may be saved or available.

[0450] Specifically, NSPT may be applied / allowed for 4x4 blocks, and LFNST4x8 and LFNST8x4 may be applied / allowed for 4xL and Lx4 blocks (8≦L). NSPT may be applied / allowed for 8x8 blocks, and LFNST8x16 and LFNST16x8 may be applied / allowed for 8xM and Mx8 blocks (M≧16). LFNST8 or LFNST16 may be applied / allowed for PxQ blocks (P≧16, Q≧16). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0451] According to the combination of points 1 and 10 mentioned above, LFNST4 can be eliminated. That is, the LFNST4 conversion kernel does not need to be saved or available.

[0452] Specifically, NSPT may be applied / allowed for 4x4, 4x8, 4x16, 4x32, 8x4, 16x4, and 32x4 blocks, while neither NSPT nor LFNST may be applied / allowed for 4xL and Lx4 blocks (L≧64). LFNST8 may be applied / allowed for PxQ blocks (P≧8, Q≧8). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0453] According to the combinations of numbers 3 and 10 mentioned above, LFNST4 may be removed, and LFNST4x8 and LFNST8x4 may be included. That is, the conversion kernel for LFNST4 may not be saved or available, while the conversion kernels for LFNST4x8 and LFNST8x4 may be saved or available.

[0454] Specifically, NSPT may be applied / allowed for 4x4 blocks, and LFNST4x8 and LFNST8x4 may be applied / allowed for 4xL and Lx4 blocks (8≦L). LFNST8 may be applied to PxQ blocks (P≧8, Q≧8). If either the width or height of a block size is greater than the maximum allowable size (e.g., 64, 128, or 256), neither NSPT nor LFNST may be applied / allowed for that block size.

[0455] In addition to the group combinations mentioned above, other group combinations are possible as long as the block sizes do not overlap between them. Within the range where the block sizes do not overlap, at least two of the aforementioned combinations 1 through 13 may be combined.

[0456] For block sizes where neither NSPT nor LFNST applies / is permitted, the NSPT index and LFNST index do not need to be signaled. In this case, the NSPT index and LFNST index may be induced to 0, respectively. An NSPT index of 0 can indicate that NSPT is not applied. An LFNST index of 0 can indicate that LFNST is not applied.

[0457] Referring to Figure 4, the current block can be reconstructed based on the current block's residual sample (S420).

[0458] Based on the current block's intra-prediction mode, predicted samples for the current block can be derived. Based on the predicted samples and residual samples for the current block, restored samples for the current block can be generated.

[0459] Figure 6 shows a schematic configuration of a decoding device 300 that performs the image decoding method according to this disclosure.

[0460] Referring to Figure 6, the decoding apparatus 300 according to this disclosure may include a conversion coefficient induction unit 600, a resistive sample induction unit 610, and a reconstruction block generation unit 620. The conversion coefficient induction unit 600 may be configured as the entropy decoding unit 310 in Figure 3, the resistive sample induction unit 610 may be configured as the resistive processing unit 320 in Figure 3, and the reconstruction block generation unit 620 may be configured as the addition unit 340 in Figure 3.

[0461] The conversion coefficient induction unit 600 can obtain the current block's residual information from the bitstream, decode it, and induce the conversion coefficients of the current block.

[0462] The resistive sample induction unit 610 can induce a resistive sample of the current block by performing at least one of inverse quantization or inverse transformation on the transformation coefficients of the current block.

[0463] The residual sample induction unit 610 can determine a transformation kernel for the inverse transformation of the current block using a predetermined transformation kernel determination method, and induce the residual sample of the current block based on this. This has been explained with reference to Figure 4, and a detailed explanation will be omitted here.

[0464] The restored block generation unit 620 can restore the current block based on the current block's residual sample.

[0465] Figure 7 is a diagram illustrating an embodiment of the present disclosure, which shows an image encoding method performed by the encoding device 200.

[0466] Referring to Figure 7, residual samples of the current block can be induced (S700).

[0467] The current block's residual samples may be derived by subtracting the predicted samples from the current block's original samples. Here, the predicted samples may be derived based on a predetermined intra-prediction mode.

[0468] Referring to Figure 7, the transform coefficients of the current block can be derived by performing at least one of the following: transformation or quantization of the current block's residual sample (S710).

[0469] The transformation method relating to this disclosure may be understood as the inverse process of the inverse transformation described with reference to Figure 4. The method for determining the transformation kernel for the said transformation is as described with reference to Figure 4, and a detailed explanation is omitted here.

[0470] For example, one or more transformation sets may be defined / configured for the current block's transformation, each transformation set may contain one or more transformation kernel candidates. In this case, one of the multiple transformation sets may be selected as the transformation set for the current block. One of the multiple transformation kernel candidates belonging to the current block's transformation set may be selected. This selection may be made implicitly based on the context of the current block. Alternatively, an index indicating the selection of the optimal transformation set and / or transformation kernel candidate for the current block may be signaled.

[0471] Alternatively, the translation kernel for the current block may be determined based on an MTS set. One of several MTS sets may be selected based on at least one of the current block size or intra-prediction mode. The selected MTS set may contain one or more translation kernel candidates. One of the one or more translation kernel candidates may be selected, and the translation kernel for the current block may be determined based on the selected translation kernel candidate. The selection of the translation kernel candidate may be performed using a translation kernel candidate index induced based on the context of the current block. Alternatively, the optimal translation kernel candidate for the current block may be selected, and a translation kernel candidate index pointing to the selected translation kernel candidate may be signaled.

[0472] Alternatively, the transformation kernel for the current block may be determined based on an unseparated linear transformation (NSPT) kernel. A forward NSPT may be applied to the current block if its size belongs to a first group of block sizes to which an NSPT can be applied, but it may not be applied if its size belongs to a second group. If the size of the current block belongs to the second group, a forward separable linear transformation (e.g., DCT-2) can be applied to the residual samples of the current block to derive transformation coefficients. A forward LFNST may be further applied to all or some of the transformation coefficients derive from the separable linear transformation.

[0473] Furthermore, the NSPT may be applied based on at least one of the tree type or component type of the current block. The NSPT kernel (or NSPT matrix) for the NSPT may be determined by utilizing the symmetry between intra-prediction modes or between block morphologies. When a forward NSPT is applied to an MxN current block, the NSPT kernel may be expressed as rxMN, where r is the output length of the NSPT or the number of transformation coefficients generated by the NSPT, and MN is the product of the width and height of the current block, which can be the input length of the NSPT or the number of residual samples to which the NSPT is applied. A method for determining the size of such an NSPT kernel is described with reference to Figure 4.

[0474] The LFNST and / or NSPT indices for the conversion may be encoded into a single unified syntax, or the LFNST and NSPT indices may be encoded separately and inserted into the bitstream. The binary coding for the LFNST and NSPT indices, and the assignment of the CABAC context and initial values, are described with reference to Figure 4.

[0475] Furthermore, a method for signaling the transformation index has been explained with reference to Figure 4, and this can also be applied to a method for encoding the transformation index.

[0476] Referring to Figure 7, the conversion coefficients of the current block can be encoded to generate a bitstream (S720).

[0477] Based on the conversion coefficients of the current block, residual information relating to the conversion coefficients may be generated, and a bitstream may be generated by encoding the residual information.

[0478] Figure 8 shows a schematic configuration of an encoding device 200 that performs the image encoding method according to this disclosure.

[0479] Referring to Figure 8, the encoding device 200 according to this disclosure may include a resistive sample induction unit 800, a conversion coefficient induction unit 810, and a conversion coefficient encoding unit 820. The resistive sample induction unit 800 and the conversion coefficient induction unit 810 may be configured in the resistive processing unit 230 of Figure 2, and the conversion coefficient encoding unit 820 may be configured in the entropy encoding unit 240 of Figure 2.

[0480] The residual sample induction unit 800 can induce the residual sample of the current block by subtracting the predicted sample from the original sample of the current block. Here, the predicted sample may be induced based on a predetermined intra-prediction mode.

[0481] The conversion coefficient induction unit 810 can induce conversion coefficients of the current block by performing at least one of conversion or quantization on the residual sample of the current block. The conversion coefficient induction unit 810 can determine a conversion kernel of the current block based on at least one of the embodiments 1 to 3 described above, and apply the conversion kernel to the residual sample of the current block to induce conversion coefficients.

[0482] The conversion coefficient encoding unit 820 can encode the conversion coefficients of the current block and generate a bitstream.

[0483] In the embodiments described above, the method is explained based on a sequence diagram in a series of steps or blocks, but the embodiments are not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps than those described above. Furthermore, those skilled in the art will understand that the steps shown in the sequence diagram are not exclusive, and other steps may be included, or one or more steps in the sequence diagram may be omitted without affecting the scope of the embodiments described herein.

[0484] The methods relating to the embodiments of this document described above may be implemented in software form, and the encoding and / or decoding devices relating to this document may be included in image processing devices such as TVs, computers, smartphones, set-top boxes, and display devices.

[0485] When the embodiments described in this document are implemented as software, the methods described above may be implemented as modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in memory and executed by a processor. The memory may be located inside or outside the processor and may be connected to the processor by various well-known means. The processor may include an ASIC (application-specific integrated circuit), other chipsets, logic circuits, and / or data processing devices. The memory may include ROM (read-only memory), RAM (random access memory), flash memory, memory cards, storage media, and / or other storage devices. In other words, the embodiments described in this document may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information on instructions or algorithms for implementation may be stored on a digital storage medium.

[0486] Furthermore, the decoding and encoding devices to which the embodiments of this specification apply may include multimedia broadcasting transceivers, mobile communication terminals, home cinema video equipment, digital cinema video equipment, surveillance cameras, video conferencing equipment, real-time communication equipment such as video communication, mobile streaming equipment, storage media, camcorders, video-on-demand (VoD) service providers, over-the-top (OTT) video equipment, internet streaming service providers, 3D video equipment, virtual reality (VR) equipment, argumentative reality (AR) equipment, image-phone video equipment, transportation terminals (e.g., vehicle terminals (including autonomous vehicles), airplane terminals, ship terminals, etc.), and medical video equipment, and may be used to process video signals or data signals. For example, over-the-top (OTT) video equipment may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs).

[0487] Furthermore, the processing methods to which the embodiments of this specification apply may be produced in the form of programs executed on a computer and stored on a computer-readable recording medium. Similarly, multimedia data having the data structures according to the embodiments of this specification may also be stored on a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices on which computer-readable data is stored. The computer-readable recording medium may include, for example, Blu-ray discs (BDs), general-purpose serial buses (USBs), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium also includes media embodied in the form of carrier waves (e.g., transmission over the Internet). Furthermore, a bitstream generated by an encoding method may be stored on a computer-readable recording medium or transmitted over a wireless communication network.

[0488] Furthermore, the embodiments of this specification may be embodied as computer program products in the form of program code, and the program code may be executed on a computer according to the embodiments of this specification. The program code may be stored on a computer-readable carrier.

[0489] Figure 9 shows an example of a content streaming system to which the embodiments of this disclosure can be applied.

[0490] Referring to Figure 9, the content streaming system to which the embodiments described herein apply may broadly include an encoding server, a streaming server, a web server, media storage, user equipment, and multimedia input devices.

[0491] The encoding server is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and transmitting this bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, or camcorder directly generates the bitstream, the encoding server may be omitted.

[0492] The bitstream may be generated by an encoding method or bitstream generation method to which any embodiment of this specification is applied, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0493] The streaming server transmits multimedia data to the user's device based on a user request via a web server, and the web server acts as an intermediary to inform the user about available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server is responsible for controlling the commands and responses between the devices in the content streaming system.

[0494] The streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0495] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0496] Each server in the aforementioned content streaming system may be operated as a distributed server, in which case the data received by each server may be processed in a distributed manner.

[0497] The claims described herein may be combined in various ways. For example, the technical features of the method claims herein may be combined to be embodied as an apparatus, and the technical features of the apparatus claims herein may be combined to be embodied as a method. Furthermore, the technical features of the method claims and the technical features of the apparatus claims herein may be combined to be embodied as an apparatus, and the technical features of the method claims and the technical features of the apparatus claims herein may be combined to be embodied as a method.

Claims

1. The step of obtaining residual information from the bitstream, The steps include: deriving the conversion coefficient of the current block based on the aforementioned residual information; The steps include: performing at least one of inverse quantization or inverse transformation on the transformation coefficients of the current block to induce a residual sample of the current block; The step includes restoring the current block based on a residual sample of the current block, The inverse transformation is performed based on at least one of the following: non-separable primary transform (NSPT), low-frequency non-separable transform (LFNST), or separate primary transform. The predefined allowable conversion block sizes are divided into multiple groups. An image decoding method in which the plurality of groups include at least two of the following: a first group including block sizes with a minimum width and height of 4; a second group including block sizes with a minimum width and height of 8; a third group including block sizes with a minimum width and height of 16; a fourth group including block sizes greater than or equal to 16x16; a fifth group including block sizes greater than or equal to 32x32; or a sixth group including block sizes greater than or equal to 8x8.

2. The image decoding method according to claim 1, wherein any one of the plurality of groups includes one or more block sizes to which the NSPT can be applied.

3. The image decoding method according to claim 2, wherein any one of the plurality of groups includes one or more block sizes to which the NSPT and LFNST do not apply.

4. The image decoding method according to claim 2, wherein any one of the plurality of groups includes one or more block sizes to which the LFNST can be applied.

5. The NSPT can be applied to the 4xL blocks and Lx4 blocks (4 ≤ L ≤ 32) belonging to the first group, The image decoding method according to claim 1, wherein the NSPT and LFNST are not applied to the 4xM block and Mx4 block (M>32) belonging to the first group.

6. The NSPT can be applied to the 4x4 blocks belonging to the first group, The image decoding method according to claim 1, wherein the LFNST can be applied to 4xL blocks and Lx4 blocks (L≧8) belonging to the first group.

7. The NSPT can be applied to the 4x4 blocks belonging to the first group, The image decoding method according to claim 1, wherein the NSPT and LFNST are not applied to the 4xL block and Lx4 block (L≧8) belonging to the first group.

8. The NSPT can be applied to the 8xL blocks and Lx8 blocks (8 ≤ L ≤ 32) belonging to the second group described above. The image decoding method according to claim 1, wherein the NSPT and LFNST are not applied to the 8xM blocks and Mx8 blocks (M>32) belonging to the second group.

9. The NSPT can be applied to the 8x8 blocks belonging to the second group, The image decoding method according to claim 1, wherein the LFNST can be applied to 8xL blocks and Lx8 blocks (L≧16) belonging to the second group.

10. The NSPT can be applied to the 8x8 blocks belonging to the second group, The image decoding method according to claim 1, wherein the NSPT and LFNST are not applied to the 8xL blocks and Lx8 blocks (L≧16) belonging to the second group.

11. The image decoding method according to claim 1, wherein NSPT is not applied to at least one of the third to sixth groups.

12. Currently, we are in the stage of inducing the residual sampling of the block, A step of inducing a transformation coefficient of the current block by performing at least one of the following: conversion to a residual sample or quantization of the current block, The step includes encoding the conversion coefficients of the current block, The aforementioned conversion is performed based on at least one of the following: non-separable primary transform (NSPT), separate primary transform (separable primary transform), or low frequency non-separable transform (LFNST). The predefined allowable conversion block sizes are divided into multiple groups. An image encoding method in which the plurality of groups include at least two of the following: a first group including block sizes with a minimum width and height of 4; a second group including block sizes with a minimum width and height of 8; a third group including block sizes with a minimum width and height of 16; a fourth group including block sizes greater than or equal to 16x16; a fifth group including block sizes greater than or equal to 32x32; or a sixth group including block sizes greater than or equal to 8x8.

13. A computer-readable recording medium for storing a bitstream generated by the image encoding method described in claim 12.

14. A step of acquiring a bitstream of image information, wherein the bitstream is generated by deriving a residual sample of the current block, deriving a conversion coefficient by performing at least one of conversion or quantization of the residual sample of the current block, and encoding the conversion coefficient of the current block. The step includes transmitting data including the bitstream, The aforementioned conversion is performed based on at least one of the following: non-separable primary transform (NSPT), separate primary transform (separable primary transform), or low frequency non-separable transform (LFNST). The predefined allowable conversion block sizes are distinguished into multiple groups. A data transmission method in which the plurality of groups include at least two of the following: a first group including a block size with a minimum width and height of 4; a second group including a block size with a minimum width and height of 8; a third group including a block size with a minimum width and height of 16; a fourth group including a block size greater than or equal to 16x16; a fifth group including a block size greater than or equal to 32x32; or a sixth group including a block size greater than or equal to 8x8.

Images