Video encoding / decoding method and apparatus, and recording medium storing a bitstream.

Abstract

The video decoding method and apparatus relating to this disclosure can derive transformation coefficients for the current block from a bitstream, determine a set of non-separable primary transforms (NSPTs) for the current block, determine an NSPT kernel for the current block from the NSPT set, perform NSPT on the transformation coefficients of the current block based on the NSPT kernel to derive residual samples, and reconstruct the current block based on the residual samples. NSPT may be applied on the basis that the size of the current block belongs to one or more block size groups to which NSPT is applicable.

Description

[Technical Field]

[0001] The present invention relates to an image encoding / decoding method and apparatus, and a recording medium storing a bitstream. [Background technology]

[0002] In recent years, the demand for high-resolution, high-quality images such as HD (High Definition) and UHD (Ultra High Definition) images has increased in various application fields, leading to discussions about highly efficient image compression technologies.

[0003] Various image compression techniques exist, such as inter-prediction techniques that predict pixel values ​​contained in the current picture from previous or subsequent pictures; intra-prediction techniques that predict pixel values ​​contained in the current picture using pixel information within the current picture; and entropy coding techniques that assign short codes to frequently occurring values ​​and long codes to less frequently occurring values. By using such image compression techniques, image data can be effectively compressed and transmitted or stored. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] This disclosure aims to provide a method and apparatus for performing a transformation using an unseparated first-order transformation.

[0005] This disclosure aims to provide a method and apparatus for performing a transformation using a reduced-dimensional non-separable linear transformation kernel.

[0006] This disclosure aims to provide a method and apparatus for determining / signaling an unseparated transform kernel based on coding parameters. [Means for solving the problem]

[0007] The video decoding method and apparatus according to this disclosure can derive conversion coefficients for the current block from a bitstream, determine a set of non-separable primary transforms (NSPTs) for the current block, determine an NSPT kernel for the current block from the NSPT set, perform an NSPT on the conversion coefficients of the current block based on the NSPT kernel to derive a residual sample for the current block, and restore the current block based on the residual sample.

[0008] In the video decoding method and apparatus relating to this disclosure, the NSPT may be applied on the basis that the size of the current block belongs to one or more block size groups to which the NSPT can be applied.

[0009] In the video decoding method and apparatus relating to this disclosure, the NSPT set may be determined to be one of 35 predefined NSPT sets.

[0010] In the video decoding method and apparatus relating to this disclosure, the NSPT set may include three NSPT kernel candidates.

[0011] In the video decoding method and apparatus relating to this disclosure, the group may include at least one block size from among 4x4, 4x8, 8x4, 8x8, 16x8, 8x16, 16x4, 4x16, 4x32, or 32x4.

[0012] In the video decoding method and apparatus relating to this disclosure, the number of conversion coefficients to which the NSPT is applied may be less than or equal to 36, based on the size of the current block being 4x32 or 32x4.

[0013] The video encoding method and apparatus according to the present disclosure can induce the residual samples of the current block, determine a non-separable primary transform (NSPT) set for the current block, apply the NSPT to the residual samples of the current block based on any one of a plurality of NSPT kernel candidates belonging to the NSPT set to induce the transform coefficients of the current block, and encode the transform coefficients of the current block to generate a bitstream.

[0014] In the video encoding method and apparatus according to the present disclosure, the NSPT may be applied based on the size of the current block belonging to a group of one or more block sizes to which the application of the NSPT is possible.

[0015] There is provided a computer-readable digital storage medium storing encoded video / image information for performing an image decoding method by a decoding apparatus according to the present disclosure.

[0016] There is provided a computer-readable digital storage medium storing video / image information generated by an image encoding method according to the present disclosure.

[0017] There are provided a method and an apparatus for transmitting video / image information generated by an image encoding method according to the present disclosure.

Advantages of the Invention

[0018] By using non-separable primary transform as the primary transform, the present disclosure can improve the performance of the transform.

[0019] By performing the transform using a non-separable primary transform kernel of a reduced dimension, the present disclosure can improve the performance of the transform.

[0020] The present disclosure can improve the encoding efficiency by effectively determining and / or signaling a non-separable transform kernel based on encoding parameters.

Brief Description of Drawings

[0021] [Figure 1] It is a diagram showing a video / image coding system according to the present disclosure. [Figure 2] It is a schematic block diagram of an encoding device to which an embodiment of the present disclosure is applicable and where encoding of a video / image signal is performed. [Figure 3] It is a schematic block diagram of a decoding device to which an embodiment of the present disclosure is applicable and where decoding of a video / image signal is performed. [Figure 4] It is an example of an embodiment according to the present disclosure and shows an image decoding method performed by a decoding device (300). [Figure 5] It is a diagram illustrating an intra prediction mode and its prediction direction according to the present disclosure. [Figure 6] It is a diagram showing a schematic configuration of a decoding device (300) that performs an image decoding method according to the present disclosure. [Figure 7] It is an example of an embodiment according to the present disclosure and shows an image encoding method performed by an encoding device (200). [Figure 8] It is a diagram showing a schematic configuration of an encoding device (200) that performs an image encoding method according to the present disclosure. [Figure 9] It is a diagram showing an example of a content streaming system to which an embodiment of the present disclosure is applicable.

Modes for Carrying Out the Invention

[0022] This disclosure may be modified in various ways and may have various embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the disclosure to any particular embodiment, but rather should be understood to include all modifications, equivalents, or substitutions that fall within the spirit and technical scope of this disclosure. In the description of each figure, similar reference numerals are used for similar components.

[0023] Terms such as "First," "Second," etc., may be used to describe various components, but these components should not be limited by such terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without exceeding the scope of rights of this disclosure, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of multiple related descriptions or any one of multiple related descriptions.

[0024] When it is stated that one component is "connected" or "linked" to another component, it should be understood that it may be directly connected or linked to the other component, and that there may be other components in between. On the other hand, when it is stated that one component is "directly connected" or "linked" to another component, it should be understood that there are no other components in between.

[0025] The terminology used in this application is solely for the purpose of describing specific embodiments and is not intended to limit the disclosure. Singular expressions include plural expressions unless otherwise specified in the context. In this application, terms such as “includes” or “having” are intended to specify the existence of features, figures, stages, operations, components, parts, or combinations thereof described in the specification, and should be understood not to preemptively exclude the possibility of the existence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0026] This disclosure relates to video / image coding. For example, the methods / examples disclosed herein may be applied to methods disclosed in the VVC (versatile video coding) standard. Also, the methods / examples disclosed herein may be applied to methods disclosed in the EVC (essential video coding) standard, AV1 (AOMedia Video 1) standard, AVS2 (2nd generation of audio video coding standard), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0027] This specification presents various embodiments relating to video / image coding, and unless otherwise specified, the above embodiments may be combined with each other.

[0028] In this specification, video can mean a collection of images over time. Picture generally means a unit representing a single image at a specific time point, and slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may contain one or more CTUs (coding tree units). A picture may consist of one or more slices / tiles. A tile is a rectangular area consisting of multiple CTUs in a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs having the same height as the picture height and the width specified by the syntax requirements of the picture parameter set. A tile row is a rectangular area of ​​CTUs having the same height as the picture parameter set and the width as the picture width. CTUs within a tile may be arranged consecutively by a CTU raster scan, while tiles within a picture may be arranged consecutively by a tile raster scan. A slice may contain an integer number of complete tiles or an integer number of consecutive complete CTU rows within a picture tile that can exclusively be contained within a single NAL unit. On the other hand, a picture may be divided into two or more subpictures. A subpicture may be a rectangular region of one or more slices in the picture.

[0029] A pixel, or pel, can refer to the smallest unit that makes up a picture (or image). The term "sample" may also be used as a counterpart to "pixel." A sample can generally represent a pixel or a pixel value, and may represent only the pixel / pixel value for the lumen component, or only the pixel / pixel value for the chroma component.

[0030] A unit can refer to a basic unit of image processing. A unit may contain at least one of the following: a specific region of a picture and information associated with that region. A unit may contain one lumen block and two chroma (e.g., cb, cr) blocks. The term unit may, as in some cases, be used interchangeably with terms such as block or area. Generally, an MxN block may contain a sample (or sample array) or a set (or array) of transform coefficients consisting of M columns and N rows.

[0031] In this specification, "A or B" can mean "A only," "B only," or "both A and B." In other words, in this specification, "A or B" may be interpreted as "A and / or B." For example, in this specification, "A, B or C" can mean "A only," "B only," "C only," or "any combination of A, B and C."

[0032] In this specification, a slash ( / ) or comma can mean "and / or". For example, "A / B" can mean "A and / or B". Thus, "A / B" can mean "A only", "B only", or "both A and B". For example, "A, B, C" can mean "A, B or C".

[0033] In this specification, "at least one of A and B" can mean "A only," "B only," or "both A and B." Furthermore, in this specification, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as equivalent to "at least one of A and B."

[0034] Furthermore, in this specification, "at least one of A, B and C" can mean "A only," "B only," "C only," or "any combination of A, B and C." Also, "at least one of A, B or C" or "at least one of A, B and / or C" can mean "at least one of A, B and C."

[0035] Furthermore, parentheses used in this specification can mean "for example." Specifically, when "prediction (intra prediction)" is displayed, "intra prediction" may be proposed as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra prediction," and "intra prediction" may be proposed as an example of "prediction." Also, when "prediction (i.e., intra prediction)" is displayed, "intra prediction" may be proposed as an example of "prediction."

[0036] In this specification, technical features described individually in the same drawings may be embodied individually or simultaneously.

[0037] Figure 1 shows the video / image coding system related to this disclosure.

[0038] Referring to Figure 1, the video / image coding system may include a first device (source device) and a second device (receiving device).

[0039] A source device can transmit encoded video / image information or data to a receiving device in the form of a file or streaming via a digital storage medium or network. The source device may include a video source, an encoding device, and a transmission unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may also be called a video / image encoding device, and the decoding device may also be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, which may consist of a separate device or external component.

[0040] A video source can acquire video / images through processes such as video / image capture, synthesis, or generation. A video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include a computer, tablet, and smartphone, and can generate video / images (electronically). For example, virtual video / images may be generated through a computer, in which case the video / image capture process may be replaced by the process of generating the associated data.

[0041] An encoding device can encode input video / images. The encoding device can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) may be output in the form of a bitstream.

[0042] The transmission unit can transmit encoded video / image information or data output in bitstream format to the receiving unit of a receiving device via a digital storage medium or network in file or streaming format. The digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, and SSD. The transmission unit may include elements for generating media files in a predetermined file format and may include elements for transmission via a broadcast / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0043] A decoding device can decode video / images by performing a series of steps, such as inverse quantization, inverse transformation, and prediction, corresponding to the operation of an encoding device.

[0044] The renderer can render the decoded video / image. The rendered video / image may be displayed on the display unit.

[0045] Figure 2 is a schematic block diagram of an encoding device to which the embodiments of this disclosure can be applied, in which video / image signals are encoded.

[0046] Referring to Figure 2, the encoding device 200 may include an image partitioner (210), a predictor (220), a residual processor (230), an entropy encoder (240), an adder (250), a filter (260), and a memory (270). The predictor (220) may include an inter-prediction unit (221) and an intra-prediction unit (222). The residual processor (230) may include a transformer (232), a quantizer (233), a dequantizer (234), and an inverse transformer (235). The residual processor (230) may further include a subtractor (231). The addition unit 250 may be called a reconstructor or a reconstructed block generator. The image segmentation unit 210, prediction unit 220, residual processing unit 230, entropy encoding unit 240, addition unit 250, and filtering unit 260 described above may be composed of one or more hardware components (e.g., an encoding device chipset or processor) depending on the embodiment. The memory 270 may also include a DPB (decoded picture buffer) and may be composed of a digital storage medium. The hardware components may further include the memory 270 as an internal / external component.

[0047] The image splitting unit 210 can split an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, the processing unit may be called a coding unit (CU). In this case, the coding unit may be recursively split from a coding tree unit (CTU) or the largest coding unit (LCU) using a QTBTTT (Quad-tree binary-tree ternary-tree) structure.

[0048] As an example, a single coding unit may be divided into multiple coding units having deeper depths based on a quad-tree structure, a binary tree structure, and / or a tertiary structure. In this case, for example, the quad-tree structure may be applied first, followed by the binary tree structure and / or tertiary structure. Alternatively, the binary tree structure may be applied before the quad-tree structure. The coding procedure according to this specification may be performed based on a final coding unit that is not further divided. In this case, based on coding efficiency due to image characteristics, the largest coding unit may be immediately used as the final coding unit, or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and reconstruction, which will be described later.

[0049] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be separated or partitioned from the final coding unit described above. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit that derives a transform coefficient and / or a unit that derives a residual signal from the transform coefficient.

[0050] The term "unit" may be used interchangeably with terms such as "block" or "area." Generally, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only the pixel / pixel value of the lumen component, or only the pixel / pixel value of the chroma component. The term "sample" may be used in conjunction with a single picture (or image), pixel, or pel.

[0051] The encoding device 200 can generate a residual signal (residual block, residual sample array) by subtracting the prediction signal (prediction block, prediction sample array) output from the inter-prediction unit 221 or intra-prediction unit 222 from the input image signal (original block, original sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, the unit that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) within the encoding device 200 may be called the subtraction unit 231.

[0052] The prediction unit 220 can make predictions for the block to be processed (hereinafter referred to as the current block) and generate a predicted block containing prediction samples for the current block. The prediction unit 220 can determine whether intra-prediction or inter-prediction is applied to the current block or on a CU basis. As will be described later in the explanation of each prediction mode, the prediction unit 220 can generate various information related to prediction, such as prediction mode information, and transmit it to the entropy encoding unit 240. The information related to prediction may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0053] The intra-prediction unit 222 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block, or it may be located at a certain distance from the current block, depending on the prediction mode. In intra-prediction, the prediction mode may include one or more non-directional modes and multiple directional modes. The non-directional mode may include at least one of DC mode or planar mode. The directional mode may include 33 or 65 directional modes, depending on the degree of fineness of the prediction direction. However, this is an example, and more or fewer directional modes may be used depending on the settings. The intra-prediction unit 222 can also determine the prediction mode to be applied to the current block using the prediction modes applied to the surrounding blocks.

[0054] The interprediction unit 221 can guide a predicted block for the current block based on a reference block (reference sample array) identified by a motion vector on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between the surrounding block and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, the surrounding block may include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture containing the reference block and the reference picture containing the temporal neighboring block may be the same or different. The temporal neighboring block may be called a collocated reference block, colCU, etc., and the reference picture containing the temporal neighboring block may be called a collocated picture (colPic). For example, the interpretation unit 221 can construct a motion information candidate list based on surrounding blocks and generate information indicating which candidates are used to derive the motion vector and / or reference picture index of the current block. Interpretation may be performed based on various prediction modes; for example, in skip mode and merge mode, the interpretation unit 221 can use the motion information of surrounding blocks as the motion information of the current block. In skip mode, unlike merge mode, the residual signal does not need to be transmitted.In motion vector prediction (MVP) mode, the motion vectors of surrounding blocks are used as motion vector predictors, and the motion vector difference is signaled to indicate the motion vector of the current block.

[0055] The prediction unit 220 can generate prediction signals based on various prediction methods described later. For example, the prediction unit can apply intra-prediction or inter-prediction for predictions for a single block, and can also apply intra-prediction and inter-prediction simultaneously. This may be called CIIP (combined inter and intra prediction) mode. The prediction unit may also be based on intra-block copy (IBC) prediction mode or palette mode for predictions for blocks. The IBC prediction mode or palette mode may be used for coding content images / videos such as games, as in SCC (screen content coding). IBC basically performs predictions within the current picture, but can be performed similarly to inter-prediction in that it derives reference blocks within the current picture. That is, IBC can use at least one of the inter-prediction methods described herein. Palette mode may be considered an example of intra-coding or intra-prediction. When palette mode is applied, in-picture sample values ​​can be signaled based on information about the palette table and palette index. The prediction signal generated by the prediction unit 220 may be used to generate a restoration signal or to generate a residual signal.

[0056] The transformation unit 232 can generate transformation coefficients by applying a transformation method to the residual signal. For example, the transformation method may include at least one of the following: DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT refers to the transformation obtained from a graph when the relationship information between pixels is represented by this graph. CNT refers to the transformation obtained by generating a prediction signal using all previously restored pixels and obtaining a transformation based on it. Furthermore, the transformation process may be applied to pixel blocks of the same size that are square, or to blocks of variable size that are not square.

[0057] The quantization unit 233 quantizes the conversion coefficients and transmits them to the entropy encoding unit 240, which can encode the quantized signal (information regarding the quantized conversion coefficients) and output it as a bitstream. The information regarding the quantized conversion coefficients may be called residual information. The quantization unit 233 can rearrange the block-shaped quantized conversion coefficients into a one-dimensional vector based on the coefficient scan order, and can generate information regarding the quantized conversion coefficients based on the one-dimensional vector-shaped quantized conversion coefficients.

[0058] The entropy encoding unit 240 can perform various encoding methods such as exponential Golomb, CAVLC (context-adaptive variable length coding), and CABAC (context-adaptive binary arithmetic coding). In addition to the quantized conversion coefficients, the entropy encoding unit 240 can also encode information necessary for video / image restoration (e.g., the values ​​of syntax elements) together with or separately from the quantized conversion coefficients.

[0059] The encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information may further include information about various parameter sets, such as an adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. In this specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded by the encoding procedure described above and included in the bitstream. The bitstream may be transmitted over a network or stored on a digital storage medium. Here, the network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 may be transmitted by a transmission unit (not shown) and / or stored by a storage unit (not shown) which are configured as internal / external elements of the encoding device 200, or the transmission unit may be included in the entropy encoding unit 240.

[0060] The quantized conversion coefficients output from the quantization unit 233 may be used to generate a prediction signal. For example, by applying inverse quantization and inverse transformation to the quantized conversion coefficients in the inverse quantization unit 234 and the inverse transformation unit 235, the residual signal (residual block or residual sample) can be reconstructed. The adder unit 250 may add the reconstructed residual signal to the prediction signal output from the inter-prediction unit 221 or the intra-prediction unit 222 to generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block may be used as the reconstructed block. The adder unit 250 may be called the reconstruction unit or the reconstructed block generation unit. The generated reconstructed signal may be used for intra-prediction of the next block to be processed in the current picture, and may be used for inter-prediction of the next picture after filtering, as described later. On the other hand, LMCS (luma mapping with chroma scaling) may be applied during the picture encoding and / or reconstruction process.

[0061] The filtering unit 260 can apply filtering to the restored signal to improve subjective / objective image quality. For example, the filtering unit 260 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 270, specifically in the DPB of the memory 270. The various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, etc. The filtering unit 260 can generate various filtering-related information and transmit it to the entropy encoding unit 240. The filtering-related information may be encoded by the entropy encoding unit 240 and output in bitstream format.

[0062] The corrected restored picture transmitted to memory 270 may be used as a reference picture in the interpretation unit 221. This allows the encoding device to avoid prediction mismatches between the encoding device 200 and the decoding device when interpretation is applied, and also improves encoding efficiency.

[0063] The DPB in memory 270 can store the corrected restored picture for use as a reference picture in the inter-prediction unit 221. Memory 270 can store motion information of blocks from which motion information in the current picture has been derived (or encoded), and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 221 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 270 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 222.

[0064] Figure 3 is a schematic block diagram of a decoding device to which the embodiments of this disclosure can be applied, in which video / image signals are decoded.

[0065] Referring to Figure 3, the decoding device 300 may be configured to include an entropy decoder (310), a residual processor (320), a predictor (330), an adder (340), a filter (350), and a memory (360). The predictor (330) may include an inter-prediction unit (331) and an intra-prediction unit (332). The residual processor (320) may include a dequantizer (321) and an inverse transformer (321).

[0066] The entropy decoding unit 310, the residual processing unit 320, the prediction unit 330, the addition unit 340, and the filtering unit 350 described above may be configured by a single hardware component (e.g., a decoding device chipset or processor) depending on the embodiment. The memory 360 may include a DPB (decoded picture buffer) and may be configured by a digital storage medium. The hardware component may further include the memory 360 as an internal / external component.

[0067] When a bitstream containing video / image information is input, the decoding device 300 can reconstruct the image in accordance with the process by which the video / image information was processed in the encoding device shown in Figure 2. For example, the decoding device 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding device 300 can perform decoding using the processing units applied in the encoding device. Therefore, the decoding processing units may be coding units, which may be divided from a coding tree unit or a maximum coding unit according to a quad-tree structure, a binary tree structure, and / or a tertiary tree structure. One or more conversion units may be derived from the coding unit. The reconstructed image signal decoded and output by the decoding device 300 may then be reproduced by a playback device.

[0068] The decoding device 300 can receive the signal output from the encoding device shown in Figure 2 in the form of a bitstream, and the received signal may be decoded by the entropy decoding unit 310. For example, the entropy decoding unit 310 can parse the bitstream and derive information necessary for image restoration (or picture restoration) (e.g., video / image information). The video / image information may further include information about various parameter sets such as the adaptation parameter set (APS), picture parameter set (PPS), sequence parameter set (SPS), or video parameter set (VPS). The video / image information may also further include general constraint information. The decoding device can decode the picture based on the parameter set information and / or the general constraint information. The signal / received information and / or syntax elements described later in this specification may be decoded by the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 can decode information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values ​​of syntax elements necessary for image reconstruction and the quantized values ​​of conversion coefficients related to the residual. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using the syntax element information to be decoded and the decoding information of the surrounding and decoded blocks or symbol / bin information decoded in a previous stage, predicts the probability of bin occurrence based on the determined context model, and generates symbols corresponding to the values ​​of each syntax element by performing arithmetic decoding of the bins. At this time, after determining the context model, the CABAC entropy decoding method can update the context model using the symbol / bin information decoded for the context model of the next symbol / bin.Information related to prediction from the information decoded by the entropy decoding unit 310 is provided to the prediction unit (inter-prediction unit 332 and intra-prediction unit 331), and residual values ​​that have been entropy decoded by the entropy decoding unit 310, i.e., quantized conversion coefficients and related parameter information, may be input to the residual processing unit 320. The residual processing unit 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). In addition, information related to filtering from the information decoded by the entropy decoding unit 310 may be provided to the filtering unit 350. On the other hand, a receiving unit (not shown) that receives signals output from the encoding device may be further configured as an internal / external element of the decoding device 300, or the receiving unit may be a component of the entropy decoding unit 310.

[0069] On the other hand, the decoding device according to this specification may be called a video / image / picture decoding device, and the decoding device may be divided into an information decoding device (video / image / picture information decoding device) and a sample decoding device (video / image / picture sample decoding device). The information decoding device may include the entropy decoding unit 310, and the sample decoding device may include at least one of the inverse quantization unit 321, inverse transformation unit 322, addition unit 340, filtering unit 350, memory 360, inter-prediction unit 332, and intra-prediction unit 331.

[0070] The inverse quantization unit 321 can inverse quantize the quantized transformation coefficients and output the transformation coefficients. The inverse quantization unit 321 can rearrange the quantized transformation coefficients in a two-dimensional block form. In this case, the rearrangement can be performed based on the coefficient scan order performed by the encoding device. The inverse quantization unit 321 can perform inverse quantization on the quantized transformation coefficients using quantization parameters (e.g., quantization step size information) and obtain the transformation coefficients.

[0071] The inverse conversion unit 322 performs an inverse conversion on the conversion coefficients to obtain a resistive signal (residual block, resistive sample array).

[0072] The prediction unit 320 can make predictions for the current block and generate a predicted block containing prediction samples for the current block. Based on the prediction information output from the entropy decoding unit 310, the prediction unit 320 can determine whether intra-prediction or inter-prediction is applied to the current block and determine a specific intra / inter-prediction mode.

[0073] The prediction unit 320 can generate prediction signals based on various prediction methods described later. For example, the prediction unit 320 can apply intra-prediction or inter-prediction for prediction of a single block, or it can apply intra-prediction and inter-prediction simultaneously. This may be called CIIP (combined inter and intra prediction) mode. The prediction unit may also be based on intra-block copy (IBC) prediction mode or palette mode for prediction of a block. The IBC prediction mode or palette mode may be used for content image / video coding such as SCC (screen content coding) for games. IBC basically performs prediction within the current picture, but can be performed similarly to inter-prediction in that it derives a reference block within the current picture. That is, IBC can use at least one of the inter-prediction methods described herein. Palette mode can be considered an example of intra-coding or intra-prediction. When palette mode is applied, information regarding the palette table and palette index may be included in the video / image information and signaled.

[0074] The intra-prediction unit 331 can predict the current block by referring to a sample in the current picture. The referenced sample may be located in the vicinity (neighbor) of the current block, or it may be located at a certain distance from the current block, depending on the prediction mode. In intra-prediction, the prediction mode may include one or more non-directional modes and multiple directional modes. The intra-prediction unit 331 can determine the prediction mode to be applied to the current block using the prediction modes applied to the surrounding blocks.

[0075] The interprediction unit 332 can derive a predicted block for the current block based on a reference block (reference sample array) identified by motion vectors on the reference picture. In this case, in order to reduce the amount of motion information transmitted in interprediction mode, motion information can be predicted in units of blocks, subblocks, or samples based on the correlation of motion information between surrounding blocks and the current block. The motion information may include motion vectors and reference picture indices. The motion information may further include interprediction direction information (L0 prediction, L1 prediction, Bi prediction, etc.). In the case of interprediction, surrounding blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. For example, the interprediction unit 332 can construct a motion information candidate list based on surrounding blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Interprediction may be performed based on various prediction modes, and the prediction information may include information indicating the interprediction mode for the current block.

[0076] The adder 340 can generate a restored signal (restored picture, restored block, restored sample array) by adding the acquired residual signal to the predicted signal (predicted block, predicted sample array) output from the prediction unit (including the inter-prediction unit 332 and / or intra-prediction unit 331). When there is no residual for the block to be processed, such as when skip mode is applied, the predicted block may be used as the restored block.

[0077] The summing unit 340 may be called the restoration unit or the restoration block generation unit. The generated restoration signal may be used for intra-prediction of the next block to be processed in the current picture, and may be output after filtering as described later, or may be used for intra-prediction of the next picture. On the other hand, LMCS (luma mapping with chroma scaling) may be applied during the picture decoding process.

[0078] The filtering unit 350 can apply filtering to the restored signal to improve subjective / objective image quality. For example, the filtering unit 350 can apply various filtering methods to the restored picture to generate a modified restored picture, and transmit the modified restored picture to the memory 360, specifically to the DPB of the memory 360. The various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like.

[0079] The restored picture stored (modified) in the DPB of memory 360 may be used as a reference picture in the inter-prediction unit 332. Memory 360 can store motion information of blocks from which motion information in the current picture has been derived (or decoded), and / or motion information of blocks in the picture that have already been restored. The stored motion information can be transmitted to the inter-prediction unit 260 for use as motion information of spatially surrounding blocks or motion information of temporally surrounding blocks. Memory 360 can store restored samples of restored blocks in the current picture and transmit them to the intra-prediction unit 331.

[0080] In this specification, the embodiments described for the filtering unit 260, the inter-prediction unit 221, and the intra-prediction unit 222 of the encoding device 200 may also be applied identically or in a corresponding manner to the filtering unit 350, the inter-prediction unit 332, and the intra-prediction unit 331 of the decoding device 300, respectively.

[0081] Figure 4 is a diagram illustrating an embodiment of the present disclosure, which shows an image decoding method performed by a decoding device.

[0082] Referring to Figure 4, the transform coefficients of the current block can be derived from the bitstream (S400). That is, the bitstream may contain the current block's residual information, and the transform coefficients of the current block can be derived by decoding this residual information.

[0083] Referring to Figure 4, residual samples of the current block can be derived by performing at least one of dequantization or inverse-transform on the transformation coefficients of the current block (S410).

[0084] When Adaptive Multiple Transform Selection (MTS) is applied, the inverse transform may be performed based on at least one of DCT-2, DST-7, or DCT-8. Here, DCT-2, DST-7, DCT-8, etc. may be called the transform type, transform kernel, or transform core.

[0085] In this disclosure, the inverse transform can mean a separable transform. However, it is not limited to this, and the inverse transform can also mean a non-separable transform, or it may be a concept that includes both separable and non-separable transforms. Furthermore, in this disclosure, the inverse transform means a primary transform, but it is not limited to this, and it may be transformed into the same / similar form and applied to a secondary transform.

[0086] For example, as a method for the inverse transform, only DCT-2 and the inseparable transform may be used, or the inseparable transform may be used in addition to at least one of DCT-2, DST-7, or DCT-8, or the inseparable transform may be used in place of one or more of the transform kernels from DCT-2, DST-7, or DCT-8.

[0087] As a more specific example, if the candidate transformation kernels for delimited transformations are (DCT-2,DCT-2), (DST-7,DST-7), (DCT-8,DST-7), (DST-7,DCT-8), and (DCT-8,DCT-8), then the non-delimited transformation may substitute for or add to one or more of the five candidate transformation kernels. Here, notation such as (transform 1, transform 2) indicates that transform 1 is applied horizontally and transform 2 is applied vertically. When the non-delimited transformation substitutes for some of the candidate transformation kernels, the remaining candidate transformation kernels, excluding (DCT-2,DCT-2) and (DST-7,DST-7), can be replaced by the non-delimited transformation. However, the candidate transformation kernels are merely examples, and other types of DCT and / or DST may be included, and a transform skip may be included as a candidate transformation kernel.

[0088] An inseparable transform can refer to a transform or inverse transform based on a non-separable transform matrix. That is, unlike a separable transform, which separates vertical and horizontal transforms and performs them independently, an inseparable transform allows both horizontal and vertical transforms to be performed in a single step.

[0089] For example, when an inseparable transformation is performed on a 4x4 block, the input data X for the inseparable transformation is given by the following equation 1.

[0090] [Formula 1]

number

[0091] When the input data X is represented in vector form, the vector X' may be represented as follows:

[0092] [Formula 2]

number

[0093] In this case, the non-separable transformation may be performed as shown in the following equation 3.

[0094] [Formula 3]

number

[0095] In equation 3, F represents the transformation coefficient vector, T represents the 16x16 inseparable transformation matrix, and · signifies the multiplication of a matrix and a vector.

[0096] The above equation 3 may derive a 16x1 transformation coefficient vector F, which may be reconstructed into a 4x4 block according to a predetermined scan sequence. The scan sequence may be a horizontal scan, a vertical scan, a diagonal scan, a z scan, a raster scan, or a predefined scan.

[0097] The set of inseparable transforms and / or transform kernels for the inseparable transforms may be configured in various ways based on at least one of the following: prediction mode (e.g., intra mode, inter mode, etc.), width, height, or number of pixels of the current block, position of subblocks within the current block, explicitly signaled syntactic elements, statistical characteristics of the surrounding samples, whether or not a quadratic transform is used, or quantization parameters (QP).

[0098] Specifically, in intra-mode, predefined intra-prediction modes are grouped to correspond to n inseparable transformation sets, and each inseparable transformation set may contain k transformation kernel candidates. Here, n and k may be arbitrary constants that follow the same rules (conditions) defined for the encoding and decoding devices.

[0099] The number of non-separable transformation sets and / or the number of transformation kernel candidates included in each non-separable transformation set may be configured to differ according to the width and / or height of the current block. For example, for a 4x4 block, n1 non-separable transformation sets and k1 transformation kernel candidates may be configured. For a 4x8 block, n2 non-separable transformation sets and k2 transformation kernel candidates may be configured. Furthermore, the number of non-separable transformation sets and the number of transformation kernel candidates included in each non-separable transformation set may be configured to differ according to the product of the width and height of the current block. For example, if the product of the width and height of the current block is 256 or greater (or exceeds 256), n3 non-separable transformation sets and k3 transformation kernel candidates may be configured; otherwise, n4 non-separable transformation sets and k4 transformation kernel candidates may be configured. In other words, since the degree of change in the statistical characteristics of the residual signal differs according to the block size, the number of non-separable transformation sets and transformation kernel candidates may be configured to reflect this.

[0100] Currently, when a block is divided into multiple subblocks, the statistical characteristics of the residual signal may differ for each subblock. Therefore, the number of non-separable transform sets and transform kernel candidates may be configured to be different for each subblock. For example, when a 4x8 or 8x4 block is divided into two 4x4 subblocks and a non-separable transform is applied to each subblock, n5 non-separable transform sets and k5 transform kernel candidates may be configured for the upper left 4x4 subblock, and n6 non-separable transform sets and k6 transform kernel candidates may be configured for the other 4x4 subblocks.

[0101] The number of inseparable transformation sets and candidate transformation kernels may be configured to be different from each other based on an explicitly signaled syntactic element. The syntactic element may be information indicating one of several inseparable transformation configurations. For example, if three types of inseparable transformation configurations are supported (i.e., n7 inseparable transformation sets and k7 candidate transformation kernels, n8 inseparable transformation sets and k8 candidate transformation kernels, and n9 inseparable transformation sets and k9 candidate transformation kernels), the syntactic element may have values ​​of 0, 1, or 2, and the value of the signaled syntactic element may determine which inseparable transformation configuration is applied to the current block.

[0102] The number of non-separable transformation sets and transformation kernel candidates may be configured differently depending on whether a quadratic transformation is applied and / or what kind of quadratic transformation is applied. For example, if no quadratic transformation is applied, n 10 A set of inseparable transformations and k 10 A non-separable transform configuration can be applied, containing n transform kernel candidates. When a quadratic transform is applied, n 11 A set of inseparable transformations and k 11 A non-separable transformation configuration can be applied, including several candidate transformation kernels.

[0103] Different non-separable transformation configurations may be applied depending on the quantization parameter (QP) and / or the range of QP values. For example, when the QP value is small, n 12 A set of inseparable transformations and k 12A non-separable transformation configuration including n transformation kernel candidates can be applied. On the other hand, when the QP value is large, n 13 A set of inseparable transformations and k 13 An inseparable transformation configuration containing n transformation kernel candidates can be applied. If the QP value is less than or equal to a threshold (e.g., 32), it can be classified as having a small QP value; otherwise, it can be classified as having a large QP value. Alternatively, the range of QP values ​​can be divided into three or more ranges, and a different inseparable transformation configuration can be applied to each range.

[0104] For relatively large blocks, instead of using an inseparable transformation corresponding to the width and height of the block, the block can be divided into multiple subblocks, and inseparable transformations corresponding to the width and height of the subblocks can be used. For example, when performing an inseparable transformation on a 4x8 block, the 4x8 block can be divided into two 4x4 subblocks, and a 4x4 block-based inseparable transformation can be used for each 4x4 subblock. Alternatively, for an 8x16 block, it can be divided into two 8x8 subblocks, and an 8x8 block-based inseparable transformation can be used.

[0105] The aforementioned non-separated transformation set may be determined based on the current block's intra-prediction mode and mapping table. The mapping table can define the mapping relationship between a predefined intra-prediction mode and the non-separated transformation set. The predefined intra-prediction mode may include two non-directional modes and 65 directional modes. In general, non-separated transformations have a larger transformation kernel size than separated transformations. This means that the computational complexity required for the transformation process is higher, and the memory required to store the transformation kernel is larger. On the other hand, separated transformations can only consider statistical properties existing in the horizontal and / or vertical directions, while non-separated transformations can simultaneously consider statistical properties in a two-dimensional space including horizontal and vertical directions, thus providing better compression efficiency. Because the statistical properties and diversity of residuals differ depending on the direction of the intra-prediction mode, there may be cases where non-separated transformations are absolutely necessary, and there may be intra-prediction modes in which the characteristics of residuals can be sufficiently captured by separated transformations alone. Therefore, by pre-defining which transformations to use in the encoding and decoding devices according to the intra-prediction mode, the transformation process can be designed with optimized complexity and memory requirements. The non-directional mode may include a planar mode (number 0) and a DC mode (number 1), and the directional mode may include intra-predictive modes (numbers 2 through 66). However, this is illustrative, and the disclosure may also apply to cases where the number of predefined intra-predictive modes differs.

[0106] By applying WAIP (wide angle intra prediction), the predefined intra-prediction modes may further include intra-prediction modes -14 to -1 and intra-prediction modes 67 to 80.

[0107] Figure 5 illustrates the intra-prediction modes and their prediction directions related to this disclosure. Referring to Figure 5, modes -14 to -1 and 2 to 33, and modes 35 to 80 are symmetric in terms of prediction direction with respect to mode 34. For example, modes 10 and 58 are symmetric with respect to the direction corresponding to mode 34, and mode -1 is symmetric with mode 67. Therefore, for vertical modes that are symmetric with respect to horizontal modes with respect to mode 34, the input data can be used after transposing. Transposing the input data means that in the 2D block of input data MxN, rows become columns and columns become rows, forming NxM data.

[0108] For example, when a 4x4 block is used, the 16 data points forming the 4x4 block can be appropriately arranged for the inseparable transformation to construct a 16x1 one-dimensional vector. In this case, the one-dimensional vector may be constructed in row-major order or column-major order. The residual samples resulting from the inseparable transformation can be arranged in the above order to construct a two-dimensional block.

[0109] For modes -14 to -1 and 2 to 33, the data array order for constructing the 16x1 input vector is row-major, while for modes 35 to 80, the input vector may be constructed using column-major order.

[0110] Mode 34 can be said to be neither a horizontal nor a vertical mode, but in this disclosure, it is classified as belonging to the horizontal mode. That is, for modes -14 to -1 and 2 to 33, the input data sorting method for horizontal modes, i.e., row-major ordering, is used, and the input data can be transposed and used for vertical modes that are symmetrical with respect to mode 34.

[0111] In non-square blocks, the symmetries found in square blocks (i.e., the symmetry between the P mode and the (68-P) mode in an NxN block (2<=P<=33) or between the Q mode and the (66-Q) mode (-14<=Q<=-1)) cannot be utilized. Therefore, in addition to symmetries based solely on the intra-predicted modes, it is also possible to utilize the symmetries between block configurations that have a prior relationship with each other, namely the symmetry between KxL blocks and LxK blocks. Specifically, a symmetry relationship exists between a KxL block predicted to be the P mode and an LxK block predicted to be the (68-P) mode. Or, a symmetry relationship exists between a KxL block predicted to be the Q mode and an LxK block predicted to be the (66-Q) mode.

[0112] Since the KxL block having mode 2 and the LxK block having mode 66 are considered symmetrical to each other, the same transformation kernel can be applied to both the KxL and LxK blocks. Assuming that an inseparable transformation set is mapped to the intra-prediction mode of the KxL block, in order to apply an inseparable transformation to the LxK block, the inseparable transformation set can be induced by the mapping table corresponding to the KxL block, based on mode (68-P) instead of mode P applied to the LxK block. Alternatively, the inseparable transformation set can be induced by the mapping table corresponding to the KxL block, based on mode (66-Q) instead of mode Q applied to the LxK block.

[0113] For example, to apply an inseparable transformation to an LxK block, the set of inseparable transformations can be selected based on mode 2 instead of mode 66. For a KxL block, the input data can be read in a predetermined order (e.g., row-major or column-major) to construct a one-dimensional vector, and then the inseparable transformation can be applied. For an LxK block, the input data can be read in a transposed order to construct a one-dimensional vector, and then the inseparable transformation can be applied. That is, if a KxL block is read in row-major order, an LxK block can be read in column-major order. Conversely, if a KxL block is read in column-major order, an LxK block can be read in row-major order.

[0114] Furthermore, when mode 34 is applied to a KxL block, the inseparable transformation set can be determined based on mode 34, the input data can be read in a predetermined order to construct a one-dimensional vector, and the inseparable transformation can be performed. Similarly, when mode 34 is applied to an LxK block, the inseparable transformation set can be determined based on mode 34, but the input data can be read in a transposed order to construct a one-dimensional vector, and the inseparable transformation can be performed.

[0115] This disclosure describes a method for determining an inseparable transformation set and a method for constructing input data based on a KxL block. However, the inseparable transformation can be performed using an LxK block as the basis, utilizing the same symmetry as described above for the KxL block. Alternatively, the use of a base block may be restricted to blocks where the width is greater than the height. Alternatively, the use of symmetry may be restricted to non-square blocks. In this case, non-square blocks may use a different number of inseparable transformation sets and / or transformation kernel candidates than square blocks, and may also use a different mapping table than square blocks to select an inseparable transformation set.

[0116] An example of a mapping table for selecting an unseparated set of transformations is as follows:

[0117] [Table 1]

[0118] Table 1 shows an example of assigning non-separable transformation sets to each intra-prediction mode when there are five non-separable transformation sets. The value of predModeIntra represents the value of the intra-prediction mode considering WAIP, and TrSetIdx is an index indicating a specific non-separable transformation set. From Table 1, it can be seen that the same non-separable transformation set is applied to modes located in a symmetrical direction relative to each other by the intra-prediction mode. Table 1 is just one example of using five non-separable transformation sets and does not limit the total number of non-separable transformation sets for non-separable transformation.

[0119] Alternatively, as shown in Table 2, non-destructive conversion may not be applied to WAIP for the sake of compression performance.

[0120] [Table 2]

[0121] Alternatively, as shown in Table 3, instead of configuring a separate non-separated conversion set for WAIP, a non-separated conversion set corresponding to an adjacent intra-predictive mode may be shared.

[0122] [Table 3]

[0123] The aforementioned non-separable transformation set may include multiple transformation kernel candidates, and one of the multiple transformation kernel candidates may be selectively used. For this purpose, an index signaled by a bitstream may be used. Alternatively, one of the multiple transformation kernel candidates may be implicitly determined based on the context information of the current block. Here, the context information can mean the size of the current block, or whether or not a non-separable transformation is applied to surrounding blocks. Here, the size of the current block may be defined as the width, height, the maximum / minimum values ​​of the width and height, the sum of the width and height, or the product of the width and height.

[0124] The following describes in detail how to determine the transformation kernel for the inverse transformation of the current block.

[0125] Example 1

[0126] As mentioned above, inverse transformations can be distinguished into separable and inseparable transformations. A separable transformation means performing transformations on a 2D block in both the horizontal and vertical directions, while an inseparable transformation means performing a single transformation on samples that constitute the entire 2D block or a portion of it. When describing a separable transformation, it can be described as a pair of horizontal and vertical transformations, and in this disclosure, it will be described as (horizontal transformation, vertical transformation).

[0127] Multiple transformation sets may be defined for the inverse transformation of a block. Each transformation set may contain one or more candidate transformation kernels.

[0128] For example, any one of (DST-7,DST-7), (DCT-8,DST-7), (DST-7,DCT-8), or (DCT-8,DCT-8) may be applied as a separate transform, and these four candidate transform kernels may be considered as a single transform set. Also, (DCT-2,DCT-2) may be considered as a single transform set. A transform skip, where no transform is applied, may also be considered as a single transform set, and (DCT-2,DCT-2) and the transform skip may be considered as a single transform set. In this disclosure, a transform kernel may represent a single transform (e.g., DCT-2, DST-7) or a pair of transforms (e.g., (DCT-2,DCT-2)).

[0129] Furthermore, another example of a set of transformations is the non-separable transformation set mentioned above. In this disclosure, a non-separable transformation applied as a primary transformation can be denoted as an NSPT (Non-Separable Primary Transform). An NSPT may consist of multiple sets of non-separable transformations, each of which may contain one or more transformation kernels as candidate transformation kernels. In the case of an NSPT, one of the multiple sets of non-separable transformations is selected by an intra-prediction mode, and the multiple sets of non-separable transformations for an NSPT can be denoted as an NSPT set list. This is as described above, and a detailed explanation is omitted here.

[0130] A group of one or more transformation sets available to the current block can be formed from a predefined set of transformations. This group of one or more transformation sets may consist of a predetermined region unit to which the current block belongs, and will hereinafter be referred to as a collection. Here, the predetermined region unit may be at least one of a picture, a slice, a coding tree unit row (CTU row), or a coding tree unit (CTU).

[0131] For example, let the transformation set composed of (DCT-2, DCT-2) be called S1, and the transformation sets composed of (DST-7, DST-7), (DCT-8, DST-7), (DST-7, DCT-8), and (DCT-8, DCT-8) be called S2, respectively. Also, the above-mentioned NSPT set list may include N non-separable transformation sets, and the N non-separable transformation sets are called S 3,1 , S 3,2 ,..., S 3,N , respectively. Here, N may be 35, but is not limited thereto.

[0132] When S 3,13 is selected as the non-separable transformation set for NSPT according to the intra prediction mode of the current block, the transformation kernel applicable to the current block may belong to any one of S1, S2, or S 3,13 . In this case, the collection available to the current block can be denoted as {S1, S2, S 3,13}.

[0133] As described above, since the collection according to the present disclosure is a group of one or more transformation sets available to the current block, the collection may be configured differently according to the context of the current block. Here, the context may include at least one of form, size, or intra prediction mode. If a total of K contexts are defined, K collections may be generated, and each collection may be denoted as C i (i = 1, 2,..., N). For example, when the sizes of the blocks to which NSPT can be applied are 4x4, 8x8, 16x16, 32x32, and any one of a total of 35 non-separable transformation sets is selected according to the intra prediction mode, if the transformation kernels applied for each block size are different, a total of 4x35 = 140 contexts may be defined.

[0134] A collection may be constructed based on the context of the current block. In this case, one of several transformation sets belonging to the collection may be selected, and one of several transformation kernel candidates belonging to the selected transformation set may be selected. Here, the selection of the transformation set and the transformation kernel candidate may be performed implicitly based on the context of the current block, or based on an explicitly signaled index. Alternatively, the process of selecting one of several transformation sets belonging to the collection and the process of selecting one of several transformation kernel candidates belonging to the selected transformation set may be performed separately. For example, an index for selecting a transformation set may be signaled first, and based on this, one of several transformation sets belonging to the collection can be selected. Then, an index pointing to one of several transformation kernel candidates belonging to the transformation set may be signaled, and based on the signaled index, one transformation kernel candidate can be selected from the transformation set. The transformation kernel of the current block may be determined based on the selected transformation kernel candidate. Alternatively, the selection of one transformation set from the collection may be implicitly performed based on the current block's context, and the selection of one transformation kernel candidate from the selected transformation set may be performed based on a signaled index. Alternatively, the selection of one transformation set from the collection may be performed based on a signaled index, and the selection of one transformation kernel candidate from the selected transformation set may be implicitly performed based on the current block's context. Alternatively, the selection of one transformation set from the collection may be implicitly performed based on the current block's context, and the selection of one transformation kernel candidate from the selected transformation set may also be implicitly performed based on the current block's context. Of course, if the number of transformation sets belonging to the collection is one, an index for selecting a transformation set does not need to be signaled.Similarly, if there is only one candidate transformation kernel belonging to the selected transformation set, an index pointing to that candidate does not need to be signaled. Alternatively, an index pointing to any one of the transformation kernel candidates currently belonging to the collection may be signaled. In this case, the process of selecting one transformation set from the collection may be omitted. At this time, all transformation sets belonging to the collection may be rearranged considering priority. For example, when assigning small-length binary code to small-value indices, such as truncated unary code, it may be advantageous to assign small-value indices to transformation kernel candidates that are relatively advantageous in improving coding performance. When rearranging all transformation kernel candidates belonging to a collection according to priority (shuffling), different shuffling can be applied to each collection. Also, instead of rearranging all transformation kernel candidates belonging to a collection, only some can be selectively rearranged.

[0135] Example 2

[0136] The transform kernel for the inverse transform of the current block may be determined based on MTS (Multiple Transform Selection).

[0137] The MTS relating to this disclosure may use at least one of DST-7, DCT-8, DCT-5, DST-4, DST-1, or IDT (identity transform) as a transformation kernel. The MTS relating to this disclosure may further include a DCT-2 transformation kernel.

[0138] In this disclosure, multiple sets of MTS may be defined for an MTS. Currently, one of the multiple sets of MTS may be determined based on the block size and / or intra-prediction mode. For example, when determining one set of MTS, 16 transformation block sizes may be considered, and for directional modes, the morphology of the transformation block and the symmetry between intra-prediction modes may be considered. For WAIP (Wide Angle Intra Prediction) modes (i.e., -1 to -14 (or -15), 67 to 80 (or 81)), the MTS set corresponding to mode 2 may be applied to modes -1 to -14 (or -15), and the MTS set corresponding to mode 66 may be applied to modes 67 to 80 (or 81). A different set of MTS may be assigned to MIP (Matrix-based Intra Prediction) modes.

[0139] For example, the MTS set based on the conversion block size and intra prediction mode may be allocated / defined as shown in Table 4 below.

[0140] [Table 4]

[0141] Table 4 shows the assignment of MTS sets based on 16 different conversion block sizes and intra-prediction modes. There are 80 predefined MTS sets, and the index that points to any one of these 80 MTS sets may range from 0 to 79, as shown in Table 4.

[0142] [Table 5-1] [Table 5-2]

[0143] Table 5 shows the transformation kernel candidates included in each MTS set described in Table 4. Each MTS set may consist of six transformation kernel candidates. The transformation kernel candidate index has one value from 0 to 5 and can indicate any one of the six transformation kernel candidates. Here, each transformation kernel candidate may be a combination of a horizontal transformation kernel and a vertical transformation kernel for decoupling, and 25 transformation kernel candidates with indices from 0 to 24 may be defined.

[0144] [Table 6]

[0145] Table 6 shows an example of the 25 transformation kernel candidates described in Table 5. Specifically, the horizontal and vertical transformations of the transformation kernel candidates are shown as (horizontal transformation, vertical transformation). For each transformation kernel candidate index, the horizontal / vertical transformation when the intra-prediction mode is less than 35 may be the opposite of the horizontal / vertical transformation when the intra-prediction mode is 35 or greater. When the value of the intra-prediction mode is 35 or greater, a mode that is symmetrical around mode 34 can be induced, and an MTS set can be selected from Table 4 based on that mode. Furthermore, the symmetry of the block shape may be further considered. If the original transformation block has a WxH size, the original transformation block can be made symmetrical and considered to have an HxW size, and an MTS set can be selected from Table 4. Here, the value of the intra-prediction mode may be the value of the modified intra-prediction mode. In other words, for WAIP, modes -14 (or -15) through -1 are modified to mode 2, modes 67 through 80 (or 81) are modified to mode 66, and the remaining modes can be set to the modified intra-prediction mode values ​​using the original intra-prediction mode values. In this case, the extended modes for WAIP are also configured symmetrically around mode 34, so symmetry around mode 34 can be used for all directional modes except Planar mode and DC mode.

[0146] For example, if a 16x32 block is predicted to be mode 54, then mode 14 (=68-54) is induced as a mode symmetric to mode 54, and the block size may be considered to be 32x16. In this case, an MTS set with 72 indices may be selected, as defined in Table 4.

[0147] When MIP mode is applied, the MTS set assigned to MIP mode may be selected based on the current block size, without considering the symmetry of the block configuration. Alternatively, when MIP mode is applied, the MTS set assigned to MIP mode may be selected based on the symmetrical block size, taking into account the symmetry of the block configuration. For example, when MIP mode is applied to an 8x16 block, the 8x16 block may be considered a symmetrical 16x8 block, and the MTS set with 49 indices, as defined in Table 4, may be selected. Alternatively, when MIP mode is applied, the intra-prediction mode may be considered Planar mode. In this case, the MTS set assigned to MIP mode may be selected based on the current block size, without considering the symmetry of the block configuration. Alternatively, the MTS set assigned to MIP mode may be selected based on the symmetrical block size, taking into account the symmetry of the block configuration.

[0148] In MIP mode, a flag may be used to indicate whether or not MIP mode is applied as transpose mode. When MIP mode is applied to the current block of MxN and the flag indicates the application of transpose mode, the intra-prediction mode is considered to be Planar mode, and the current block of MxN may be considered to be an NxM block. That is, an MTS set corresponding to the NxM block size and Planar mode may be selected from Table 4. As explained in Table 6, when the value of intra-prediction mode is 35 or greater, horizontal and vertical transformations are swapped with each other, but since the intra-prediction mode of the current block is considered to be Planar mode, it is not necessary to swap the horizontal and vertical transformations of the transformation kernel candidates. Alternatively, when MIP mode is applied to the current block of MxN and the flag indicates the application of transpose mode, the intra-prediction mode is not considered to be Planar mode, and the current block of MxN may be considered to be an NxM block. That is, an MTS set corresponding to the NxM block size and MIP mode may be selected from Table 4.

[0149] In Table 5, the transformation kernel candidate selected by the transformation kernel candidate index may be set as the transformation kernel for the current block. Alternatively, depending on the size of the current block, at least one of the horizontal or vertical transformations of the selected transformation kernel candidate may be changed to another transformation kernel. For example, if the transformation kernel candidate index is 3 and the width and height of the current block are both 16 or less, at least one of the horizontal or vertical transformations of the transformation kernel candidate corresponding to the transformation kernel candidate index of 3 may be changed to another transformation kernel. In this case, the horizontal and vertical transformations may be changed independently of each other. If the difference (or the absolute value of the difference) between the intra-prediction mode value and the horizontal mode value of the current block is less than or equal to a predetermined threshold, the vertical transformation of the selected transformation kernel candidate may be changed to the IDT (identity transformation). If the difference (or the absolute value of the difference) between the intra-prediction mode value and the vertical mode value of the current block is less than or equal to a predetermined threshold, the horizontal transformation of the selected transformation kernel candidate may be changed to the IDT (identity transformation). Here, the threshold may be determined based on the width and height of the current block as shown in Table 7.

[0150] [Table 7]

[0151] Table 7 defines thresholds based on the size of the transformation block for changing the horizontal and / or vertical transformations of the transformation kernel candidate selected by the transformation kernel candidate index to other transformation kernels.

[0152] The six translation kernel candidates constituting a single MTS set may be distinguished by translation kernel candidate indices from 0 to 5, as defined in Table 5. These translation kernel candidate indices may be signaled by a bitstream. A flag indicating whether an MTS set is available / applicable (MTS enabled flag or MTS flag) may be signaled, and if the flag indicates that the MTS set is available / applicable, the translation kernel candidate index may be signaled. The MTS flag may consist of a single bin, and one or more CABAC-based entropy coding contexts (hereinafter referred to as CABAC contexts) may be assigned to that bin. For example, different CABAC contexts may be assigned for when MIP mode is not active and for when MIP mode is active.

[0153] The number of available transformation kernel candidates for the current block may be set differently depending on the context of the current block as described above. For example, the sum of the absolute values ​​of all or some of the transformation coefficients in the current block may be considered as the context of the current block. This sum of the absolute values ​​of the transformation coefficients is called AbsSum. If AbsSum is less than or equal to T1, only one transformation kernel candidate corresponding to the transformation kernel candidate index of 0 may be available. If AbsSum is greater than T1 and less than or equal to T2, four transformation kernel candidates corresponding to the transformation kernel candidate indexes from 0 to 3 may be available. If AbsSum is greater than T2, six transformation kernel candidates corresponding to the transformation kernel candidate indexes from 0 to 5 may be available. Here, T1 may be 6 and T2 may be 32, but this is just an example.

[0154] If AbsSum is less than or equal to T1, there is currently one available translation kernel candidate for the block, so the translation kernel candidate corresponding to the translation kernel candidate index of 0 may be set as the translation kernel for the block without signaling of the translation kernel candidate index. If AbsSum is greater than T1 and less than or equal to T2, there are four available translation kernel candidates, so one of the four translation kernel candidates may be selected based on the translation kernel candidate index having two bins. That is, the translation kernel candidate indices 0 to 3 may be signaled as 00, 01, 10, and 11, respectively. The MSB (Most Significant Bit) may be signaled first for the two bins, and the LSB (Least Significant Bit) may be signaled later. Different CABAC contexts may be assigned to each bin. For example, a different CABAC context than the one assigned for the MTS flag may be assigned to each of the two bins. Alternatively, a CABAC context may not be assigned to two bins, and bypass coding may be applied. When AbsSum is greater than T2, the candidate translation kernel index has values ​​from 0 to 5, so the candidate translation kernel index cannot be represented by only two bins. In this case, the candidate translation kernel index can be represented by assigning two or more bins, as in truncated binary coding. A CABAC context may be assigned to each bin assigned by the truncated binary coding scheme, or bypass coding may be applied without assigning a CABAC context. Alternatively, a CABAC context may be assigned to some of the bins (for example, the first bin, or the first and second bins), and bypass coding may be applied to the remaining bins.

[0155] Example 3

[0156] The current block's translation kernel may be determined based on a translation set containing one or more translation kernel candidates. The current block's translation kernel may be induced to be one of the one or more translation kernel candidates belonging to the translation set.

[0157] The process of determining the current block's translation kernel may include at least one of the following: 1) the process of determining the current block's translation set, or 2) the process of selecting one translation kernel candidate from the current block's translation set. The process of determining the translation set may be the process of selecting one of a plurality of translation sets that are identically predefined for the encoding device and the decoding device. Alternatively, the process of determining the translation set may be the process of configuring one or more translation sets that are available to the current block from a plurality of translation sets that are identically predefined for the encoding device and the decoding device, and selecting one of the configured translation sets. Alternatively, the process of determining the translation set may be the process of configuring one translation set based on the translation kernel candidate that is available to the current block from a plurality of translation kernel candidates that are identically predefined for the encoding device and the decoding device.

[0158] If the current block's translation set contains multiple translation kernel candidates, the process of selecting one of the multiple translation kernel candidates for the current block may be performed. However, if the current block's translation set contains only one translation kernel candidate (i.e., if there is only one translation kernel candidate available for the current block), the current block's translation kernel may be set to that translation kernel candidate.

[0159] The transformation set relating to this disclosure may mean the (non-separable) transformation set in Example 1 described above, or the MTS set in Example 2. Alternatively, the transformation set may be defined separately from the (non-separable) transformation set of Example 1 or the MTS set of Example 2. In this case, the transformation set may include one or more specific transformation kernels as candidate transformation kernels. A specific transformation kernel may be defined as a pair of transformation kernels for horizontal transformation and transformation kernels for vertical transformation, or as a single transformation kernel that is applied identically to both horizontal and vertical transformations.

[0160] In the embodiments of this disclosure, the NSPT application process, which is an unseparable transformation applied as a linear transformation, will be described in detail. The NSPT may be applied to all or part of the transformation block. Based on the forward NSPT, the residual samples present in the region to which the NSPT is applied may be input to a one-dimensional vector of the NSPT. That is, the residual samples present in the entire or a part of a transformation block (referred to in this disclosure as a Region of Interest, ROI) can be collected into a one-dimensional vector and configured as input. Subsequently, by applying the forward NSPT, linear transformation coefficients can be obtained. Conversely, by applying the reverse NSPT to the linear transformation coefficients, a one-dimensional vector output can be obtained. By placing each element value constituting the output vector at a fixed position inside the 2D transformation block, residual samples for the ROI can be obtained.

[0161] The dimension of the matrix of the non-separable transform kernel for NSPT may be determined by the size of the ROI. In this disclosure, the transform kernel may be referred to as the transform type or the transform matrix, and the non-separable transform kernel for NSPT may be referred to as the NSPT kernel. For example, if the current block is an MxN transform block, the ROI is the entire area of ​​the MxN transform block, and a square NSPT is applied, the dimension of the transform matrix may be MNxMN. For example, if the ROI is the entire area of ​​an 8x8 transform block, the dimension of the NSPT kernel may be 64x64.

[0162] According to one embodiment of the present disclosure, when NSPT is applied to a residual generated by intra-prediction, the NSPT kernel can be adaptively determined by the intra-prediction mode. Since the statistical properties of the residual block may differ for each intra-prediction mode, the compression efficiency can be improved by adaptively determining the NSPT kernel by the intra-prediction mode.

[0163] The NSPT kernel can be configured to share an application for one or more intra-prediction modes. As mentioned above, the non-separable transformation set may be determined based on the intra-prediction mode of the current block and the mapping table. The mapping table can define the mapping relationships between predefined intra-prediction modes and the non-separable transformation set. The predefined intra-prediction modes may include two non-directional modes and 65 directional modes.

[0164] As one embodiment, intra-prediction modes can be grouped into intra-prediction mode groups. An intra-prediction mode group may be assigned one NSPT kernel, or multiple NSPT kernels. In other words, an intra-prediction mode group may be assigned an inseparable transformation set (NSPT set) containing one or more NSPT kernels. An inseparable transformation set is mapped to an intra-prediction mode, and one of the N NSPT kernels included in the inseparable transformation set may be selected.

[0165] As an example, an intra-prediction group may include adjacent prediction modes (e.g., modes 17, 18, and 19). Furthermore, an intra-prediction group may include modes that are symmetrical. For example, directional modes may be symmetrical around the diagonal mode in Figure 5 (i.e., intra-prediction mode 34). In this case, two symmetrical modes can form a group (or pair). For example, modes 18 and 50 are symmetrical around mode 34 and can therefore be included in the same group. However, for symmetrical modes, an additional step may be taken to transpose the 2D input block before applying the forward NSPT kernel and then construct a one-dimensional input vector. For example, if the intra-prediction mode is 34 or less, a one-dimensional input vector can be derived from the input block in row-first order without transposing the 2D input block. If the intra-prediction mode is greater than 34, a one-dimensional input vector can be constructed by first transposing the 2D input block and then reading the input block in row-first order, or by leaving the 2D input block as is and reading the input block in column-first order.

[0166] Table 8 below illustrates an NSPT set assignment mapping table for intra-predictive mode. Referring to Table 8, a total of 35 NSPT sets, from 0 to 34, may be defined. Extended WAIP modes (i.e., modes -14 to -1 and modes 67 to 80 in Figure 5) may be assigned the NSPT set assigned to the nearest general directional mode. That is, extended WAIP modes may be assigned the second NSPT set.

[0167] [Table 8]

[0168] An NSPT set may contain one or more NSPT kernels (or kernel candidates). That is, an NSPT set may contain N NSPT kernel candidates. For example, N may be set to a value equal to or greater than 1, such as 1, 2, 3, 4, etc. Of the one or more NSPT kernels contained in the NSPT set, the kernel that applies to the current block can be signaled using an index. In this disclosure, such index may be referred to as an NSPT index. For example, an NSPT index may have values ​​of 0, 1, 2, ..., N-1.

[0169] Furthermore, as one embodiment, if there is only one candidate NSPT kernel, the NSPT index value may be fixed to 0. In this case, the NSPT index may be inferred without being separately signaled. In addition, a flag indicating whether or not NSPT is applied may be signaled separately from the NSPT index. In this disclosure, this flag may be referred to as the NSPT flag.

[0170] If the value of the NSPT flag is 1, NSPT may be applied. If the value of the NSPT flag is 0, NSPT may not be applied. If the NSPT flag is not signaled, the value of the NSPT flag may be inferred to be 0. As an example, if the value of the NSPT flag is 1, the NSPT index may be signaled. Based on the signaled NSPT index, one of the N kernel candidates included in the NSPT set selected by the intra-prediction mode may be identified.

[0171] In one embodiment, the entropy coding method for NSPT indices may be defined in various ways, taking into account the number (N) of NSPT kernels included in the NSPT set. For example, truncated unary binarization, truncated binary, or fixed-length binary may be used as methods for mapping values ​​from 0 to N-1 to binstrings (i.e., binary methods).

[0172] For example, if the number of kernel candidates constituting the NSPT set (N-value) is 2, one bin can identify one of the two candidates. For example, 0 could indicate the first candidate, and 1 could indicate the second candidate. Also, if the N-value is 3 and truncated unary binary evolution is applied, two bins may be used to identify the candidates. For example, the first, second, and third candidates can be signaled by binary evolution to 0, 10, and 11, respectively. In practice, the binary-evolved bins may be coded by context coding or bypass coding.

[0173] This disclosure describes a reduced primary transform (RPT) method using a reduced-dimensional transformation kernel as a primary transformation. As mentioned above, when forward NSPT is applied, samples belonging to a 2D residual block may be arranged (or rearranged) into a 1D vector according to row priority (or column priority). Then, a transformation matrix for NSPT may be multiplied by the arranged vector. When the 2D residual block is an MxN block (M is horizontal, N is vertical), the length of the rearranged 1D vector may be M*N. That is, the 2D residual block can be represented by a column vector having M*Nx1 dimensions. In this disclosure, M*N may be denoted as MN for convenience. In this case, the dimension of the transformation matrix may be MNxMN. In short, forward NSPT can operate in a way that obtains an MNx1 transformation coefficient vector by multiplying the left side of the MNx1 vector by the MNxMN transformation matrix.

[0174] When RPT is applied, instead of multiplying the forward NSPT transformation matrix by an MNxMN matrix as described above, r transformation coefficients can be obtained by multiplying by an rxMN matrix. Here, r represents the number of rows in the transformation matrix, and MN represents the number of columns in the transformation matrix. According to the embodiments of this disclosure, the value of r may be set to be less than or equal to MN. That is, an existing forward NSPT transformation matrix contains MN rows, each row being a 1xMN row vector, which is the transform basis vector of the NSPT transformation matrix. The transformation coefficients may be obtained by multiplying each transform basis vector by an MNx1 sample column vector.

[0175] An existing forward NSPT transformation matrix consists of MN row vectors, so applying forward NSPT can yield MN transformation coefficients (i.e., an MNx1 transformation coefficient column vector). On the other hand, in the case of forward RPT, the transformation matrix may be constructed with r transformation basis vectors instead of MN transformation basis vectors. This allows for obtaining r transformation coefficients (i.e., an rx1 transformation coefficient column vector) instead of MN when forward RPT is applied.

[0176] An RPT kernel can be constructed by selecting r transformation basis vectors, which are part of the transformation basis vectors that constitute an MNxMN forward NSPT kernel. In this disclosure, a transformation kernel may be referred to as a transformation type or transformation matrix, and an inseparable transformation kernel for NSPT may be referred to as an RPT kernel. That is, when selecting r 1xMN row vectors from an MNxMN forward NSPT kernel, it may be advantageous to select the transformation basis vectors that are most important in terms of coding performance. Specifically, in terms of energy compaction by transformation, greater energy may be concentrated in the transformation coefficients that appear earlier when multiplied by the forward NSPT transformation matrix. In other words, the higher the transformation basis vector is located on the forward NSPT transformation matrix, the greater the energy of the transformation coefficients that can be generated. Taking these points into consideration, an rxMN forward RPT kernel can be constructed (or induced) by taking r vectors from the top of the forward NSPT kernel.

[0177] The RPT relating to this disclosure takes only a portion (i.e., r) of the conversion coefficients obtained by applying the existing NSPT, which may result in a loss of some of the energy inherent in the signal. In other words, this process may cause distortion between the original signal and the original signal. Nevertheless, by applying RPT, only r conversion coefficients are generated instead of MN, thus reducing the number of bits required to code these conversion coefficients. Therefore, in the case of signals where a large amount of energy is concentrated in a small number of conversion coefficients (e.g., image resistive signals), the gain obtained by reducing the signaling bits is significantly large, thereby improving coding performance.

[0178] The inverse NSPT is a transformation matrix, which may be the transpose of the forward NSPT kernel described above. In this case, the input data may be a transformation coefficient signal instead of a sample signal such as a residual signal. Specifically, if the forward NSPT transformation matrix is ​​G and the sample signal rearranged into a 1D vector is x, the transformation coefficient vector obtained by multiplying the transformation matrix to the left may be expressed as shown in equation 4 below.

[0179] [Equation 4]

number

[0180] Referring to Equation 4, x and y may be MNx1 column vectors. G may have the form of an MNxMN matrix. The inverse NSPT process may be expressed using the same variables as in Equation 5 below.

[0181] [Formula 5]

number

[0182] In formula 5, G Trepresents the transpose matrix of G. The forward and reverse RPT operations according to this disclosure may also be expressed by the two formulas above. However, when RPT is applied, y is an rx1 column vector instead of an MNx1 column vector, and G is an rxMN matrix instead of an MMxMN matrix. That is, applying RPT instead of NSPT does not change the dimension of the sample signal (e.g., image residual signal), but this means that the original number of sample signals (i.e., MN sample signals) can be restored using only r transformation coefficients by reverse RPT. In other words, the original MN sample signals can be restored by coding only r transformation coefficients, which is fewer than MN, resulting in improved coding performance.

[0183] In one embodiment of this disclosure, we propose an RPT structure that defines an r-value considering the statistical properties of a residual block and derives a residual block of the existing transformed block size from a residual block of reduced size determined by the defined r-value. If an additional transformation (i.e., a quadratic transformation) is applied to predict the statistical distribution of the linear transformation coefficients, a quantization process is applied to the linear transformation coefficients, so that quantized non-zero coefficients may be concentrated in the relatively low frequency domain. Thus, a reduced secondary transform for the statistical distribution of the linear transformation coefficients can relatively easily define the statistical properties of the linear transformation coefficients in the form of setting an r-value for a given low frequency domain. However, the RPT according to this disclosure is a technique for defining an r-value considering the statistical properties of a sample in a residual block, which have properties very different from the distribution of the linear transformation coefficients, and thus has a fundamental difference from a reduced secondary transform. Below, we describe various embodiments for determining the RPT kernel, which is a transformation matrix of reduced dimensions. In other words, we describe below a method for determining or defining the r-value in RPT.

[0184] In one embodiment of this disclosure, the r value in RPT can be determined by considering the worst-case complexity tolerable by the conversion system. In one embodiment, the worst-case complexity can be calculated based on the number of multiplications per sample. To apply RPT in both the forward and reverse directions based on an MxN block, MN*r multiplications are required. Since a 2D block consists of a total of MN samples, the number of multiplications per sample can be calculated as (MN*r) / MN=r. Therefore, the r value can be configured to be kept below the maximum allowable number of multiplications per sample. For example, if the maximum possible number of multiplications per sample for a 16x16 block is set to 16, the r value may be determined to be 16 or less. That is, the forward RPT kernel may be set to 16x256.

[0185] In other embodiments, memory usage can be considered as a measure of worst-case complexity. For example, the allowable memory size per kernel can be set. For instance, if p bytes are required for each kernel coefficient (in this disclosure, each element constituting the translation kernel is referred to as a kernel coefficient), and the memory usage is set to be q bytes or less per kernel, then the r value may be set to q / (MN*p) or less. For example, if p is 1 byte for a forward RPT kernel for a 16x16 block, and the memory usage is set to be 8KB or less per kernel (q=8KB=2 13 (Bytes), the r value may be set to 32 or less.

[0186] Another example is to consider memory usage and / or the multiplication per sample as measures of worst-case complexity. For example, if the maximum possible multiplication per sample for a 16x16 block is set to 16 and the memory usage is set to 8KB or less per kernel (kernel coefficients are represented by 1 byte), then the r value may be set to 16 or less.

[0187] Furthermore, in one embodiment, the r value constituting the RPT kernel may be determined by specific information. In other words, the r value constituting the RPT kernel may be determined based on predefined coding parameters. For example, the r value may be determined by the size of the block. In other words, the RPT kernel may be variably determined by the size of the block. Here, the block may be at least one of a coding block, a transformation block, and a prediction block. Also, for example, the r value may be determined based on prediction information. Here, the prediction information may include information about inter / intra prediction, intra prediction mode information, etc. Also, for example, the r value may be determined based on signaled information (values ​​of syntax elements). For example, the r value may be variably determined by quantization parameter values. Also, in terms of complexity improvement, a predefined fixed value may be used as the r value, and the predefined fixed value may be determined based on signaled information.

[0188] Applying the RPT kernel rxMN to a sampled signal yields r conversion coefficients. These r conversion coefficients may be arranged according to a predefined scan order of conversion coefficients (e.g., forward / reverse zig-zag scan order, forward / reverse horizontal scan order, forward / reverse vertical scan order, forward / reverse diagonal scan order, scan order determined based on an intra-prediction mode, etc.). When arranging the conversion coefficients obtained from forward RPT application according to such a scan order (e.g., a coefficient group (CG) unit scan order is also applicable), if the r value is smaller than MN, it may not be necessary to fill the entire MxN block with r conversion coefficients, resulting in empty space. In an embodiment of this disclosure, the aforementioned empty space may be predicted in the following way, taking into account the characteristics of the residual signal.

[0189] - The values ​​of the available surrounding pixels can be used to fill in the values ​​in the empty space.

[0190] - The values ​​of the empty spaces can be filled in based on the values ​​of the available surrounding pixels and the intra-prediction mode. For example, the values ​​of the empty spaces can be predicted by performing intra-prediction based on the values ​​of the available surrounding pixels and the intra-prediction mode.

[0191] - The empty spaces can be filled with values ​​using predefined, fixed values ​​(for example, 0).

[0192] - The empty space can be filled with values ​​from available peripheral pixels using a predetermined intra-prediction mode (e.g., planar mode).

[0193] In this disclosure, filling empty spaces with zeros, as described above, may be referred to as the zero-out process. When filling empty spaces with zeros, the following embodiments may be applied: If a non-zero conversion coefficient is detected (or parsed) in the relevant empty space portion during the passing of conversion coefficients on the decoding device side, it can be considered (or inferred) that RPT will not be applied. In other words, if a non-zero conversion coefficient exists in a predefined region representing the relevant empty space, it can be considered that RPT will not be applied. In this case, signaling (or parsing) is not performed for a flag indicating whether or not RPT is applied and / or for an index specifying one of several RPT kernel candidates. As an example, if a non-zero conversion coefficient exists in a predefined region representing the relevant empty space, a predefined variable value may be updated, and based on the updated variable value, it can be inferred that RPT will not be applied.

[0194] In one embodiment of this disclosure, the application of RPT may be determined by the size and / or shape of the block. Furthermore, the RPT kernel may be variably determined by the size and / or shape of the block. Since the r value may differ depending on the size and / or shape of the block (i.e., for each MxN block), the available space may vary depending on the size and / or shape of the block. Consequently, the region for checking whether a non-zero conversion coefficient is detected may be defined differently for each block size and / or shape. In other words, the zero-out region may be variably determined.

[0195] For example, when a 16x64 matrix is ​​applied as the forward RPT matrix for an 8x8 block, the r value may be 16. In this case, when the CG is a 4x4 subblock, a non-zero RPT conversion coefficient may be filled only in the top-left 4x4 block, and the remaining three 4x4 subblocks (i.e., the top-right, bottom-left, and bottom-right subblocks) may be filled with zero values. If a non-zero conversion coefficient is detected in these remaining three 4x4 subblock regions during the decoding process, it can be considered that RPT is not applied. Furthermore, as mentioned above, a flag indicating whether or not RPT is applied, or an index specifying one of several RPT kernel candidates, does not need to be signaled.

[0196] As an example, when a 32x128 matrix is ​​applied as the forward RPT matrix for a 16x8 block (i.e., the r value is 32), and the CG is a 4x4 subblock, then, in terms of scan order, non-zero RPT conversion coefficients may be filled into only two CGs. For example, these RPT conversion coefficients may be filled into the upper-left 4x4 subblock and the 4x4 subblock adjacent to the lower side of the upper-left subblock. The area to be filled with zeros as empty space may be determined to be the remaining area other than those two 4x4 subblocks. The RPT kernel may be determined variably depending on the size and / or shape of the block, and as described, the empty space may be determined differently for 8x8 blocks and 16x8 blocks.

[0197] As an example, when the r value is a multiple of the CG size and the conversion coefficients are scanned in units of CG, if it is detected that a non-zero conversion coefficient exists in a CG belonging to an empty space, the flag and / or index associated with RPT does not need to be signaled. That is, the internal conversion coefficients of each CG can be scanned in a specified order, and the system can move to the next CG according to the scan order for each CG unit and scan the internal conversion coefficients of the CG identically. In conventional image compression techniques, a flag indicating whether or not a non-zero conversion coefficient exists within each CG is signaled in advance, so the application of RPT can be determined based solely on this information, thereby reducing signaling overhead and decreasing the complexity of associated implementation.

[0198] As mentioned above, if a non-zero transformation coefficient is detected in the empty space region that would normally be filled with zeros when RPT is applied, RPT does not need to be applied. In this case, signaling for information related to RPT may be omitted. However, if no non-zero transformation coefficient is detected in the empty space region, it is not possible to determine whether RPT is applied or not. Therefore, after parsing (or signaling) the related transformation coefficients, a flag indicating whether RPT is applied or not can be parsed to make a final decision on whether RPT is applied or not.

[0199] As one embodiment, a forward quadratic transformation can be further applied to the transformation coefficients generated by the application of RPT. Alternatively, a forward quadratic transformation can be further applied to the region in an MxN block where the generated transformation coefficients are located. In this disclosure, such region or a part thereof may be referred to as an ROI from the perspective of the forward quadratic transformation. For the reverse direction, a reverse quadratic transformation can be applied first, followed by a reverse RPT. Specifically, a region or a part thereof where the r transformation coefficients generated by the application of forward RPT are located can be set as an ROI, and a forward quadratic transformation can be applied to it. In this case, when a 16x64 forward RPT transformation matrix is ​​applied to an 8x8 region, the 16 generated transformation coefficients may be located in a 4x4 sub-block at the top left, and this sub-block region can be set as an ROI, and a forward quadratic transformation can be applied to this ROI.

[0200] Furthermore, the coefficient values ​​of the RPT kernel may be adjusted to accommodate operations such as integer arithmetic or fixed-point arithmetic. That is, the RPT kernel can be configured to appropriately scale the kernel coefficients belonging to the kernel that are not theoretical orthogonal or non-orthogonal transformations (where orthogonal and non-orthogonal transformations represent transformations in which the norm of each transformation basis vector is 1) so that the transformation is performed by integer arithmetic (or fixed-point arithmetic) in the actual codec system. The scaling factor applied when applying a deseparated transformation in existing image compression techniques can be similarly reflected when applying RPT. In this case, deseparated or non-deseparated transformations (including RPT) can be performed while maintaining other processes other than the transformation (e.g., quantization, de-quantization).

[0201] By multiplying the transformation basis vector by the scaling value described above, the integerized coefficients of the RPT kernel can be obtained. In practice, multiplying by the scaling value may include applying operations such as rounding, truncation, and rounding up to each kernel coefficient. That is, the integerized RPT kernel obtained by the method described above is defined and may be used in the transformation / inverse transformation process. As described above, by obtaining the scaled integer kernel coefficients through operations such as rounding, truncation, and rounding up, the maximum and minimum values ​​can be found for all kernel coefficients, and from these maximum and minimum values, a number of bits sufficient to represent all kernel coefficients can be obtained. For example, if the maximum value is 127 or less and the minimum value is -128 or greater, all integer kernel coefficients can be represented with 8 bits (especially by two's complement representation, etc.).

[0202] In general, if the maximum value is (2 (N-1) -1) or less, and the minimum value is -2 (N-1) If the above is true, then all integer kernel coefficients can be represented with N bits. For example, if the maximum value is (2 (N-1) -1) greater than or with a minimum value of -2 (N-1)If the range is smaller, it may not be possible to represent all integer kernel coefficients with N bits. In such cases, 1) all kernel coefficients can be further scaled to fit within the N-bit range, or 2) the number of bits required to represent the kernel coefficients can be increased (i.e., N+1 bits or more). For example, if all kernel coefficients are multiplied by a scaling value to represent them with N bits, -p If only multiplication is needed (p>=1), then, to integrate with the existing encoding / decoding process, 2 p It can be compensated by multiplying only by 2. p Multiplying by only this can be implemented by methods such as performing a further shift operation of p bits to the left, or by reducing the amount of right-side shift applied in the quantization or dequantization process by p.

[0203] Using the methods described above, all kernel coefficients can be represented using 8 bits, 9 bits, 10 bits, etc. Of course, the block size and the scaling value of the kernel coefficients can be set differently for each kernel, and the number of bits used to represent the kernel coefficients can also be set differently.

[0204] The aforementioned NSPT may be applied based on at least one of the current block size, tree type, or component type. For example, the decision to apply NSPT may be made based on at least one of the current block size, tree type, or component type. An NSPT index may be signaled based on at least one of the current block size, tree type, or component type. An NSPT set or NSPT kernel may be induced based on at least one of the current block size, tree type, or component type.

[0205] The allowed transform block sizes predefined for the decoding device can be broadly divided into two groups. Either of the two groups (hereinafter referred to as the first group) can represent the set of block sizes to which NSPT can be applied. The first group may consist of any one of the allowed transform block sizes, or it may consist of two or more block sizes from the allowed block sizes. A block size to which NSPT can be applied may be defined as a block size in which at least one of the width and height is less than or equal to a predetermined threshold. Alternatively, a block size to which NSPT can be applied may be defined as a block size in which the product of the width and height is less than or equal to a predetermined threshold. Alternatively, a block size to which NSPT can be applied may be defined as a block size in which the maximum value of the width and height is less than or equal to a predetermined threshold. The threshold may be an integer of 4, 8, 16, 32, 64, 128 or higher.

[0206] The other of the two groups mentioned above (hereinafter referred to as the second group) can represent the set of block sizes to which NSPT does not apply. The aforementioned separable linear transformation may be applied to the block sizes belonging to the second group. In addition, an unseparable quadratic transformation may be applied to all or part of the block sizes belonging to the second group.

[0207] For example, if the current block size belongs to the first group, the inverse NSPT may be applied to the (inversely quantized) transformation coefficients of the current block. If the current block size belongs to the second group, the inverse separated linear transformation may be applied to the (inversely quantized) transformation coefficients of the current block. Alternatively, if the current block size belongs to the second group, the inverse non-separable quadratic transformation (e.g., LFNST (low frequency non-separable transform)) may be applied first to the (inversely quantized) transformation coefficients of the current block, and then the inverse separated linear transformation (e.g., DCT-2) may be applied to the resulting transformation coefficients.

[0208] As an example, the first group, which is the set of block sizes to which NSPT can be applied, may be defined as the sets of 4x4, 4x8, 8x4, and 8x8. Or, the first group may be defined as the sets of 4x8, 8x4, and 8x8. Or, the first group may be defined as the sets of 4x8 and 8x4. Or, the first group may be defined as the sets of 4x4, 4x8, 4x16, 8x4, 8x8, and 16x4. Or, the first group may be defined as the sets of 4x8, 4x16, 8x4, 8x8, and 16x4. Or, the first group may be defined as the sets of 4x8, 4x16, 8x4, and 16x4. Or, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, and 16x16. Alternatively, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8. Alternatively, the first group may be defined as the sets of 4x8, 8x4, 8x8, 8x16, and 16x8. Alternatively, the first group may be defined as the sets of 4x8, 8x4, 8x16, and 16x8. Alternatively, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, 32x16, and 32x32. Alternatively, the first group may be defined as the sets of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 8x4, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 8x4, 8x16, 16x8, 16x32, and 32x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, and 16x4. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x16, 16x4, and 16x8.Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 8x4, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x8, 4x16, 8x4, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x16, 8x32, 16x4, 16x8, 32x4, and 32x8. Alternatively, the first group may be defined as the set of 4x8, 4x16, 4x32, 8x4, 8x16, 8x32, 16x4, 16x8, 32x4, and 32x8. The first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x16, 32x4, and 32x8. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x32, 32x4, 32x8, and 32x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x16, 16x32, 32x4, 32x8, and 32x16. Alternatively, the first group may be defined as the set of 4x8, 4x16, 4x32, 8x4, 8x16, 8x32, 16x4, 16x8, 16x32, 32x4, 32x8, and 32x16. Alternatively, the first group may be defined as the set of 4x4, 4x8, 4x16, 4x32, 8x4, 8x8, 8x16, 8x32, 16x4, 16x8, 16x16, 16x32, 32x4, 32x8, 32x16, and 32x32.

[0209] For block sizes belonging to the first group, an NSPT matrix (or NSPT kernel) with a predetermined dimension may be applied. Here, the NSPT matrix is ​​an inverse transformation matrix and may be expressed as a matrix with dimension PxQ, where PxQ represents a matrix with P and Q rows and columns, respectively.

[0210] For example, a 16x16 NSPT matrix may be applied to a 4x4 block. A 32x20 NSPT matrix may be applied to at least one of a 4x8 or 8x4 block. A 64x24 NSPT matrix may be applied to at least one of a 4x16 or 16x4 block. A 64x32 NSPT matrix may be applied to an 8x8 block. A 128x40 NSPT matrix may be applied to at least one of an 8x16 or 16x8 block. A 256x44 NSPT matrix may be applied to a 16x16 block. A 128x36, 128x38, or 128x40 NSPT matrix may be applied to a 4x32 or 32x4 block. A 256x48 NSPT matrix may be applied to an 8x32 or 32x8 block. For 16x32 or 32x16 blocks, a 512x52 or 512x54 NSPT matrix may be applied.

[0211] Since the PxQ matrix is ​​an inverse NSPT matrix, the Px1 output vector can be obtained by applying the PxQ matrix to the Qx1 input vector (i.e., (PxQ matrix) x (Qx1 input vector)). Here, the Qx1 input vector may correspond to the inversely quantized transformation coefficients within the current block to which NSPT is applied. In this case, the value of Q can represent the number of transformation coefficients to which NSPT is applied, and may be less than or equal to the product of the width and height of the current block. The value of Q may be variably determined based on the size of the current block from among the block sizes belonging to the first group described above. Alternatively, the value of Q may be set to be the same for all block sizes belonging to the first group. The Px1 output vector may correspond to the residual signal (or the decoded residual sample). The value of P may be equal to the product of the width and height of the current block.

[0212] Conversely, the forward NSPT matrix may be expressed as the QxP matrix, which is the transpose of the PxQ matrix. The Qx1 output vector can be obtained by applying the QxP matrix to the Px1 input vector (i.e., (QxP matrix) x (Px1 input vector)). Here, the Px1 input vector may correspond to the residual sample in the current block to which NSPT is applied. The value of P may be the same as the product of the width and height of the current block. The Qx1 output vector may correspond to the transformation coefficients in the current block induced by NSPT. In this case, the value of Q can represent the number of transformation coefficients output by NSPT, and may be less than or the same as the product of the width and height of the current block. Similarly, the value of Q may be variably determined based on the size of the current block from among the block sizes belonging to the first group described above. Alternatively, the value of Q may be set to be the same for all block sizes belonging to the first group.

[0213] As in the example above, NSPT may be applied to non-square blocks such as MxN and NxM blocks. For example, NSPT may be applied to 4x8 and 8x4 blocks. Alternatively, NSPT may be applied to 4x16 and 16x4 blocks, or to 8x16 and 16x8 blocks, or to 16x32 and 32x16 blocks.

[0214] By applying NSPT to a specific block size belonging to the first group described above, the transformation can be performed with greater accuracy, potentially improving coding performance. When forward LFNST is applied, the linearly transformed transformation coefficients of the remaining region outside the region to which LFNST is applied (i.e., Region-Of-Interest, ROI) may be zeroed out. Furthermore, LFNST may consist of a small number of transform basis vectors. In such cases, applying a separable linear transformation such as DCT-2 and an unseparable quadratic transformation such as LFNST instead of NSPT to the block size may result in performance degradation. Applying NSPT instead of LFNST in such cases omits the zero-out process, potentially improving coding performance compared to applying LFNST. Performance improvement can also be expected depending on the method of applying NSPT. NSPT or LFNST may be applied using symmetry, which will be described later. Here, LFNST uses symmetry only with respect to the ROI region to perform a transpose operation on the input block. In contrast, NSPT performs transpose operations using symmetry across the entire block. Therefore, in NSPT, it becomes possible to train and apply a more precise symmetry method to the NSPT kernel, which is expected to improve performance.

[0215] Furthermore, when applying LFNST instead of NSPT to an 8x8 block, a 32x64 transformation matrix can be applied instead of a 16x64 transformation matrix from the perspective of forward transformation. Here, the 16x64 transformation matrix may be constructed by sampling the top 16 rows of the 32x64 transformation matrix. When applying LFNST based on a 16x64 transformation matrix to an 8x8 block, 16 multiplications are required per sample to apply LFNST, but when using a 32x64 transformation matrix, 32 multiplications are required per sample to apply LFNST. However, when using a 32x64 transformation matrix in this way, an improvement in coding performance can be expected.

[0216] If the current block's tree type is single tree, NSPT can be applied to the rumor component of the current block, but not to the chroma component. If the current block's tree type is dual tree, NSPT can be applied to both the rumor and chroma components of the current block.

[0217] Alternatively, regardless of whether the current block's tree type is single-tree or not, NSPT may be applied to the rumor component of the current block, but not to the chroma component. Alternatively, regardless of whether the current block's tree type is single-tree or not, NSPT may be applied to both the rumor and chroma components of the current block.

[0218] For example, if the current block's tree type is a single tree, NSPT is permitted for the lumern and chroma components, and the current block's size belongs to the first group, then one NSPT index may be signaled, and the lumern and chroma components of the current block can share this NSPT index. Here, the NSPT index may be an index for selecting one of the candidate transformation kernels for NSPT. If the sizes of the lumern and chroma blocks of the current block belong to the first group, then the candidate transformation kernel selected by the same NSPT index may be applied to the lumern and chroma components. If the current block's tree type is a single tree and NSPT is applied only to the lumern component, then LFNST may not be applied to the chroma component of the current block, and a separate transformation may be applied instead. Alternatively, if the current block's tree type is a single tree and NSPT is applied only to the lumern component, then LFNST may be applied to the chroma component of the current block.

[0219] In the case of a single tree, the rumor and chroma components may have a high degree of correlation. In this case, applying NSPT only to the rumor component, or applying a conversion kernel candidate selected by a single NSPT index to both the rumor and chroma components, can reduce unnecessary signaling and improve compression efficiency. On the other hand, in the case of a non-single tree, the rumor and chroma components each have their own partitioning and encoding structures. In this case, signaling NSPT indices for each component can reflect the characteristics of each component and improve compression efficiency.

[0220] The NSPT kernel for the aforementioned NSPT may be induced based on at least one of the symmetries between intra-prediction modes or between block forms. For example, the NSPT kernel may be induced as an NSPT kernel corresponding to at least one of the modes symmetric to the intra-prediction mode of the current block, or a block form symmetric to the block form of the current block. Alternatively, the NSPT kernel may be induced based on an NSPT set containing one or more NSPT kernel candidates, where the NSPT set may be induced as an NSPT set corresponding to at least one of the modes symmetric to the intra-prediction mode of the current block, or a block form symmetric to the block form of the current block. One or more NSPT kernel candidates belonging to the NSPT set may be set as the NSPT kernel for the current block. For this purpose, an NSPT index may be used to identify one or more NSPT kernel candidates belonging to the NSPT set. The NSPT index may be signaled in a bitstream and may be induced based on the aforementioned symmetries.

[0221] Symmetry may exist between at least two of the intra-prediction modes predefined in the decoding device. For the sake of explanation, symmetry centered on the upper left diagonal mode (i.e., mode 34) will be described below. Referring to Figure 5, symmetry exists between the directional modes. All modes except Planar mode (mode 0) and DC mode (mode 1) have a prediction direction. Modes 2 through 6 are named normal directional modes (can be written as [2,66]), and modes -14 through -1 (can be written as [-14,-1]) and modes 67 through 80 (can be written as [67,80]) can be named wide directional modes. Wide directional modes may include at least one mode with a value less than -14 or a mode with a value greater than 80. Referring to Figure 5, all modes except mode 0 and mode 1 are symmetrical with respect to mode 34. Specifically, with respect to the [2,66] mode, the x-th mode and the (68-x)-th mode are symmetric, and between the [-14,-1] mode and the [67,80] mode, the x-th mode and the (66-x)-th mode are symmetric. A similar symmetry relationship may hold between the [N,-1] mode and the [67,66-N] mode. Here, N may be an integer less than or equal to -14.

[0222] On the other hand, in relation to the symmetry between block configurations, MxN blocks and NxM blocks may be defined as blocks that are symmetrical with each other. Here, M and N may be the same or different. Alternatively, if the ratio of the width to the height of an M1xN1 block (M1 / N1) and the ratio of the height to the width of an M2xN2 block (N2 / M2) are the same, then M1xN1 blocks and M2xN2 blocks may be defined as blocks that are symmetrical with each other. Alternatively, if the ratio of the width to the height of an M1xN1 block (M1 / N1) and the ratio of the width to the height of an M2xN2 block (M2 / N2) are the same, then M1xN1 blocks and M2xN2 blocks may be defined as blocks that are symmetrical with each other.

[0223] In a square block, modes that are symmetric to each other may share at least one of the following: an NSPT set, an NSPT index, or an NSPT kernel. That is, at least one of the NSPT sets, NSPT indexes, or NSPT kernels for one of the symmetric modes may be applied identically to the other of the symmetric modes.

[0224] As an example, modes that are symmetric to each other may share a single NSPT kernel. However, for one of the symmetric modes, the NSPT kernel can be applied to the input data, while for the other, the NSPT kernel can be applied after the input data has been transposed. Specifically, if mode x belongs to the [2,33] mode, for mode x, a one-dimensional vector (1D vector) can be constructed from the input data MxM block in a row-first order, and the NSPT kernel can be applied to this one-dimensional vector. Here, constructing a one-dimensional vector in a row-first order may be done by reading the input data from the input data MxM block column by column to obtain M columns, and then arranging these sequentially to construct a one-dimensional vector. On the other hand, for mode (68-x), which is symmetric to mode x, a one-dimensional vector can be constructed in a column-first order, and the same NSPT kernel can be applied to this one-dimensional vector. Here, the construction of a one-dimensional vector following row-first order may be done by reading the input data row by row from the input data, which is an MxM block, obtaining M rows, and arranging them sequentially to construct a one-dimensional vector. When mode x belongs to the [N,-1] mode (N≦-14), a one-dimensional vector can be constructed for the (66-x) mode, which is symmetric to mode x, following a row-first order, and the same NSPT kernel as for mode x can be applied to this one-dimensional vector. For modes 0 and 1, a column-first order or row-first order can be applied, and for modes 34, a column-first order or row-first order can also be applied. Furthermore, for intra-prediction modes belonging to the [2,33] mode, a row-first order can be applied, and for modes symmetric to that intra-prediction mode, a column-first order can be applied. For intra-prediction modes belonging to the [N,-1] mode, a row-first order can be applied, and for modes symmetric to it, a column-first order can be applied.

[0225] In the case of non-square blocks, in addition to the symmetry between intra-predictive modes, symmetry between block shapes may also be considered. A non-square block with widths and heights of M and N, respectively, can be considered to have a symmetric relationship with a non-square block with widths and heights of N and M, respectively. For example, in the [2,66] mode, symmetry may exist between the x-th mode of an MxN block and the (68-x)-th mode of an NxM block. Similarly, if the x-th mode of an MxN block belongs to the [N,-1] mode (N ≤ -14), symmetry may exist between the x-th mode of an MxN block and the (66-x)-th mode of an NxM block.

[0226] The method for constructing a one-dimensional vector from an input data block is as described above. That is, if column-major ordering is applied to mode x, row-major ordering may be applied to the mode symmetric to it. Or, if row-major ordering is applied to mode x, column-major ordering may be applied to the mode symmetric to it. Specifically, when column-major ordering is applied to mode x, input data can be read column by column from the input data MxN block to obtain M columns, and these can be arranged sequentially to construct a one-dimensional vector. Here, each column may have a length of N. For modes symmetric to mode x, input data can be read row by row from the input data MxN block to obtain N rows, and these can be arranged sequentially to construct a one-dimensional vector. Here, each row may have a length of M. Or, when row-major ordering is applied to mode x, input data can be read row by row from the input data MxN block to obtain N rows, and these can be arranged sequentially to construct a one-dimensional vector. Here, each row may have a length of M. For modes symmetric to mode x, we can read the input data column by column from the MxN block of input data to obtain M columns, and then sequentially arrange these to construct a one-dimensional vector. Here, each column may have a length of N.

[0227] If the current block is an MxN block having mode x, and the aforementioned symmetry is used with respect to the current block, the NSPT set and / or NSPT kernel of the current block may be determined based on at least one of the following: an intra-prediction mode symmetric to mode x, or an NxM block size symmetric to the MxN block size. Here, the NSPT kernel may be set as an NSPT kernel for an NxM block rather than an NSPT kernel for an MxN block. That is, when symmetry is used with respect to the current block, the NSPT set and / or NSPT kernel for a block symmetric to the current block may be used identically. As described above, a one-dimensional vector can be constructed from input data blocks according to a predetermined priority order, and this may correspond to the input of the NSPT kernel.

[0228] Furthermore, the use of the symmetry may be restricted to cases where the value of the current block's intra-prediction mode is greater than 34. That is, when the value of the current block's intra-prediction mode is greater than 34, a transpose operation may be applied when constructing a one-dimensional vector from the input data block, and an NSPT set or NSPT kernel corresponding to a mode and / or block form that is symmetric to the current block may be used. Specifically, the symmetry may not be used for the current block when the current block's intra-prediction mode belongs to the [N,-1] mode and the [2,34] mode. On the other hand, the symmetry may be used for the current block when the current block's intra-prediction mode belongs to the [35,66] mode and the [67,66-N] mode. Here, N may be an integer less than or equal to -14.

[0229] The induction of NSPT sets or NSPT kernels based on the aforementioned symmetry may now be performed adaptively based on the size of the block. For example, NSPT sets or NSPT kernels may be induced based on symmetry for 4x4 and 8x8 blocks, but not for 4x8 and 8x4 blocks.

[0230] The number of available NSPT sets may differ depending on whether or not the aforementioned symmetry is utilized. For example, when the symmetry is utilized, the number of available NSPT sets may be 35, while when the symmetry is not utilized, the number of available NSPT sets may be 67.

[0231] Table 9 below shows an example where NSPT sets are determined using symmetry, illustrating the mapping relationship between intra-prediction modes and NSPT sets when there are 35 available NSPT sets.

[0232] [Table 9]

[0233] Referring to Table 9, if the value (X) of the current block's intra-prediction mode is less than 0, the current block's NSPT set may be determined to be the NSPT set with NSPT set index 2 out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than or equal to 0 and less than or equal to 34, the current block's NSPT set may be determined to be the NSPT set with NSPT set index X out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than or equal to 35 and less than or equal to 66, the current block's NSPT set may be determined to be the NSPT set with NSPT set index (68-X) out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than or equal to 35 and less than or equal to 66, the current block's NSPT set may be the same as the NSPT set corresponding to the value (68-X) of the mode symmetric to the current block's intra-prediction mode. Similarly, if the value (X) of the current block's intra-prediction mode is greater than 66, the current block's NSPT set may be determined to be an NSPT set with 2 NSPT set indices out of 35 NSPT sets. If the value (X) of the current block's intra-prediction mode is greater than 66, the current block's NSPT set may be identical to the NSPT set corresponding to the mode symmetric to the current block's intra-prediction mode.

[0234] Table 10 below shows an example where the NSPT set is determined without using symmetry, illustrating the mapping relationship between the intra-prediction mode and the NSPT set when there are 67 available NSPT sets.

[0235] [Table 10]

[0236] Referring to Table 10, if the value of the current block's intra-prediction mode (X) is less than 0, the current block's NSPT set may be determined to be an NSPT set with 2 NSPT set indices out of 67 NSPT sets. If the value of the current block's intra-prediction mode (X) is greater than or equal to 0 and less than or equal to 66, the current block's NSPT set may be determined to be an NSPT set with X NSPT set indices out of 67 NSPT sets. If the value of the current block's intra-prediction mode (X) is greater than 66, the current block's NSPT set may be determined to be an NSPT set with 66 NSPT set indices out of 67 NSPT sets.

[0237] By utilizing the aforementioned symmetry, it is possible to save the memory size required to store the transformation kernel while maintaining the performance achieved by applying the transformation. For example, by using 35 NSPT sets instead of 67 NSPT sets by utilizing symmetry, the memory size required to store the NSPT kernel can be significantly reduced.

[0238] The number of available NSPT sets and / or the number of NSPT kernel candidates belonging to an NSPT set may differ depending on the block size. For example, there may be 35 available NSPT sets for a 4x4 block, 19 available NSPT sets for a 4x8 and 8x4 block, and 10 available NSPT sets for an 8x8 block. An NSPT set for a 4x4 block may consist of 3 NSPT kernel candidates, an NSPT set for a 4x8 and 8x4 block may consist of 3 or 2 NSPT kernel candidates, and an NSPT set for an 8x8 block may consist of 1 NSPT kernel candidate.

[0239] The size of the translation kernel may increase as the block size increases. Therefore, by reducing the number of available NSPT sets and / or the number of NSPT kernel candidates belonging to those sets, the memory size required to store the translation kernel can be saved. In addition, as the block size increases, the residual signal characteristics within the block tend to become more generalized. Therefore, reducing the number of available NSPT sets and / or the number of NSPT kernel candidates belonging to those sets helps maintain compression efficiency while reducing the realization complexity, reflecting these statistical characteristics.

[0240] An NSPT kernel may be constructed with 8 bits of precision. The coefficients within the NSPT kernel may be greater than or equal to -128 and less than or equal to -127. If the precision is increased beyond 8 bits, the result obtained from matrix multiplication can be shifted to the right by the increased precision. For example, if the value obtained after matrix multiplication based on an NSPT kernel with 8 bits of precision is shifted to the right by S bits and stored in a buffer, if the kernel coefficients are constructed with N bits of precision, then it can be shifted to the right by (S + (N-8)) bits and stored in the buffer.

[0241] When configuring the NSPT kernel with 8-bit precision, it is possible to prevent an excessive increase in internal precision within the code / decoder performing the conversion, minimizing the decline in compression efficiency while reducing the complexity of implementation in terms of memory requirements and computational complexity.

[0242] When applying an inverse NSPT to a current block of size MxN, the size of the NSPT kernel (or NSPT matrix) can be expressed as MNxr, where MN represents the product of the width and height of the current block, which can represent the output length of the NSPT or the number of residual samples generated by the NSPT. r represents the input length of the NSPT or the number of (inversely quantized) transformation coefficients to which the NSPT is applied. r can be an integer greater than or equal to 0 and less than or equal to MN. The following is an example for an MNxr NSPT matrix with a block size.

[0243] The NSPT matrix for a 4x4 block may be a 16x16 matrix. The NSPT matrices for 4x8 and 8x4 blocks may be a 32x20, 32x16, 32x24, 32x28, or 32x32 matrix. The NSPT matrix for an 8x8 block may be a 64x16, 64x24, 64x32, 64x40, 64x48, 64x56, or 64x64 matrix. The NSPT matrices for 4x16 and 16x4 blocks may be a 64x16, 64x24, 64x32, 64x40, 64x48, 64x56, or 64x64 matrix. The NSPT matrices for 8x16 and 16x8 blocks may consist of 128x96, 128x64, 128x48, or 128x32 matrices. The NSPT matrices for 16x16 blocks may consist of 256x128, 256x96, or 256x64 matrices. The NSPT matrices for 16x32 and 32x16 blocks may consist of 512x256 or 512x128 matrices. The NSPT matrices for 32x32 blocks may consist of 1024x512, 1024x256, or 1024x128 matrices.

[0244] Alternatively, a 16x16 matrix may be applied to 4xN blocks and Nx4 blocks, where N may be an integer greater than or equal to 4. A 64x16 matrix may be applied to an 8x8 block. A 64x32 matrix may be applied to 8xN blocks and Nx8 blocks, where N may be an integer greater than or equal to 16. A 96x32 matrix may be applied to 16xN blocks and Nx16 blocks, where N may be an integer greater than or equal to 16.

[0245] Alternatively, the value of r in the NSPT matrix of MNxr can be determined by predetermined criteria. These criteria may be: (1) the sum of the computational complexity for the linear transformation and the computational complexity for the quadratic transformation is below a certain level; and (2) the number of multiplications per sample required for the NSPT operation is below a certain number.

[0246] When performing a separable linear transformation on an MxN block using matrix multiplication, based on the inverse transformation, (M+N) multiplications are required per sample to perform the linear transformation. Also, when applying LFNST to a fixed ROI (Region-Of-Interest) region, if we assume that the inverse LFNST matrix is ​​a PxQ matrix, then (P*Q) / (M*N) multiplications are required per sample. Here, a PxQ matrix can be defined as a matrix with P columns and Q rows.

[0247] When applying NSPT to an MxN block instead of applying a DCT-2 transformation (or a separation transformation such as KLT) and LFNST, the value of r such that the number of multiplications per sample for the case where NSPT is applied is smaller than or equal to the number of multiplications per sample for the case where DCT-2 transformation and LFNST are applied may be determined as follows:

[0248] [Formula 6]

number

[0249] If the above equation 6 is satisfied and the value of r is set to its maximum value (i.e., r = M + N + (P*Q) / (M*N)), the value of r in the block-size NSPT matrix may be set as follows. If the value of (P*Q) / (M*N) in equation 6 is not an integer, an integer close to the value of (P*Q) / (M*N) may be used. For example, the floor operation may be applied to the value of (P*Q) / (M*N). In this case, r may be set to (M + N + floor((P*Q) / (M*N))). Here, floor(x) can mean the largest integer not exceeding x. Alternatively, the round operation may be applied to the value of (P*Q) / (M*N). In this case, r may be set to (M + N + round((P*Q) / (M*N))). Here, round(x) can mean the value of x rounded. Alternatively, the ceil operation may be applied to the value of (P*Q) / (M*N). In this case, r may be set to (M+N+ceil((P*Q) / (M*N))). Here, ceil(x) can mean the smallest integer greater than or equal to x. If the floor operation is applied, the inequality in equation 6 above may be satisfied. However, if the round operation or the ceil operation is applied, the inequality in equation 6 may not be satisfied.

[0250] For an NSPT on a 4x4 block, the maximum value of r is 24. However, the value of r must be less than or equal to 16, and therefore, the value of r may be set to 16.

[0251] For NSPTs on 4x8 and 8x4 blocks, the maximum value of r is 20. The value of r may be set to 20.

[0252] For NSPT on an 8x8 block, the maximum value of r is 32. The value of r can be set to 32.

[0253] For NSPTs on 4x16 and 16x4 blocks, the maximum value of r is 24. The value of r may be set to 24.

[0254] For the NSPT for 8x16 blocks and 16x8 blocks, the maximum value of r is 40. The value of r may be set to 40.

[0255] For the NSPT for 16x16 blocks, the maximum value of r is 44. The value of r may be set to 44.

[0256] For the NSPT for 16x32 blocks and 32x16 blocks, the maximum value of r is 54. The value of r may be set to 54.

[0257] For the NSPT for 32x32 blocks, the maximum value of r is 67. The value of r may be set to 67.

[0258] For the NSPT for 4x32 blocks and 32x4 blocks, the maximum value of r is 38. The value of r may be set to 38. Or, the value of r may be set to 20.

[0259] For the NSPT for 8x32 blocks and 32x8 blocks, the maximum value of r is 48. The value of r may be set to 48. Or, the value of r may be set to 24.

[0260] In the NSPT matrices for the above-mentioned block sizes, the value of r may not be a multiple of 4. For ease of implementation, it would be advantageous for the value of r to be a multiple of 4. For example, when implementing parallel processing using SIMD (Single Instruction Multiple Data) instructions, etc., when processing the inner products for 4 transform basis vectors simultaneously in the process of applying the forward NSPT (that is, when generating 4 transform coefficients simultaneously), it would be advantageous to set the value of r to be a multiple of 4.

[0261] For example, for NSPTs for 16x32 and 32x16 blocks, the value of r may be set to 52 or 56 instead of 54. For NSPTs for 32x32 blocks, the value of r may be set to 64 or 68 instead of 67. For NSPTs for 4x32 and 32x4 blocks, the value of r may be set to 36 or 40 instead of 38.

[0262] More generally, the value of r may be set to be a multiple of K, where K may be an integer greater than or equal to 1. As an example, the value of r may be set to a multiple of K that satisfies the following inequality, equation 7.

[0263] [Equation 7]

number

[0264] In formula 7 above, func() can be floor, round, or ceil as previously mentioned.

[0265] The value of r according to the predetermined criteria mentioned above is for cases where zero-out is not considered. That is, when forward LFNST is applied, the linearly transformed conversion coefficients of the remaining region outside the region to which LFNST is applied are zeroed out, so the actual computational cost required to apply DCT-2 and LFNST may be less than the aforementioned computational cost. Therefore, when zero-out is considered, the value of r may be set to a value smaller than the value of r according to the predetermined criteria.

[0266] Since zero-out is not performed for 4x4 blocks, the value of r can be set to a value less than or equal to 16.

[0267] For a 4x8 block, a zero-out may be performed on the remaining region excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 8 (((4x4x8)+(4x8x4)) / (4x8)=8), and the number of multiplications per sample required in LFNST is also 8 ((16x16) / 32=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 16, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 16.

[0268] For an 8x4 block, a zero-out may be performed on the remaining region excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 6 ((4x8x4)+(4x4x4) / (8x4)=6), and the number of multiplications per sample required in LFNST is 8 ((16x16) / 32=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 14, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 14.

[0269] For 8x8 blocks, zero-out is not required for the delimited linear transformation; in this case, the value of r may be set to a value less than or equal to 32.

[0270] For an 8x16 block, a zero-out may be performed on the remaining region excluding the top-left 8x8 block, based on the forward transformation. For the top-left 8x8 block, a 64x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 16 ((8x8x16)+(8x16x8) / (8x16)=16), and the number of multiplications per sample required in LFNST is also 16 ((64x32) / 128=16). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 32, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. Since the same computational complexity is required when applying the inverse delimited linear transformation and LFNST, the value of r can be set to a value less than or equal to 32.

[0271] For a 16x8 block, a zero-out may be performed on the remaining region excluding the top-left 8x8 block, based on the forward transformation, and a 64x32 matrix, which is the forward LFNST matrix, may be applied to the top-left 8x8 block. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 12 ((8x16x8)+(8x8x8) / (16x8)=12), and the number of multiplications per sample required in LFNST is 16 ((64x32) / 128=16). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 28, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. Since the same computational complexity is required when applying the inverse delimited linear transformation and LFNST, the value of r can be set to a value less than or equal to 28.

[0272] For a 16x16 block, a zero-out may be performed on the remaining region excluding the top-left 12x12 block, based on the forward transformation. For the top-left 12x12 block, a 96x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 21 ((12x16x16)+(12x16x12) / (16x16)=21), and the number of multiplications per sample required in LFNST is 12 ((96x32) / 256=12). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 33, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. Since the same computational complexity is required when applying the inverse delimited linear transformation and LFNST, the value of r can be set to a value less than or equal to 33.

[0273] For a 4x16 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 8 (((4x4x16)+(4x16x4)) / (4x16)=8), and the number of multiplications per sample required in LFNST is 4 ((16x16) / 64=4). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 12, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 12.

[0274] For a 16x4 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 5 (((4x16x4)+(4x4x4)) / (16x4)=5), and the number of multiplications per sample required in LFNST is 4 ((16x16) / 64=4). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 9, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 9.

[0275] For a 4x32 block, a zero-out may be performed on the remaining region, excluding the top-left 4x4 block, based on the forward transformation. For the top-left 4x4 block, a 16x16 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required in the forward separable linear transformation is 8 (((4x4x32)+(4x32x4)) / (4x32)=8), and the number of multiplications per sample required in LFNST is 2 ((16x16) / 128=2). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 10, which is the sum of the number of multiplications per sample in the separable linear transformation and the number of multiplications per sample in LFNST. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 10.

[0276] For a 32x4 block, zero-out may be performed on the remaining area except for the upper-left 4x4 block based on the forward transform, and a 16x16 matrix, which is the forward LFNST matrix, may be applied to the upper-left 4x4 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 4.5 (((4x32x4)+(4x4x4)) / (32x4)=4.5), and the number of multiplications per sample required in LFNST is 2 ((16x16) / 128=2). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 6.5. Since the same amount of computation is required when applying the inverse separable linear transform and LFNST, the value of r may be set to a value less than or equal to 6.5. Here, since the value of r is a value constituting the matrix dimension, it may be set to an integer of 6 or 7 instead of 6.5.

[0277] For an 8x32 block, zero-out may be performed on the remaining area except for the upper-left 8x8 block based on the forward transform, and a 64x32 matrix, which is the forward LFNST matrix, may be applied to the upper-left 8x8 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 16 (((8x8x32)+(8x32x8)) / (8x32)=16), and the number of multiplications per sample required in LFNST is 8 ((64x32) / 256=8). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 24. Since the same amount of computation is required when applying the inverse separable linear transform and LFNST, the value of r may be set to a value less than or equal to 24.

[0278] For a 32x8 block, a zero-out may be performed on the remaining region, excluding the top-left 8x8 block, based on the forward transformation. For the top-left 8x8 block, a 64x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 10 (((8x32x8)+(8x8x8)) / (32x8)=10), and the number of multiplications per sample required for LFNST is 8 ((64x32) / 256=8). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 18. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 18.

[0279] For a 16x32 block, a zero-out may be performed on the remaining region, excluding the top-left 12x12 block, based on the forward transformation. For the top-left 12x12 block, a 96x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 21 (((12x16x32)+(12x32x12)) / (16x32)=21), and the number of multiplications per sample required for LFNST is 6 ((96x32) / 512=6). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 27. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 27.

[0280] For a 32x16 block, a zero-out may be performed on the remaining region, excluding the top-left 12x12 block, based on the forward transformation. For the top-left 12x12 block, a 96x32 matrix, which is the forward LFNST matrix, may be applied. When such a zero-out is performed, the number of multiplications per sample required for the forward separable linear transformation is 13.5 (((12x32x12)+(12x16x12)) / (32x16)=13.5), and the number of multiplications per sample required for LFNST is 6 ((96x32) / 512=6). Therefore, when replacing the separable linear transformation and LFNST with NSPT, the value of r may be set to a value less than or equal to 19.5. The same computational complexity is required when applying the reverse separable linear transformation and LFNST, so the value of r may be set to a value less than or equal to 19.5. Here, the value of r is a value that constitutes the matrix dimension, so it may be set to an integer of 19 or 20 instead of 19.5.

[0281] In the aforementioned block-size-specific NSPT matrices, there may be cases where the value of r is not a multiple of K. Here, K may be 4. In this case, for ease of implementation, the value of r can be set to be a multiple of K. The value of r that was set in advance by the method described above is r prev In this case, the values ​​of r that are multiples of K may be set as follows:

[0282] [Equation 8]

number

[0283] In formula 8 above, func() can be floor, round, or ceil as previously mentioned.

[0284] As mentioned above, the value of r in the NSPT matrix for an MxN block may be different from the value of r in the NSPT matrix for an NxM block. For example, the inverse NSPT matrix for a 4x8 block may be a 32x16 matrix, and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix. In this case, the NSPT matrix may be determined by utilizing the symmetry between the MxN block and the NxM block.

[0285] Assume the current block is an MxN block having mode x. When applying the aforementioned symmetry to the current block, instead of applying the NSPT matrix corresponding to mode x or the MxN block size, an NSPT matrix corresponding to at least one of the modes symmetric to mode x or the NxM block size symmetric to the MxN block size may be applied. In this case, the NSPT matrix corresponding to the NxM block size may be applied directly to the current block. Alternatively, the NSPT matrix corresponding to the NxM block size may be applied, but the value of r may be the value of r in the NSPT matrix corresponding to the MxN block size.

[0286] For example, the inverse NSPT matrix for a 4x8 block is a 32x16 matrix (i.e., the value of r in the NSPT matrix is ​​16), and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix (i.e., the value of r in the NSPT matrix is ​​14). When the current block is an 8x4 block having mode x, an NSPT matrix for at least one of the modes symmetric to mode x or a 4x8 block symmetric to the 8x4 block may be applied to the current block. In this case, the 32x16 matrix, which is the inverse NSPT matrix for the 4x8 block, may be used as is, or a 32x14 matrix with the value of r in the inverse NSPT matrix for the 8x4 block may be used. Here, the 32x14 matrix may be derived by sampling the 14 rows from the left of the 32x16 matrix. In this way, when applying the 32x14 matrix to the current block having a block size of 8x4, the predetermined criteria mentioned above will be satisfied.

[0287] Conversely, if the current block is a 4x8 block having mode x, then an NSPT matrix for at least one mode symmetric to mode x or an 8x4 block symmetric to the 4x8 block may be applied to the current block. In this case, a 32x14 matrix for the 8x4 block may be applied to the current block instead of a 32x16 matrix for the 4x8 block. This allows NSPT to be performed on the 4x8 block using fewer multiplications than the allowed number of multiplications.

[0288] For NSPTs for MxN and NxM blocks, if the values ​​of r that satisfy the aforementioned predetermined conditions are r1 and r2, respectively, the inverse NSPT matrices for MxN and NxM blocks may be set to MNxmax(r1,r2). Here, max(r1,r2) can be interpreted as selecting the larger or equal value of r1 and r2.

[0289] For example, the inverse NSPT matrix for a 4x8 block is a 32x16 matrix (i.e., the value of r in the NSPT matrix is ​​16), and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix (i.e., the value of r in the NSPT matrix is ​​14). If the current block is a 4x8 block having mode x, then an NSPT matrix for at least one of the modes symmetric to mode x or an 8x4 block symmetric to the 4x8 block may be applied to the current block. In this case, the 32x16 matrix, which is the inverse NSPT matrix for the 8x4 block, may be used. If the inverse NSPT matrix is ​​not constructed as MNxmax(r1,r2), then the 32x14 matrix will be used as the NSPT matrix for the 8x4 block. However, if the NSPT matrix for the 4x8 block and the NSPT matrix for the 8x4 block are constructed as 32xmax(16,14) matrices, then the 32x16 matrix may be fully applied.

[0290] Conversely, if the current block is an 8x4 block having mode x, an NSPT matrix for at least one of the modes symmetric to mode x or a 4x8 block symmetric to the 8x4 block may be applied to the current block. In this case, a 32x16 matrix, which is the inverse NSPT matrix for a 4x8 block, may be used, or a 32x14 matrix may be used. Here, the 32x14 matrix may be derived by sampling the 14 rows from the left of the 32x16 matrix.

[0291] As mentioned above, when constructing an NSPT matrix, it is possible to apply a transformation composed of the maximum number of transformation basis vectors while satisfying the predetermined conditions, thereby maximizing coding performance.

[0292] In the embodiments described above, the value of r may be set to be a multiple of 16. For example, in the case of reverse NSPT for 4x8 and 8x4 blocks, a 32x16 matrix may be applied instead of a 32x20 matrix. The transformation coefficients of the transformation block may be encoded in units of a predetermined coefficient group (Coefficient Group, CG). Here, CG may be defined as a group of 16 transformation coefficients, and for example, CG may be subblocks of sizes such as 4x4, 2x8, and 8x2. There may be cases where no non-zero transformation coefficients exist within a single CG, in which case the encoding process of the transformation coefficients may be skipped for that CG. Therefore, setting the value of r to a multiple of 16 has the advantage of reducing the complexity of implementation.

[0293] A forward NSPT can be applied to a resistive sample of an MxN block to induce transformation coefficients. In this case, the number of induced transformation coefficients may be less than or equal to the value of (M*N) by zero-out. That is, the forward NSPT matrix may be defined as an rx(M*N) matrix, where r represents the output length of the NSPT or the number of transformation coefficients induced by the NSPT, and (M*N) represents the input length of the NSPT or the number of resistive samples to which the NSPT is applied.

[0294] The derived conversion coefficients may be arranged within an MxN block according to a predetermined scan order, and any region where the conversion coefficients are not filled may be filled with zeros (i.e., zero-out). Therefore, if, during the process of scanning the conversion coefficients with the decoding device, a non-zero conversion coefficient is found in a region that would have been filled with zeros if NSPT had been applied (or if the scan position of the last effective coefficient in the MxN block is greater than or equal to r), it is considered that NSPT has not been applied to that MxN block, and the NSPT index does not need to be signaled. The upper left coefficient in the MxN block (i.e., the DC component coefficient) may be assigned an index indicating the scan position of zero, and the remaining coefficients in the MxN block may be assigned indices increasing by 1 according to a predetermined order.

[0295] One or more values ​​of r may be defined for the block sizes to which NSPT can be applied. For example, one or more values ​​of r may be defined for each of the block sizes to which NSPT can be applied. Alternatively, one value of r may be defined for each block size to which NSPT can be applied, and the value of r for one of the block sizes to which NSPT can be applied may be different from the value of r for any other size. Alternatively, one value of r may be defined for some of the block sizes to which NSPT can be applied, and two or more values ​​of r may be defined for the rest.

[0296] If multiple values ​​for r are available, an index that identifies any one of the multiple values ​​of r, or r itself, may be signaled. This index may be signaled using high-level syntax (HLS) such as VPS, SPS, PPS, PH, SH, or at the block level such as CTU, CU, TU. If the value of r belongs to a specific range, the bits that may include that range can be allocated and signaled. For example, if the value of r belongs to the range of 1 to 256, 8 bits can be specified as fixed-length and signaled.

[0297] The current block's conversion kernel may be determined based on any one of the aforementioned Examples 1 to 3. Alternatively, the current block's conversion kernel may be determined based on a combination of at least two of Examples 1 to 3, provided that the inventions described in Examples 1 to 3 do not conflict with each other.

[0298] A transformation index for the inverse transformation of the current block may be signaled. Here, the transformation index can identify one of the one or more transformation kernels (or transformation matrices) belonging to the transformation set. Here, the transformation index can mean an NSPT index that identifies one of the one or more NSPT kernels belonging to the NSPT set. Or, the transformation index can mean an LFNST index that identifies one of the one or more LFNST kernels belonging to the LFNST set.

[0299] Whether the aforementioned conversion index corresponds to an NSPT index may be determined based on whether the current block size is one of the block sizes belonging to the first group described above. This assumes that the block sizes to which NSPT can be applied and the block sizes to which LFNST can be applied are distinct from each other. In this case, if the current block size belongs to the first group, the conversion index signaled to the current block corresponds to an NSPT index, and an NSPT kernel may be determined from the NSPT set based on that conversion index. On the other hand, if the current block size does not belong to the first group, the conversion index signaled to the current block corresponds to an LFNST index, and an LFNST kernel may be determined from the LFNST set based on that conversion index. If the current block size does not belong to the first group, it may mean that the current block size belongs to the second group described above. Alternatively, if the current block size does not belong to the first group, it may mean that the current block size corresponds to a block size among the block sizes belonging to the second group to which LFNST can be applied. Thus, NSPT indexes and LFNST indexes do not need to be composed of separate syntaxes, but rather can be composed of a single, integrated syntax.

[0300] As an example, let's assume that the block sizes to which NSPT can be applied that belong to the first group are 4x4, 4x8, 8x4, and 8x8. For the block sizes belonging to the first group, NSPT may be applied instead of LFNST. Specifically, NSPT may be applied instead of the combination of a separate linear transform (e.g., DCT-2, separate KLT) and LFNST. An NSPT index may be signaled for the four block sizes belonging to the first group, and an LFNST index may be signaled for the remaining block sizes (to which LFNST is permitted).

[0301] Thus, signaling with a single syntax that integrates NSPT and LFNST indices reduces the amount of information encoded. Furthermore, the complexity of implementation can be reduced by applying at least one of the following—binary encoding, CABAC context, or initial value—to both NSPT and LFNST indices for entropy coding.

[0302] Alternatively, the NSPT index and LFNST index can be signaled as separate syntaxes. In this case, the complexity of implementation may increase to some extent, but compression performance can be improved by performing optimized entropy coding for each index.

[0303] If the number of LFNST kernel candidates belonging to the LFNST set is the same as the number of NSPT kernel candidates belonging to the NSPT set, the same binarization may be applied to the LFNST index and the NSPT index. The same CABAC context (or CABAC context increment) may be assigned to the bins of the LFNST index and the NSPT index.

[0304] Different binary evolutions and / or CABAC contexts may be used for LFNST and NSPT indexes. Different CABAC initial values ​​may be assigned to LFNST and NSPT indexes. For example, one of the LFNST and NSPT indexes may be binary-evolved based on fixed-length binarization, and the other may be binary-evolved based on truncated unary binarization. Even if the binary evolutions of the LFNST and NSPT indexes are the same, different CABAC contexts and / or CABAC initial values ​​may be assigned to them. Different binary evolutions and / or CABAC contexts may be used for LFNST and NSPT indexes when the number of LFNST kernel candidates belonging to the LFNST set and the number of NSPT kernel candidates belonging to the NSPT set are different from each other.

[0305] The number of NSPT kernel candidates belonging to an NSPT set may be set to differ from one another for each block size. Alternatively, the block sizes belonging to the first group may be divided into multiple subgroups. In this case, the number of NSPT kernel candidates belonging to an NSPT set may be set to differ from one another for each of the multiple subgroups. Here, at least one of the multiple subgroups may contain multiple different block sizes.

[0306] Depending on the number of NSPT kernel candidates belonging to the NSPT set, the binary evolution applied to the NSPT index may differ from one another.

[0307] For example, if there are three NSPT kernel candidates in an NSPT set for a given block size, the NSPT index may have a value between 0 and 3. If the NSPT index value is 0, this can indicate that NSPT is not currently applied to the block. If the NSPT index value is not 0, this can indicate the NSPT kernel candidate corresponding to that index among the three NSPT kernel candidates. A bin can be assigned to distinguish between cases where NSPT is applied and cases where it is not. If the value of the bin is 0, this may correspond to the case where the NSPT index value is 0. On the other hand, if the value of the bin is 1, this may correspond to the case where the NSPT index value is 1, 2, or 3. In this case, truncated unative binary evolution may be applied to distinguish between the three NSPT kernel candidates. That is, two bins can be assigned to distinguish between 0, 10, 11 and the three NSPT kernel candidates.

[0308] When there are two NSPT kernel candidates in the NSPT set for a given block size, the NSPT index may have a value between 0 and 2. A value of 0 in the NSPT index indicates that NSPT is not currently applied to the block. A non-zero value in the NSPT index indicates the NSPT kernel candidate corresponding to that index among the two candidates. A bin can be assigned to distinguish between cases where NSPT is applied and cases where it is not. A bin can be assigned to indicate one of the two NSPT kernel candidates, thus distinguishing between the two NSPT kernel candidates.

[0309] When there is only one candidate NSPT kernel in the NSPT set for a given block size, the NSPT index may have a value of either 0 or 1. A value of 0 in the NSPT index indicates that NSPT is not currently applied to the block. A value of 1 indicates that there is one candidate NSPT kernel. In such cases, the presence or absence of NSPT application and the candidate NSPT kernel can be determined from a single bin.

[0310] The inverse transform of the current block may be a separated linear transform and / or an LFNST-based inverse transform. That is, a reverse LFNST can be applied to all or some of the (inversely quantized) transformation coefficients of the current block, and then a reverse separated linear transform can be applied to the transformation coefficients induced by the LFNST to induce a residual sample. As an example, a reverse LFNST may be applied to the (inversely quantized) transformation coefficients belonging to a portion of the current block. Here, the portion of the block means the region to which a forward LFNST has been applied, and will be hereinafter referred to as the ROI (region of interest) region. The transformation coefficients induced by the LFNST may be arranged in the ROI region according to a predetermined scan order. The predetermined scan order may be row-first order or column-first order. A reverse separated linear transform may be applied to the transformation coefficients induced by the LFNST and the transformation coefficients belonging to the remaining region of the current block excluding the ROI region. Alternatively, if zero-out is performed on the remaining region of the current block excluding the ROI region during the forward transformation process (i.e., the transformation coefficients in the remaining region are set to 0), a reverse separation linear transformation may be applied to the transformation coefficients induced by LFNST. The size of the ROI region may be determined based on at least one of the width or height of the current block. Here, the size of the ROI region may mean at least one of the width or height of the ROI region, or it may mean the number of sample positions belonging to the ROI region.

[0311] The following describes how to signal the transformation index for the inverse transformation of the current block. Here, the inverse transformation can mean the inseparable transformation in the reverse direction. The inseparable transformation can mean the LFNST or NSPT mentioned above, and the transformation index can mean the LFNST index or the NSPT index.

[0312] Currently, when a block is encoded as a single tree, an inseparable transform may be applied to the lumar component of the current block, but not to the chroma component. In this case, the transform index for the current block may be signaled based on at least one of the first condition regarding the lumar component or the second condition regarding the chroma component. For example, the transform index for the current block may be signaled if both the first condition regarding the lumar component and the second condition regarding the chroma component are met, and not signaled otherwise (i.e., if neither the first condition regarding the lumar component nor the second condition regarding the chroma component is met). Alternatively, the transform index for the current block may be signaled if the first condition regarding the lumar component is met, without checking whether the second condition regarding the chroma component is met, and not signaled otherwise. If the transform index is not signaled, it may be set so that an inseparable transform is not applied to the current block, and for example, the transform index for the current block may be induced to 0.

[0313] The first condition relating to the rumor component in this disclosure can be interpreted as meaning that no non-zero transformation coefficients exist in a predetermined region within the rumor component block of the current block. Here, the predetermined region within the rumor component block may be defined as the remaining region in the rumor component block other than the first region, which will be referred to as the second region to distinguish it from the first region. The first region may be defined as a region consisting of the same number of samples (or sample positions) as the number of transformation coefficients input to the inverse inseparable transformation (or the number of transformation coefficients output by the forward inseparable transformation). The first region may include the upper leftmost sample position of the rumor component block. The first region may include at least one non-zero transformation coefficient. The first region may be the region to which the last significant coefficient in the rumor component block belongs. The width and height of the first region may be less than or equal to the width and height of the rumor component block, respectively.

[0314] As previously mentioned, the number of transformation coefficients to which the aforementioned inverse non-separable transformation is applied may be determined based on the size of the current block. The sizes of the current block and the rumor component block may be the same. The size of the current block may be defined as a combination of width (W) and height (H), such as WxH. However, it is not limited to this, and the size of the current block may be defined as either the width or the height, the minimum / maximum value of the width and height, or the product of the width and height.

[0315] As an example, when an unseparated transformation is applied to an MxN block in an encoding device, r transformation coefficients smaller than or equal to (M*N) may be output. This means that the forward unseparated transformation has r output lengths. The r output transformation coefficients may be arranged sequentially within the MxN block according to a predetermined scan order, starting from the upper left sample position of the MxN block (i.e., the DC position). The region currently composed of r transformation coefficients within the block may correspond to the aforementioned first region. Subsequently, zero-out may be applied to the r-th and subsequent sample positions within the MxN block (i.e., the second region within the current block). The zero-out means assigning 0 to the sample positions within the second region. In this way, when an unseparated transformation is applied in an encoding device, there will be no non-zero transformation coefficients for the sample positions to which zero-out is applied.

[0316] In the process of deriving conversion coefficients based on residual information in the decoding device, if a non-separable conversion is applied in the encoding device, and a non-zero conversion coefficient is found from a sample position to which zero-out is applied (i.e., the r-th sample position or the second region), this means that a non-separable conversion has not been applied to the MxN block. In such a case, a conversion index for non-separable conversion does not need to be signaled for the MxN block. In this case, it may be set that a non-separable conversion is not applied to the MxN block, and as an example, the conversion index may be induced to 0.

[0317] The second condition relating to the chroma component in this disclosure can be interpreted as meaning that no non-zero transformation coefficients exist in a predetermined region within the chroma component block of the current block. Here, the predetermined region within the chroma component block may be defined as the remaining region in the chroma component block other than the first region, which will be referred to as the second region to distinguish it from the first region. The first region may be defined as a region consisting of the same number of samples (or sample positions) as the input length of the inverse non-separable transformation corresponding to the size of the chroma component block. The input length can be interpreted as the number of transformation coefficients input to the inverse non-separable transformation. The first region may include the upper-leftmost sample position of the chroma component block. The first region may include at least one non-zero transformation coefficient. The first region may be the region to which the last significant coefficient belongs in the chroma component block. The width and height of the first region may be less than or equal to the width and height of the chroma component block, respectively.

[0318] Although the inseparable transformation is not applied to the chroma component block, the input length of the inverse inseparable transformation may be determined based on the size of the chroma component block. Here, the input length of the inseparable transformation may be determined as the input length of the inseparable transformation determined based on the size of the chroma component block. Alternatively, the matrix size (or input / output length) of the inseparable transformation may be determined based on the size of the luma component block corresponding to the chroma component block, and half of the input length of the inseparable transformation may be determined as the input length of the inseparable transformation. The method for determining the inseparable transformation matrix based on the block size is as described above, and a detailed explanation is omitted here.

[0319] Specifically, although non-separable transformation is not applied to the chroma component, zero-out may be applied to the transformation coefficients belonging to a predetermined region within the chroma component block, just as would be the case if non-separable transformation were applied to the chroma component. That is, the encoding device can retain only the transformation coefficients belonging to a portion of the region within the chroma component block, and set the transformation coefficients belonging to the remaining region to 0. Here, the portion of the region can mean the region in which the transformation coefficients output by non-separable transformation are arranged if non-separable transformation is applied to the chroma component block.

[0320] As an example, suppose the data is encoded in a 4:2:0 color format, the current block tree type is single-tree, the lumens and chroma component blocks are 8x16 and 4x8 respectively, and both 8x16 and 4x82 are block sizes that allow for non-separated transformation. In this case, non-separated transformation is applied to the lumens component blocks, but not to the chroma component blocks. However, zero-out may be applied to the chroma component blocks in the same way as when non-separated transformation is applied.

[0321] Assuming that forward NSPT is performed on a 4x8 block based on a 20x32 NSPT matrix, only 20 transformation coefficients may be output by NSPT. These 20 transformation coefficients will be arranged sequentially from the top-left sample position in the 4x8 block according to a predetermined scan order, and zero-out will be applied to the remaining 12 sample positions. If zero-out is applied to the chroma component block in the same manner, the already outputted transformation coefficients can be left as they are for the chroma component block from the top-left sample position up to the 20th sample position according to the predetermined scan order, and 0 can be assigned to the 21st to 32nd sample positions. Furthermore, the chroma component block may consist of a Cb component block and a Cr component block, and zero-out may be applied to both component blocks in the same manner.

[0322] The size of the first region within the chroma component block described above may be determined based on the size of the chroma component block. The first region may be defined as a region consisting of r sample positions. Alternatively, the first region may be defined as a region consisting of min(threshold,r) sample positions. min(threshold,r) is a function that outputs the minimum value of threshold and r. The threshold is a value predefined for the encoding and decoding devices and may be an integer of 4, 8, 16, 32, 64 or more. The threshold may be variable based on the size of the chroma component block or the size of the chroma component block corresponding to the chroma component block (i.e., the size of the current block).

[0323] The aforementioned r may be determined based on the input length of the inverse inseparable transformation corresponding to the size of the chroma component block. For example, r may be the same as the input length of the inverse inseparable transformation corresponding to the size of the chroma component block. Here, the input length can mean the number of transformation coefficients to which the inverse inseparable transformation is applied. This means that the inverse inseparable transformation corresponding to the size of the chroma component block is a Pxr matrix. Alternatively, r may be determined based on the output length of the forward inseparable transformation corresponding to the size of the chroma component block. For example, r may be the same as the output length of the forward inseparable transformation corresponding to the size of the chroma component block. Here, the output length can mean the number of transformation coefficients output by the forward inseparable transformation. This means that the forward inseparable transformation corresponding to the size of the chroma component block is an rxP matrix. In the aforementioned inseparable transformation matrix, the value of P may be the product of the width and height of the chroma component block. Alternatively, the value of P may be the number of transformation coefficients (or residual samples) induced by the inverse inseparable transformation. The value of P may be the number of samples belonging to the region to which the forward inseparable transformation is applied within the chromatic component block. For example, when the inseparable transformation is NSPT or LFNST, the value of P may be the same as the product of the width and height of the chromatic component block. When the inseparable transformation is LFNST, the value of P may be the same as the number of transformation coefficients belonging to the region (ROI) to which the forward inseparable transformation is applied.

[0324] As mentioned above, when the non-separable transformation is NSPT, an NSPT matrix having a predetermined dimension may be determined / mapped depending on the block size to which the NSPT can be applied. For example, the forward NSPT matrices corresponding to 4x4 blocks, 4x8 blocks, 8x4 blocks, 8x8 blocks, 4x16 blocks, 16x4 blocks, 8x16 blocks, and 16x8 blocks may be 16x16 matrices, 20x32 matrices, 20x32 matrices, 32x64 matrices, 24x64 matrices, 24x64 matrices, 40x128 matrices, and 40x128 matrices, respectively. That is, when the chroma component block is a 4x4 block, the value of r may be 16. When the chroma component block is a 4x8 block or an 8x4 block, the value of r may be 20. When the chroma component block is an 8x8 block, the value of r may be 32. When the chroma component block is a 4x16 block or a 16x4 block, the value of r may be 24. When the chroma component block is an 8x16 block or a 16x8 block, the value of r may be 40.

[0325] As mentioned above, when the non-separable transformation is LFNST, an LFNST matrix having a predetermined dimension may be determined / mapped depending on the block size. For example, the forward LFNST matrix corresponding to a 4xN block and / or Nx4 block may be a 16x16 matrix. That is, when the chroma component block is a 4xN block or an Nx4 block, the value of r may be 16. Here, N may be an integer greater than or equal to 4. Alternatively, the forward LFNST matrix corresponding to an 8x8 block may be a 16x64 matrix. That is, when the chroma component block is an 8x8 block, the value of r may be 16. Alternatively, the forward LFNST matrix corresponding to an 8xN block and / or an Nx8 block may be a 32x64 matrix. That is, when the chroma component block is an 8xN block or an Nx8 block, the value of r may be 32. Here, N may be an integer greater than or equal to 16. The 16x64 matrix, which is the forward LFNST matrix corresponding to the aforementioned 8x8 block, can be obtained by sampling 16 rows from the top row of a 32x64 matrix, which is the forward LFNST matrix corresponding to an 8xN block or an Nx8 block. Alternatively, the forward LFNST matrix corresponding to a 16xN block and / or an Nx16 block may be a 32x96 matrix. That is, when the chroma component block is a 16xN block or an Nx16 block, the value of r may be 32. Here, N may be an integer greater than or equal to 16.

[0326] The predefined permissible conversion block sizes may be distinguished into a first group, which is the set of block sizes to which NSPT can be applied, and a second group, which is the set of block sizes to which NSPT cannot be applied, as described above.

[0327] For example, the first group may be defined to include at least one of 4x4, 4x8, 8x4, 8x8, 4x16, 16x4, 8x16, or 16x8, and the second group may include the remaining block sizes. In this case, NSPT may be applied to the block sizes belonging to the first group, and LFNST may be applied to all or part of the block sizes belonging to the second group. Alternatively, the first group may be defined to include at least one of 4x4, 4x8, 8x4, or 8x8, and the second group may include the remaining block sizes. In this case, NSPT may be applied to the block sizes belonging to the first group, and LFNST may be applied to all or part of the block sizes belonging to the second group. Alternatively, the first group may be defined to include at least one of 4x4, 4x8, 8x4, 8x8, 4x16, or 16x4, and the second group may include the remaining block sizes. In this case, NSPT may be applied to block sizes belonging to the first group, and LFNST may be applied to all or part of block sizes belonging to the second group.

[0328] On the other hand, LFNST may be applied to 4x4 blocks. This means that 4x4 blocks fall into the block size category to which both NSPT and LFNST can be applied. Alternatively, it means that 4x4 blocks fall into the block size category to which LFNST can be applied, but not to which NSPT can be applied. In other words, it means that the 4x4 block size is excluded from the first group.

[0329] Alternatively, LFNST may be applied to 4x4 and 8x8 blocks. This means that 4x4 and 8x8 blocks are block sizes to which both NSPT and LFNST can be applied. Or, it means that 4x4 and 8x8 blocks are block sizes to which LFNST can be applied, but not to which NSPT can be applied. In other words, it means that 4x4 and 8x8 block sizes are excluded from the first group.

[0330] Alternatively, LFNST may be applied to 8x8 blocks. This means that 8x8 blocks fall under the block size to which both NSPT and LFNST can be applied. Or, it means that 8x8 blocks fall under the block size to which LFNST can be applied, but not to which NSPT can be applied. In other words, the 8x8 block size is excluded from the first group.

[0331] The value of r for a chroma component block may be set by the method described above. The value of r may be set by the method described above regardless of whether or not an inseparable transformation (e.g., NSPT and / or LFNST) is applied to the chroma component block. Alternatively, the value of r may be set by the method described above when an inseparable transformation (e.g., NSPT and / or LFNST) is not applied to the chroma component block. Alternatively, the value of r may be set by the method described above when an inseparable transformation (e.g., NSPT and / or LFNST) is applied to the chroma component block.

[0332] Even if a non-separating transformation is not applied to a chroma component block, if the size of the chroma component block is such that NSPT or LFNST can be applied, the value of r may be set using the method described above. In this case, if the size of the chroma component block is such that NSPT can be applied, the value of r may be set based on the input length of the inverse NSPT matrix (or the output length of the forward NSPT matrix) corresponding to the block size. If the size of the chroma component block is such that LFNST can be applied, the value of r may be set based on the input length of the inverse LFNST matrix (or the output length of the forward LFNST matrix) corresponding to the block size. Below, the zero-out method for chroma component blocks will be described when the current block tree type is a single tree.

[0333] If the chroma component block is a 4xN block or an Nx4 block (where N is an integer of 4 or greater), a predetermined number of transformation coefficients in the chroma component block may be retained, and the remaining transformation coefficients may be set to 0. The transformation coefficients in the chroma component block may be derived by a forward separation linear transformation. Alternatively, the transformation coefficients in the chroma component block may be derived by at least one of the forward NSPT or LFNST. The predetermined number may be r or min(16,r). The retained transformation coefficients may be arranged within the chroma component block according to a predetermined scan order. 0 may be assigned to any remaining sample positions that are not filled by the retained transformation coefficients. During the decoding process, an inverse transformation may be applied to the r or min(16,r) transformation coefficients within the chroma component block.

[0334] If the chroma component block is an MxN block (where M and N are integers of 16 or greater), a predetermined number of transformation coefficients in the chroma component block may be retained, and the remaining transformation coefficients may be set to 0. The transformation coefficients in the chroma component block may be derived by a forward separation linear transformation. Alternatively, the transformation coefficients in the chroma component block may be derived by at least one of the forward NSPT or LFNST. The predetermined number may be r or min(256,r). The retained transformation coefficients may be arranged within the chroma component block according to a predetermined scan order. 0 may be assigned to any remaining sample positions that cannot be filled by the retained transformation coefficients. During the decoding process, an inverse transformation may be applied to the r or min(256,r) transformation coefficients in the chroma component block.

[0335] If the chroma component block is an 8xN block or an Nx8 block (where N is an integer of 8 or greater), a predetermined number of transformation coefficients in the chroma component block may be retained, and the remaining transformation coefficients may be set to 0. The transformation coefficients in the chroma component block may be derived by a forward separation linear transformation. Alternatively, the transformation coefficients in the chroma component block may be derived by at least one of the forward NSPT or LFNST. The predetermined number may be r or min(64,r). The retained transformation coefficients may be arranged within the chroma component block according to a predetermined scan order. 0 may be assigned to the remaining sample positions that cannot be filled by the retained transformation coefficients. During the decoding process, an inverse transformation may be applied to the r or min(64,r) transformation coefficients in the chroma component block.

[0336] The zero-out for the chromatic component block may be performed by NSPT or by LFNST.

[0337] Specifically, if the size of the chroma component block (or the size of the lumana component block corresponding to the chroma component block) is within the range of block sizes to which NSPT can be applied, the zero-out method using NSPT may be applied to the chroma component block. On the other hand, if the size of the chroma component block (or the size of the lumana component block corresponding to the chroma component block) is within the range of block sizes to which LFNST can be applied, the zero-out method using LFNST may be applied to the chroma component block.

[0338] Depending on the current block's tree type, it may be determined whether the NSPT zero-out method or the LFNST zero-out method is applied to the chroma component block by selectively using either the size of the lumana component block or the size of the chroma component block. For example, if the current block's tree type is a single tree, the determination can be based on the size of the lumana component block, and if the current block's tree type is a dual tree, the determination can be based on the size of the chroma component block.

[0339] If it is determined that both NSPT and LFNST can be applied to a chroma component block, zero-out may be applied based on either the output length of the NSPT or the output length of the LFNST. If both NSPT and LFNST can be applied to a chroma component block, zero-out may be applied based on the output length of the non-separated transformation with a predetermined priority. For example, NSPT may have a higher priority than LFNST. Alternatively, if both NSPT and LFNST can be applied to a chroma component block, zero-out may be applied based on the minimum or maximum value of the output length of the NSPT or the output length of the LFNST. For example, if both NSPT and LFNST are applied to a chroma component block, and the forward output length of the NSPT corresponding to the chroma component block is 20, and the forward output length of the LFNST corresponding to the chroma component block is 16, then only the transformation coefficients from the top-left sample position to the 16th sample position in the chroma component block may be retained, and the transformation coefficients for the remaining sample positions may be set to 0.

[0340] As mentioned above, currently a block is encoded as a single tree, and an inseparable transform is applied only to the rumor component, but not to the chroma component. However, if zero-out is applied, the first condition regarding the rumor component and the second condition regarding the chroma component may be checked, and if the first and second conditions are met, the transform index for the current block (especially the rumor component block) may be signaled.

[0341] Alternatively, if the current block's tree type is a single tree and the non-separable transformation is applied only to the lumar component, the aforementioned zero-out may not be applied to the chroma component. In this case, the second condition for the chroma component may not be checked, and only the first condition for the lumar component may be checked. If the first condition for the lumar component is met, the transformation index for the current block (especially the lumar component block) may be signaled. On the other hand, if the first condition for the lumar component is not met (i.e., if there are non-zero transformation coefficients in the second region within the lumar component block), the transformation index for the current block may not be signaled. In this case, the transformation index may be induced to 0.

[0342] Alternatively, if the current block's tree type is a single tree and an inseparable transformation is applied to the lumern and chroma components, both the first condition regarding the lumern component and the second condition regarding the chroma component described above can be checked. In this case, if both the first and second conditions are met, the transformation index for the current block may be signaled. If neither the first nor the second condition is met, the transformation index for the current block does not need to be signaled.

[0343] When the color format is 4:2:0, the current block's tree type is single-tree, and the lumens component block size is MxN, the chromens component block size may be (M / 2)x(N / 2). This assumes that ISP (Intra Sub-Partition) mode is not applied to the current block. When an unseparated transformation is applied to the MxN block for the lumens component, an unseparated transformation such as NSPT or LFNST may be applied to the chromens component, or neither NSPT nor LFNST may be applied. Specifically, this can be classified into the following cases:

[0344] 1) When LFNST is applied to MxN blocks for luma components

[0345] 1-a) Apply LFNST to the (M / 2)x(N / 2) transformation block for the chromatic component.

[0346] 1-b) Apply NSPT to the (M / 2)x(N / 2) transform block for the chroma component.

[0347] 1-c) Do not apply both NSPT and LFNST to the (M / 2)x(N / 2) transform block for the chroma component (in this case, apply a DCT-2 based horizontal / vertical linear transform to the chroma component).

[0348] 2) When NSPT is applied to the MxN transformation block for the luma component

[0349] 2-a) Apply LFNST to the (M / 2)x(N / 2) transform block for the chroma component.

[0350] 2-b) Apply NSPT to the (M / 2)x(N / 2) transform block for the chroma component.

[0351] 2-c) Do not apply both NSPT and LFNST to the (M / 2)x(N / 2) transform block for the chroma component (in this case, apply a DCT-2 based horizontal / vertical linear transform to the chroma component).

[0352] When both M and N are 4 or greater, NSPT and LFNST are applicable, and we assume that NSPT is applicable to 4x4, 4x8, 8x4, and 8x8 blocks. In this case, when the current block's tree type is a single tree, the following situations may occur:

[0353] If the lumern component block is a 16x8 block, the chroma component block may be an 8x4 block. The lumern component may be configured to have LFNST applied, and the chroma component to have NSPT applied. Alternatively, the lumern component may be configured to have LFNST applied, but neither NSPT nor LFNST may be applied to the chroma component. In this case, even if neither NSPT nor LFNST is applied to the chroma component, zero-out may be applied in the same manner as when NSPT or LFNST is applied, as described above. Also, in this example, since the chroma component block is an 8x4 block, zero-out using NSPT may be applied.

[0354] When the lumern component block is an 8x8 block, the chroma component block may be a 4x4 block. The system may be configured so that NSPT is applied to both the lumern component and the chroma component. Alternatively, the system may be configured so that NSPT is applied to the lumern component, but neither NSPT nor LFNST is applied to the chroma component. In this case, even if neither NSPT nor LFNST is applied to the chroma component, zero-out may be applied in the same manner as when NSPT or LFNST is applied, as described above. Also, in this example, since the chroma component block is a 4x4 block, zero-out using NSPT may be applied.

[0355] If the lumens component block is an 8x4 block, the chromens component block may be a 4x2 block. NSPT may be applied to the lumens component, while neither NSPT nor LFNST may be applied to the chromens component. In this case, zero-out by NSPT or LFNST may not be applied to the chromens component. Instead, the conversion coefficients from the top-left sample position in the chromens component block to the nth sample position according to a predetermined scan order may be left as they are, and zero-out may be applied to sample positions from the nth onward. Here, n is a value predefined identically for both the encoding and decoding devices, and may be an integer greater than or equal to 2. For example, n may be 4. The range from the top-left sample position to the fourth sample position is the region containing the top-left sample of the chromens component block, and may belong to a 2x2 region.

[0356] When the lumern component block is a 32x32 block, the chroma component block may be a 16x16 block. The lumern component may be configured to have LFNST applied, and the chroma component may also be configured to have LFNST applied. Alternatively, the lumern component may be configured to have LFNST applied, but neither NSPT nor LFNST may be applied to the chroma component. In this case, even if neither NSPT nor LFNST is applied to the chroma component, zero-out may be applied in the same manner as when NSPT or LFNST is applied, as described above. Also, in this example, since the chroma component block is a 16x16 block, zero-out using LFNST may be applied.

[0357] If the current block's tree type is a single tree and ISP (Intra Sub-Partition) mode is applied to the current block, or if the current block's tree type is a dual tree, the width and height of the chroma component block do not necessarily have to be half the width and height of the luma component block, respectively. In such cases, for the luma component, whether or not NSPT or LFNST is applied and / or the matrix size (or input / output length) of the non-separable transformation may be determined based on the size of the transformation block. For the chroma component, whether or not NSPT or LFNST is applied and / or the matrix size (or input / output length) of the non-separable transformation may be determined based on the size of the transformation block. Furthermore, if the current block's tree type is a single tree, as mentioned above, the chroma component may be configured not to be subjected to NSPT or LFNST, and zero-out may be applied in the same manner as when NSPT or LFNST is applied.

[0358] When an inseparable transformation is applied to a lumen component or a chroma component, the transformation index for the inseparable transformation may be signaled if the following conditions are met. The conditions for signaling the transformation index, described later, may be considered as additional conditions to the first condition for the lumen component and the second condition for the chroma component. That is, the transformation index may be signaled only if both the first and second conditions are met and at least one of the conditions described later is met. Alternatively, the conditions described later may be considered as independent conditions regardless of the first and second conditions. That is, the transformation index may be signaled if at least one of the conditions described later is met, regardless of whether the first and second conditions are met.

[0359] 1) A conversion index is signaled if a non-zero conversion coefficient exists at a position other than the top-left sample position for at least one conversion block of the color components (i.e., Y, Cb, Cr). Otherwise (i.e., no non-zero conversion coefficient exists at a position other than the top-left sample position for any of the color component conversion blocks), the conversion index does not need to be signaled.

[0360] For example, if the current block's tree type is a dual tree and the current block is a chroma component transformation block, the transformation index may only be signaled if a non-zero transformation coefficient exists at a position other than the upper left sample position for at least one of the transformation blocks of the Cb component and the Cr component.

[0361] Alternatively, if the current block's tree type is a single tree, the transformation index may be signaled only if a non-zero transformation coefficient exists at a position other than the upper-left sample position for at least one of the transformation blocks, which consists of a lumen component and a chroma component (the chroma component can consist of a Cb component and a Cr component).

[0362] Alternatively, if the current block's tree type is a single tree and a non-separable transformation is applied only to the lumern component, a transformation index may be signaled if a non-zero transformation coefficient exists at a position other than the upper-left sample position for at least one transformation block of the lumern component or chromar component, and if the first condition for the lumern component and the second condition for the chromar component are satisfied. Even if no non-zero transformation coefficient exists in the transformation block of the lumern component, a transformation index may be signaled if the first and second conditions are satisfied.

[0363] 2) If at least one of the color components of the current block (i.e., Y, Cb, Cr) is encoded with a conversion skip (i.e., the conversion skip flag for that component is 1), then the unseparated conversion does not need to be applied to the conversion blocks of all color components. If any one of the color components is encoded with a conversion skip, the conversion index for the current block does not need to be signaled. The conversion index may be induced to 0.

[0364] Alternatively, a conversion skip may be applied to the conversion blocks of some of the color components in the current block, while the conversion skip may not be applied to the conversion blocks of the other components. In this case, the non-separated conversion may not be applied to the conversion blocks of the color components to which the conversion skip was applied, while the non-separated conversion may be applied to the conversion blocks of the remaining color components.

[0365] Based on whether a predetermined condition is met for the color component to which the conversion skip is applied, a non-separated conversion may be applied to the color component to which the conversion skip is not applied, and a conversion index for that color component may be signaled. For example, a conversion index may be signaled for the color component to which the conversion skip is not applied only if a predetermined condition is met for the color component to which the conversion skip is applied. Here, the predetermined condition may include at least one of the first condition relating to the lumens component, the second condition relating to the chroma component, or the third condition that a non-zero conversion coefficient exists at a position other than the upper left sample position in the conversion block for at least one color component. Alternatively, the system may be configured not to check the predetermined condition for the color component to which the conversion skip is applied.

[0366] An NSPT kernel may be constructed with 8-bit precision. The coefficients within the NSPT kernel may be greater than or equal to -128 and less than or equal to 127. If the precision is increased beyond 8 bits, the result obtained from matrix multiplication can be shifted to the right by the amount of the increased precision. For example, if matrix multiplication is performed based on an NSPT kernel with 8-bit precision, and the resulting value is shifted to the right by S bits and stored in a buffer, then if the kernel coefficients are constructed with N-bit precision, the value can be shifted to the right by (S + (N-8)) bits and stored in the buffer.

[0367] When configuring the NSPT kernel with 8-bit precision, it is possible to prevent an excessive increase in internal precision within the code / decoder performing the conversion, thereby minimizing the decrease in compression efficiency while reducing the complexity of implementation in terms of memory requirements and computational complexity.

[0368] Tables 11 and 12 show examples of NSPT kernels applicable to a 4x32 block, and Tables 13 and 14 show examples of NSPT kernels applicable to a 32x4 block. The coefficients constituting the NSPT kernel are expressed with 8-bit precision. The coefficients of the NSPT kernel may have values ​​from -128 to 127. In this example, the total number of predefined NSPT sets identical to those of the encoding and decoding devices is 35, and each NSPT set may consist of 3 candidate NSPT kernels.

[0369] In this example, g_nspt4x32

[35] [3]

[36]

[0128] represents an NSPT kernel applicable to a 4x32 block, and g_nspt32x4

[35] [3]

[36]

[0128] represents an NSPT kernel applicable to a 32x4 block. However, the NSPT set and / or NSPT kernel may be determined considering the aforementioned symmetry. If the current block is a 4x32 block and has a specific intra-prediction mode (e.g., a mode with vertical directionality), then the NSPT set and / or NSPT kernel for a block symmetric to such a block, i.e., a 32x4 block, may be applied identically to the current block. Alternatively, if the current block is a 32x4 block and has a specific intra-prediction mode (e.g., a mode with vertical directionality), then the NSPT set and / or NSPT kernel for a block symmetric to such a block, i.e., a 4x32 block, may be applied identically to the current block. In this case, the input data from the current block is transposed and then multiplied by the NSPT kernel.

[0370] In the following g_nspt4x32

[35] [3]

[36]

[0128] and g_nspt32x4

[35] [3]

[36]

[0128] ,

[35] represents that it consists of 35 NSPT sets, and [3] represents that each NSPT set consists of 3 NSPT kernel candidates. For example, in Tables 11 to 14, / / k can mean an NSPT set with set index k, where k may be in the range of 0 to 34. The three matrices belonging to / / k can represent NSPT kernel candidates with transformation indices 0, 1, and 2, respectively. Also,

[36]

[0128] can represent a 36x128 matrix based on the forward NSPT matrix. Specifically, the first

[36] represents 36 transformation basis vectors, and the second

[0128] represents that each transformation basis vector is a one-dimensional vector composed of 128 coefficients. This can be expressed, from the perspective of forward NSPT, as the input to the forward NSPT being a one-dimensional vector of length 128. The inverse NSPT matrix may be induced by transposing an NSPT kernel selected from the g_nspt4x32 array or g_nspt32x4 array described below. In this case, the inverse NSPT matrix may be a 128x36 matrix.

[0371] Alternatively, R transformation basis vectors may be used, where R may be an integer less than 36. In this case, for each NSPT kernel in Tables 11 to 14, only R rows may be selected from the top instead of the top 36, and the forward NSPT matrix may be an Rx128 matrix. In this case, only the remaining (36-R) rows, excluding the R rows selected from the top (i.e., the transformation basis vectors), may be removed. An NSPT kernel applicable to a 4x32 block may be expressed as g_nspt4x32

[35] [3][R]

[0128] , and an NSPT kernel applicable to a 32x4 block may be expressed as g_nspt32x4

[35] [3][R]

[0128] . The reverse NSPT matrix may be induced by transposing R rows selected from the g_nspt4x32 array or g_nspt32x4 array below. In this case, the reverse NSPT matrix may be a 128xR matrix.

[0372] Table 11-1 Table 11-2 Table 11-3 Table 11-4 Table 11-5 Table 11-6 Table 11-7 Table 11-8 Table 11-9 Table 11-10 Table 11-11 Table 11-12 Table 11-13

[0373] Table 12-1 Table 12-2 Table 12-3 Table 12-4 Table 12-5 Table 12-6 Table 12-7 Table 12-8 Table 12-9 Table 12-10 Table 12-11 Table 12-12 Table 12-13

[0374] Table 13-1 Table 13-2 Table 13-3 Table 13-4 Table 13-5 Table 13-6 Table 13-7 Table 13-8 Table 13-9 Table 13-10 Table 13-11 Table 13-12 Table 13-13

[0375] Table 14-1 Table 14-2 Table 14-3 Table 14-4 Table 14-5 Table 14-6 Table 14-7 Table 14-8 Table 14-9 [Table 14-10] [Table 14-11] [Table 14-12] [Table 14-13]

[0376] Referring to Figure 4, the current block can be reconstructed based on the current block's residual sample (S420).

[0377] Based on the current block's intra-prediction mode, predicted samples for the current block can be derived. Based on the predicted samples and residual samples for the current block, restored samples for the current block can be generated.

[0378] Figure 6 shows a schematic configuration of a decoding device 300 that performs the image decoding method according to this disclosure.

[0379] Referring to Figure 6, the decoding apparatus 300 according to this disclosure may include a conversion coefficient induction unit 600, a resistive sample induction unit 610, and a reconstruction block generation unit 620. The conversion coefficient induction unit 600 may be configured as the entropy decoding unit 310 in Figure 3, the resistive sample induction unit 610 may be configured as the resistive processing unit 320 in Figure 3, and the reconstruction block generation unit 620 may be configured as the addition unit 340 in Figure 3.

[0380] The conversion coefficient induction unit 600 can obtain the current block's residual information from the bitstream, decode it, and induce the conversion coefficients of the current block.

[0381] The resistive sample induction unit 610 can induce a resistive sample of the current block by performing at least one of inverse quantization or inverse transformation on the transformation coefficients of the current block.

[0382] The residual sample induction unit 610 can determine a transformation kernel for the inverse transformation of the current block using a predetermined transformation kernel determination method, and induce the residual sample of the current block based on this. This has been explained with reference to Figure 4, and a detailed explanation will be omitted here.

[0383] The restored block generation unit 620 can restore the current block based on the current block's residual sample.

[0384] Figure 7 is a diagram illustrating an embodiment of the present disclosure, which shows an image encoding method performed by an encoding device 200.

[0385] Referring to Figure 7, residual samples of the current block can be induced (S700).

[0386] The current block's residual samples may be derived by subtracting the predicted samples from the current block's original samples. Here, the predicted samples may be derived based on a predetermined intra-prediction mode.

[0387] Referring to Figure 7, the transform coefficients of the current block can be derived by performing at least one of the following: transformation or quantization of the current block's residual sample (S710).

[0388] The transformation method relating to this disclosure may be understood as the inverse process of the inverse transformation described with reference to Figure 4. The method for determining the transformation kernel for the said transformation is as described with reference to Figure 4, and a detailed explanation is omitted here.

[0389] For example, one or more transformation sets may be defined / configured for the current block's transformation, each transformation set may contain one or more transformation kernel candidates. In this case, one of the multiple transformation sets may be selected as the transformation set for the current block. One of the multiple transformation kernel candidates belonging to the current block's transformation set may be selected. This selection may be made implicitly based on the context of the current block. Alternatively, an index indicating the selection of the optimal transformation set and / or transformation kernel candidate for the current block may be signaled.

[0390] Alternatively, the translation kernel for the current block may be determined based on an MTS set. One of several MTS sets may be selected based on at least one of the current block size or intra-prediction mode. The selected MTS set may contain one or more translation kernel candidates. One of the one or more translation kernel candidates may be selected, and the translation kernel for the current block may be determined based on the selected translation kernel candidate. The selection of the translation kernel candidate may be performed using a translation kernel candidate index induced based on the context of the current block. Alternatively, the optimal translation kernel candidate for the current block may be selected, and a translation kernel candidate index pointing to the selected translation kernel candidate may be signaled.

[0391] Alternatively, the transformation kernel for the current block may be determined based on an unseparated linear transformation (NSPT) kernel. A forward NSPT may be applied to the current block if its size belongs to a first group of block sizes to which an NSPT can be applied, but it may not be applied if its size belongs to a second group. If the size of the current block belongs to the second group, a forward separable linear transformation (e.g., DCT-2) can be applied to the residual samples of the current block to derive transformation coefficients. A forward LFNST may be further applied to all or some of the transformation coefficients derive from the separable linear transformation.

[0392] Furthermore, the NSPT may be applied based on at least one of the tree type or component type of the current block. The NSPT kernel (or NSPT matrix) for the NSPT may be determined by utilizing the symmetry between intra-prediction modes or between block morphologies. When a forward NSPT is applied to an MxN current block, the NSPT kernel may be expressed as rxMN, where r is the output length of the NSPT or the number of transformation coefficients generated by the NSPT, and MN is the product of the width and height of the current block, which can be the input length of the NSPT or the number of residual samples to which the NSPT is applied. A method for determining the size of such an NSPT kernel is described with reference to Figure 4.

[0393] The LFNST and / or NSPT indices for the conversion may be encoded into a single unified syntax, or the LFNST and NSPT indices may be encoded separately and inserted into the bitstream. The binary coding for the LFNST and NSPT indices, and the assignment of the CABAC context and initial values, are described with reference to Figure 4.

[0394] Furthermore, the method for signaling the transformation index is explained with reference to Figure 4, and this can also be applied to the method for encoding the transformation index.

[0395] Referring to Figure 7, the conversion coefficients of the current block can be encoded to generate a bitstream (S720).

[0396] Based on the conversion coefficients of the current block, residual information relating to the conversion coefficients may be generated, and a bitstream may be generated by encoding the residual information.

[0397] Figure 8 shows a schematic configuration of an encoding device 200 that performs the image encoding method according to this disclosure.

[0398] Referring to Figure 8, the encoding device 200 according to this disclosure may include a resistive sample induction unit 800, a conversion coefficient induction unit 810, and a conversion coefficient encoding unit 820. The resistive sample induction unit 800 and the conversion coefficient induction unit 810 may be configured in the resistive processing unit 230 of Figure 2, and the conversion coefficient encoding unit 820 may be configured in the entropy encoding unit 240 of Figure 2.

[0399] The residual sample induction unit 800 can induce the residual sample of the current block by subtracting the predicted sample from the original sample of the current block. Here, the predicted sample may be induced based on a predetermined intra-prediction mode.

[0400] The conversion coefficient induction unit 810 can induce conversion coefficients of the current block by performing at least one of conversion or quantization on the residual sample of the current block. The conversion coefficient induction unit 810 can determine a conversion kernel of the current block based on at least one of the embodiments 1 to 3 described above, and apply the conversion kernel to the residual sample of the current block to induce conversion coefficients.

[0401] The conversion coefficient encoding unit 820 can encode the conversion coefficients of the current block and generate a bitstream.

[0402] In the embodiments described above, the method is explained based on a sequence diagram in a series of steps or blocks, but the embodiments are not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps than those described above. Furthermore, those skilled in the art will understand that the steps shown in the sequence diagram are not exclusive, and other steps may be included, or one or more steps in the sequence diagram may be omitted without affecting the scope of the embodiments described herein.

[0403] The methods relating to the embodiments of this document described above may be implemented in software form, and the encoding and / or decoding devices relating to this document may be included in image processing devices such as TVs, computers, smartphones, set-top boxes, and display devices.

[0404] When the embodiments described in this document are implemented as software, the methods described above may be implemented as modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in memory and executed by a processor. The memory may be located inside or outside the processor and may be connected to the processor by various well-known means. The processor may include an ASIC (application-specific integrated circuit), other chipsets, logic circuits, and / or data processing devices. The memory may include ROM (read-only memory), RAM (random access memory), flash memory, memory cards, storage media, and / or other storage devices. In other words, the embodiments described in this document may be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure may be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, information on instructions or algorithms for implementation may be stored on a digital storage medium.

[0405] Furthermore, the decoding and encoding devices to which the embodiments of this specification apply may include multimedia broadcasting transceivers, mobile communication terminals, home cinema video equipment, digital cinema video equipment, surveillance cameras, video conferencing equipment, real-time communication equipment such as video communication, mobile streaming equipment, storage media, camcorders, video-on-demand (VoD) service providers, over-the-top (OTT) video equipment, internet streaming service providers, 3D video equipment, virtual reality (VR) equipment, argumentative reality (AR) equipment, image-phone video equipment, transportation terminals (e.g., vehicle terminals (including autonomous vehicles), airplane terminals, ship terminals, etc.), and medical video equipment, and may be used to process video signals or data signals. For example, over-the-top (OTT) video equipment may include game consoles, Blu-ray players, internet-connected TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs).

[0406] Furthermore, the processing methods to which the embodiments of this specification apply may be produced in the form of programs executed on a computer and stored on a computer-readable recording medium. Similarly, multimedia data having the data structures according to the embodiments of this specification may also be stored on a computer-readable recording medium. The computer-readable recording medium includes all kinds of storage devices and distributed storage devices on which computer-readable data is stored. The computer-readable recording medium may include, for example, Blu-ray discs (BDs), general-purpose serial buses (USBs), ROMs, PROMs, EPROMs, EEPROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices. The computer-readable recording medium also includes media embodied in the form of carrier waves (e.g., transmission over the Internet). Furthermore, a bitstream generated by an encoding method may be stored on a computer-readable recording medium or transmitted over a wireless communication network.

[0407] Furthermore, the embodiments of this specification may be embodied as computer program products in the form of program code, and the program code may be executed on a computer according to the embodiments of this specification. The program code may be stored on a computer-readable carrier.

[0408] Figure 9 shows an example of a content streaming system to which the embodiments of this disclosure can be applied.

[0409] Referring to Figure 9, the content streaming system to which the embodiments described herein apply may broadly include an encoding server, a streaming server, a web server, media storage, user equipment, and multimedia input devices.

[0410] The encoding server is responsible for compressing content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and transmitting this bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, or camcorder directly generates the bitstream, the encoding server may be omitted.

[0411] The bitstream may be generated by an encoding method or bitstream generation method to which any embodiment of this specification is applied, and the streaming server may temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0412] The streaming server transmits multimedia data to the user's device based on a user request via a web server, and the web server acts as an intermediary to inform the user about available services. When a user requests a desired service from the web server, the web server transmits it to the streaming server, and the streaming server transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, in which case the control server is responsible for controlling the commands and responses between the devices in the content streaming system.

[0413] The streaming server can receive content from media storage and / or encoding servers. For example, when receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0414] Examples of user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, PDAs (personal digital assistants), PMPs (portable multimedia players), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (such as smartwatches, smart glasses, and HMDs), digital TVs, desktop computers, and digital signage.

[0415] Each server in the aforementioned content streaming system may be operated as a distributed server, in which case the data received by each server may be processed in a distributed manner.

[0416] The claims described herein may be combined in various ways. For example, the technical features of the method claims herein may be combined to be embodied as an apparatus, and the technical features of the apparatus claims herein may be combined to be embodied as a method. Furthermore, the technical features of the method claims and the technical features of the apparatus claims herein may be combined to be embodied as an apparatus, and the technical features of the method claims and the technical features of the apparatus claims herein may be combined to be embodied as a method.

Claims

1. The step of deriving the transformation coefficients of the current block from the bitstream, The steps include determining a set of non-separable primary transforms (NSPTs) for the current block, The step of determining the NSPt kernel of the current block from the aforementioned NSPt set, The steps include: performing NSPT on the transformation coefficients of the current block based on the NSPT kernel to induce the residual sample of the current block; The step includes restoring the current block based on the said residual sample, A video decoding method in which the NSPT is applied based on the fact that the current block size belongs to one or more block size groups to which the NSPT can be applied.

2. The video decoding method according to claim 1, wherein the NSPT set is determined to be one of 35 predefined NSPT sets.

3. The video decoding method according to claim 1, wherein the NSPT set includes three NSPT kernel candidates.

4. The video decoding method according to claim 1, wherein the group includes at least one block size from 4x4, 4x8, 8x4, 8x8, 16x8, 8x16, 16x4, 4x16, 4x32, or 32x4.

5. The video decoding method according to claim 1, wherein, based on the size of the current block being 4x32 or 32x4, the number of conversion coefficients to which the NSPT is applied is less than or equal to 36.

6. Currently, we are in the stage of inducing the residual sampling of the block, The steps include determining a set of non-separable primary transforms (NSPTs) for the current block, The steps include: applying NSPT to the current block's residual sample based on one of several NSPT kernel candidates belonging to the NSPT set to derive the transformation coefficients of the current block; The step includes encoding the conversion coefficients of the current block to generate a bitstream, A video encoding method in which the NSPT is applied based on the fact that the current block size belongs to one or more block size groups to which the NSPT can be applied.

7. The video encoding method according to claim 6, wherein the NSPT set is determined to be one of 35 predefined NSPT sets.

8. The video encoding method according to claim 6, wherein the NSPT set includes three NSPT kernel candidates.

9. The video encoding method according to claim 6, wherein the group includes at least one block size from 4x4, 4x8, 8x4, 8x8, 16x8, 8x16, 16x4, 4x16, 4x32, or 32x4.

10. The video encoding method according to claim 6, wherein, based on the current block size being 4x32 or 32x4, the number of conversion coefficients induced by the NSPT is less than or equal to 36.

11. A computer-readable recording medium for storing a bitstream generated by the video encoding method described in claim 6.

12. A step of acquiring a bitstream of video information, wherein the bitstream is generated by: inducing the current block's residual samples; determining a set of non-separable primary transforms (NSPTs) for the current block; applying the NSPTs to the current block's residual samples based on one of a plurality of NSPT kernel candidates belonging to the NSPT set to induce the current block's transformation coefficients; and encoding the current block's transformation coefficients. The step of transmitting data including the bitstream, A data transmission method in which the NSPT is applied based on the fact that the current block size belongs to one or more block size groups to which the NSPT is applicable.

Images