Image encoding / decoding method and apparatus, and recording medium storing bitstreams

Abstract

The image decoding method and apparatus according to the present disclosure can obtain residual information from a bitstream, derive transform coefficients of a current block based on the residual information, determine a transform kernel for the current block based on a transform index for inverse transform of the current block, apply an inverse transform to at least one of the transform coefficients based on the transform kernel to derive residual samples, and reconstruct the current block based on the residual samples of the current block.

Description

[Technical Field]

[0001] The present invention relates to an image encoding / decoding method and apparatus, and a recording medium storing a bitstream. [Background technology]

[0002] 2. Description of the Related Art In recent years, the demand for high-resolution, high-quality images such as HD (High Definition) images and UHD (Ultra High Definition) images has increased in various application fields, and as a result, highly efficient image compression techniques have been discussed.

[0003] There are various image compression techniques, such as inter-prediction techniques that predict pixel values ​​contained in a current picture from pictures before or after the current picture, intra-prediction techniques that predict pixel values ​​contained in a current picture using pixel information within the current picture, and entropy coding techniques that assign short codes to values ​​that occur frequently and long codes to values ​​that occur less frequently. Using such image compression techniques, image data can be effectively compressed and transmitted or stored. Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure seeks to provide a method and apparatus for performing transformations using non-separable linear transforms.

[0005] The present disclosure seeks to provide a method and apparatus for performing a transform using a reduced-dimensional non-separable linear transform kernel.

[0006] The present disclosure seeks to provide a method and apparatus for constructing a non-separable transform set and signaling one or more transform kernels that belong to it.

[0007] The present disclosure seeks to provide a method and apparatus for determining and signaling a non-separable transform kernel based on coding parameters. [Means for solving the problem]

[0008] The image decoding method and apparatus according to the present disclosure can obtain residual information from a bitstream, derive transform coefficients of a current block based on the residual information, determine a transform kernel for the current block based on a transform index for inverse transform of the current block, apply the inverse transform to at least one of the transform coefficients based on the transform kernel to derive residual samples, and reconstruct the current block based on the residual samples of the current block.

[0009] In the image decoding method and apparatus according to the present disclosure, the transform kernel may be determined from a non-separable linear transform (NSPT) set including one or more transform kernel candidates based on the size of the current block belonging to a first group, which is a set of block sizes to which an NSPT can be applied.

[0010] In the image decoding method and apparatus according to the present disclosure, based on the size of the current block not belonging to the first group, the transform kernel may be determined from an LFNST set including one or more low frequency non-separable transform (LFNST) kernel candidates.

[0011] In the image decoding method and apparatus according to the present disclosure, the transform index may represent either a non-separable linear transform (NSPT) index or a low-frequency non-separable transform (LFNST) index.

[0012] In the image decoding method and apparatus according to the present disclosure, the NSPT index may be coded based on at least one of a binarization method different from that of the LFNST index or a CABAC context.

[0013] In the image decoding method and apparatus according to the present disclosure, the inverse transform is performed based on a non-separable linear transform (NSPT), and an NSPT set for the NSPT may include one or more NSPT kernel candidates.

[0014] In the image decoding method and apparatus according to the present disclosure, the number of the one or more NSPT kernel candidates belonging to the NSPT set may be determined based on the size of the current block.

[0015] In the image decoding method and apparatus according to the present disclosure, the inverse transform is performed based on a low frequency non-separable transform (LFNST), and based on the size of the current block being MxN or NxM, the number of transform coefficients output by the LFNST is (2*M*M), where N may be an integer greater than or equal to (2*M).

[0016] In the image decoding method and apparatus according to the present disclosure, the size of the current block may be 4x8, 8x4, 8x16, or 16x8.

[0017] In the image decoding method and apparatus according to the present disclosure, the number of transform coefficients output by the LFNST may be 16 or 48 depending on whether the size of the current block is MxN or NxM.

[0018] The image encoding method and apparatus according to the present disclosure can derive residual samples of a current block, apply a transform to the residual samples of the current block to derive transform coefficients of the current block, and encode the transform coefficients of the current block to generate a bitstream.

[0019] In the image encoding method and apparatus according to the present disclosure, the bitstream may further include a transform index for the transform, where the transform index may be a non-separable linear transform (NSPT) index indicating one or more NSPT kernel candidates belonging to an NSPT set, or a low-frequency non-separable transform (LFNST) index indicating one or more LFNST kernel candidates belonging to an LFNST set.

[0020] A computer-readable digital storage medium is provided having encoded video / image information stored thereon that enables an image decoding method to be performed by a decoding device according to the present disclosure.

[0021] A computer-readable digital storage medium is provided having stored thereon video / image information generated by the image encoding method of the present disclosure.

[0022] A method and apparatus for transmitting video / image information generated by the image encoding method according to the present disclosure is provided. [Effects of the Invention]

[0023] The present disclosure can improve the performance of the transform by using a non-separable linear transform as the primary transform.

[0024] The present disclosure can improve the performance of the transform by performing the transform with a non-separable linear transform kernel of reduced dimensionality.

[0025] The construction of transform sets and signaling of transform indices according to the present disclosure can reduce implementation complexity, and entropy coding optimized for non-separable transforms can improve compression performance.

[0026] According to this disclosure, coding parameters such as intra prediction mode, block geometry, etc. can be used to effectively determine or signal non-separable transform kernels. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 illustrates a video / image coding system according to the present disclosure. [Figure 2] 1 is a schematic block diagram of an encoding device to which embodiments of the present disclosure can be applied, in which video / image signals are encoded. [Figure 3] 1 is a schematic block diagram of a decoding device to which embodiments of the present disclosure can be applied, in which video / image signals are decoded. [Figure 4] 1 illustrates an image decoding method performed by a decoding device (300) according to an embodiment of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating intra-prediction modes and their prediction directions according to the present disclosure. [Figure 6] FIG. 3 is a diagram showing a schematic configuration of a decoding device (300) that performs the image decoding method according to the present disclosure. [Figure 7] 1 illustrates an image encoding method performed by an encoding device (200) according to one embodiment of the present disclosure. [Figure 8] FIG. 1 is a diagram showing a schematic configuration of an encoding device (200) that performs an image encoding method according to the present disclosure. [Figure 9] FIG. 1 illustrates an example of a content streaming system to which embodiments of the present disclosure can be applied. DETAILED DESCRIPTION OF THE INVENTION

[0028] While the present disclosure may be modified in various ways and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the present disclosure to the specific embodiments, and it should be understood that the present disclosure includes all modifications, equivalents, and alternatives within the spirit and technical scope of the present disclosure. In the description of each figure, similar reference numerals are used to refer to similar components.

[0029] Terms such as "first," "second," etc. may be used to describe various components, but these components should not be limited by such terms. These terms are used merely to distinguish one component from another. For example, a first component could be termed a second component, and similarly, a second component could be termed a first component, without departing from the scope of the present disclosure. The term "and / or" includes a combination of multiple associated listed items or any item of multiple associated listed items.

[0030] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that the component may be directly coupled or connected to the other component, and that there may be additional components in between. On the other hand, when a component is referred to as being "directly coupled" or "directly connected" to another component, it should be understood that there are no additional components in between.

[0031] The terms used in this application are merely for the purpose of describing particular embodiments and are not intended to limit the present disclosure. The singular terms also include the plural terms unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "have" (comprising; constituting; constructing; setting; including; containing; containing) are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0032] The present disclosure relates to video / image coding. For example, the methods / embodiments disclosed herein may be applied to methods disclosed in the versatile video coding (VVC) standard. The methods / embodiments disclosed herein may also be applied to methods disclosed in the essential video coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the second generation audio video coding standard (AVS2), or next-generation video / image coding standards (e.g., H.267 or H.268).

[0033] This specification presents various embodiments relating to video / image coding, and unless otherwise stated, the above embodiments may be performed in combination with each other.

[0034] In this specification, video may refer to a collection of a series of images over time. A picture generally refers to a unit that represents an image at a specific time period, and a slice / tile is a unit that constitutes part of a picture in coding. A slice / tile may include one or more coding tree units (CTUs). One picture may be composed of one or more slices / tiles. A tile is a rectangular area composed of multiple CTUs in a specific tile column and a specific tile row of a picture. A tile column is a rectangular area of ​​CTUs with a height equal to the height of the picture and a width specified by a picture parameter set syntax requirement. A tile row is a rectangular area of ​​CTUs with a height specified by a picture parameter set and a width equal to the width of the picture. CTUs within a tile are arranged consecutively by CTU raster scanning, while tiles within a picture may be arranged consecutively by tile raster scanning. A slice may contain an integer number of complete tiles or an integer number of consecutive complete CTU rows within the tiles of a picture that may be contained exclusively in a single NAL unit, while a picture may be partitioned into two or more sub-pictures, which may be rectangular regions of one or more slices in a picture.

[0035] A picture element, pixel, or pel can refer to the smallest unit that makes up a picture (or image). A "sample" can also be used as a term corresponding to a pixel. A sample can generally indicate a pixel or a pixel value, and may indicate only a pixel / pixel value of a luminance (luma) component, or may indicate only a pixel / pixel value of a color difference (chroma) component.

[0036] A unit may refer to a basic unit of image processing. A unit may include at least one of a specific region of a picture and information related to that region. One unit may include one luma block and two chroma (e.g., cb, cr) blocks. The term unit may sometimes be used interchangeably with terms such as block or area. In general, an MxN block may include a set (or array) of samples or transform coefficients consisting of M columns and N rows.

[0037] As used herein, "A or B" can mean "A only," "B only," or "both A and B." In other words, as used herein, "A or B" can be interpreted as "A and / or B." For example, as used herein, "A, B, or C" can mean "A only," "B only," "C only," or "any combination of A, B, and C."

[0038] As used herein, a slash ( / ) or a comma can mean "and / or." For example, "A / B" can mean "A and / or B." Thus, "A / B" can mean "A only," "B only," or "both A and B." For example, "A, B, C" can mean "A, B, or C."

[0039] As used herein, "at least one of A and B" can mean "A only," "B only," or "both A and B." Furthermore, as used herein, the expressions "at least one of A or B" or "at least one of A and / or B" may be interpreted as being the same as "at least one of A and B."

[0040] Furthermore, in this specification, "at least one of A, B, and C" can mean "A only," "B only," "C only," or "any combination of A, B, and C." Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" can mean "at least one of A, B, and C."

[0041] Furthermore, parentheses used in this specification may mean "for example." Specifically, when "prediction (intra prediction)" is displayed, "intra prediction" may be suggested as an example of "prediction." In other words, "prediction" in this specification is not limited to "intra prediction," and "intra prediction" may be suggested as an example of "prediction." Furthermore, when "prediction (i.e., intra prediction)" is displayed, "intra prediction" may be suggested as an example of "prediction."

[0042] In this specification, technical features individually described in the same drawing may be embodied individually or simultaneously.

[0043] FIG. 1 is a diagram illustrating a video / image coding system according to the present disclosure.

[0044] Referring to FIG. 1, a video / image coding system may include a first device (a source device) and a second device (a receiving device).

[0045] A source device can transmit encoded video / image information or data to a receiving device via a digital storage medium or a network in the form of a file or streaming. The source device may include a video source, an encoding device, and a transmitting unit. The receiving device may include a receiving unit, a decoding device, and a renderer. The encoding device may be called a video / image encoding device, and the decoding device may be called a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display unit, which may be a separate device or an external component.

[0046] A video source can acquire video / images through a video / image capture, synthesis, or generation process. A video source may include a video / image capture device and / or a video / image generation device. A video / image capture device may include one or more cameras, a video / image archive containing previously captured video / images, etc. A video / image generation device may include a computer, tablet, smartphone, etc., and can (electronically) generate video / images. For example, a virtual video / image may be generated through a computer, etc., in which case the video / image capture process may be replaced by a process in which the associated data is generated.

[0047] An encoding device can encode input video / images. The encoding device can perform a series of steps such as prediction, transformation, and quantization for compression and coding efficiency. The encoded data (encoded video / image information) can be output in the form of a bitstream.

[0048] The transmitting unit can transmit the encoded video / image information or data output in the form of a bitstream to a receiving unit of a receiving device via a digital storage medium or a network in the form of a file or streaming. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray (registered trademark: the same applies hereinafter), HDD, SSD, etc. The transmitting unit can include elements for generating a media file according to a predetermined file format and elements for transmission via a broadcasting / communication network. The receiving unit can receive / extract the bitstream and transmit it to a decoding device.

[0049] The decoding device can decode the video / image by performing a series of steps such as inverse quantization, inverse transform, and prediction, which correspond to the operations of the encoding device.

[0050] The renderer can render the decoded video / images, and the rendered video / images can be displayed on a display unit.

[0051] FIG. 2 is a schematic block diagram of an encoding device to which the embodiments of the present disclosure can be applied, where encoding of video / image signals is performed.

[0052] Referring to FIG. 2, the encoding device 200 may include an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter predictor 221 and an intra predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may further include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstructed block generator. The image divider 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filterer 260 may be configured by one or more hardware components (e.g., an encoding device chipset or processor) depending on the embodiment. Furthermore, the memory 270 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware components may further include the memory 270 as an internal / external component.

[0053] The image division unit 210 may divide an input image (or picture, frame) input to the encoding device 200 into one or more processing units. For example, the processing units may be called coding units (CUs). In this case, the coding units may be recursively divided into coding tree units (CTUs) or largest coding units (LCUs) according to a QTBTTT (Quad-tree, Binary-tree, Ternary-tree) structure.

[0054] For example, one coding unit may be divided into multiple coding units having deeper depths based on a quadtree structure, a binary tree structure, and / or a tertiary structure. In this case, for example, the quadtree structure may be applied first, and then the binary tree structure and / or the tertiary structure may be applied later. Alternatively, the binary tree structure may be applied before the quadtree structure. The coding procedure according to the present specification may be performed based on a final coding unit that is not further divided. In this case, based on coding efficiency according to image characteristics, the largest coding unit may be immediately used as the final coding unit, or, if necessary, the coding unit may be recursively divided into coding units of lower depths, and the coding unit with the optimal size may be used as the final coding unit. Here, the coding procedure may include procedures such as prediction, transformation, and restoration, which will be described later.

[0055] As another example, the processing unit may further include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit may be divided or partitioned from the final coding unit. The prediction unit may be a unit of sample prediction, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving a residual signal from the transform coefficients.

[0056] The term "unit" may be used interchangeably with terms such as "block" or "area." In general, an MxN block may represent a set of samples or transform coefficients consisting of M columns and N rows. A sample may generally represent a pixel or a pixel value, and may represent only a pixel / pixel value of a luma component, or may represent only a pixel / pixel value of a chroma component. A sample may be used in terms corresponding to one picture (or image), pixel, or pel.

[0057] The encoding apparatus 200 may subtract a prediction signal (prediction block, prediction sample array) output from the inter prediction unit 221 or the intra prediction unit 222 from an input image signal (original block, original sample array) to generate a residual signal (residual block, residual sample array), and the generated residual signal is transmitted to the conversion unit 232. In this case, a unit in the encoding apparatus 200 that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) may be referred to as a subtraction unit 231.

[0058] The prediction unit 220 may perform prediction on a current block (hereinafter referred to as a current block) and generate a predicted block including prediction samples for the current block. The prediction unit 220 may determine whether intra prediction or inter prediction is applied to the current block or CU. The prediction unit 220 may generate various information related to prediction, such as prediction mode information, as will be described later in the description of each prediction mode, and transmit the information related to prediction to the entropy encoding unit 240. The entropy encoding unit 240 may encode the information related to prediction and output it in the form of a bitstream.

[0059] The intra prediction unit 222 may predict the current block by referring to samples in the current picture. The referenced samples may be located in the neighborhood of the current block or at a certain distance from the current block depending on the prediction mode. In intra prediction, prediction modes may include one or more non-directional modes and multiple directional modes. The non-directional modes may include at least one of DC mode and planar mode. The directional modes may include 33 directional modes or 65 directional modes depending on the granularity of the prediction direction. However, this is merely an example, and more or less directional modes may be used depending on the settings. The intra prediction unit 222 may also determine the prediction mode to be applied to the current block using the prediction modes applied to neighboring blocks.

[0060] The inter prediction unit 221 may derive a prediction block for a current block based on a reference block (reference sample array) identified by a motion vector in a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring blocks may be the same or different. The temporal neighboring blocks may be called collocated reference blocks, collocated control units (colCUs), etc., and the reference picture including the temporal neighboring blocks may be called collocated pictures (colPic). For example, the inter prediction unit 221 may configure a motion information candidate list based on neighboring blocks and generate information indicating which candidates are used to derive a motion vector and / or a reference picture index for the current block. Inter prediction may be performed based on various prediction modes. For example, in the case of a skip mode or a merge mode, the inter prediction unit 221 may use motion information of neighboring blocks as motion information for the current block. In the case of the skip mode, unlike in the merge mode, a residual signal may not be transmitted.In the case of motion vector prediction (MVP) mode, the motion vector of the current block can be indicated by using the motion vector of a neighboring block as a motion vector predictor and signaling the motion vector difference.

[0061] The prediction unit 220 may generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit may apply intra prediction or inter prediction for predicting a block, or may simultaneously apply intra prediction and inter prediction. This may be referred to as a combined inter and intra prediction (CIIP) mode. The prediction unit may also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for coding content images / videos, such as games, such as screen content coding (SCC). IBC basically performs prediction within a current picture, but may be similar to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, a sample value within the picture may be signaled based on information about a palette table and a palette index. The predicted signal generated by the prediction unit 220 may be used to generate a reconstructed signal or a residual signal.

[0062] The transform unit 232 may generate transform coefficients by applying a transform technique to the residual signal. For example, the transform technique may include at least one of a Discrete Cosine Transform (DCT), a Discrete Sine Transform (DST), a Karhunen-Loeve Transform (KLT), a Graph-Based Transform (GBT), and a Conditionally Non-Linear Transform (CNT). Here, GBT refers to a transform obtained from a graph representing inter-pixel relationship information. CNT refers to a transform obtained based on a predicted signal generated using all previously reconstructed pixels. The transform process may be applied to square pixel blocks of the same size, or to non-square blocks of variable sizes.

[0063] The quantization unit 233 quantizes the transform coefficients and transmits the quantized signal to the entropy encoding unit 240. The entropy encoding unit 240 encodes the quantized signal (information about the quantized transform coefficients) and outputs it as a bitstream. The information about the quantized transform coefficients may be referred to as residual information. The quantization unit 233 rearranges the quantized transform coefficients in a block form into a one-dimensional vector form based on a coefficient scan order, and generates information about the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0064] The entropy encoding unit 240 can perform various encoding methods such as exponential Golomb, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), etc. The entropy encoding unit 240 can encode information necessary for video / image restoration (e.g., values ​​of syntax elements) together with or separately from the quantized transform coefficients.

[0065] Encoded information (e.g., encoded video / image information) may be transmitted or stored in the form of a bitstream in network abstraction layer (NAL) units. The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / image information may also include general constraint information. In this specification, information and / or syntax elements transmitted / signaled from an encoding device to a decoding device may be included in the video / image information. The video / image information may be encoded using the encoding procedure described above and included in the bitstream. The bitstream may be transmitted over a network or stored in a digital storage medium. Here, the network may include a broadcast network and / or a communication network, and the digital storage medium may include various storage media, such as a USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The signal output from the entropy encoding unit 240 may be transmitted to a transmitting unit (not shown) and / or stored to a storing unit (not shown) configured as an internal / external element of the encoding device 200, or the transmitting unit may be included in the entropy encoding unit 240.

[0066] The quantized transform coefficients output from the quantization unit 233 may be used to generate a prediction signal. For example, the inverse quantization unit 234 and the inverse transform unit 235 may apply inverse quantization and inverse transform to the quantized transform coefficients to reconstruct a residual signal (residual block or residual sample). The adder 250 may generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the reconstructed residual signal to a prediction signal output from the inter prediction unit 221 or the intra prediction unit 222. When there is no residual for the current block, such as when a skip mode is applied, the predicted block may be used as the reconstructed block. The adder 250 may be referred to as a reconstruction unit or a reconstructed block generator. The generated reconstructed signal may be used for intra prediction of the next block to be processed in the current picture, or may be used for inter prediction of the next picture after filtering, as described below. Meanwhile, LMCS (luma mapping with chroma scaling) may be applied during picture encoding and / or reconstruction.

[0067] The filtering unit 260 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 260 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture and store the modified reconstructed picture in the memory 270, specifically, in the DPB of the memory 270. The various filtering methods may include deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc. The filtering unit 260 may generate various information related to filtering and transmit it to the entropy encoding unit 240. The information related to filtering may be encoded by the entropy encoding unit 240 and output in the form of a bitstream.

[0068] The modified reconstructed picture transmitted to the memory 270 may be used as a reference picture in the inter prediction unit 221. This allows the encoding apparatus to avoid prediction mismatch between the encoding apparatus 200 and the decoding apparatus when inter prediction is applied, and also improves coding efficiency.

[0069] The DPB of the memory 270 may store the modified reconstructed picture to be used as a reference picture in the inter predictor 221. The memory 270 may store motion information of a block from which motion information in the current picture is derived (or encoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter predictor 221 to be used as motion information of a spatially neighboring block or a temporally neighboring block. The memory 270 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 222.

[0070] FIG. 3 is a schematic block diagram of a decoding device to which the embodiments of the present disclosure can be applied, in which video / image signals are decoded.

[0071] 3, the decoding device 300 may include an entropy decoding unit (entropy decoder 310), a residual processor (residual processor 320), a predictor (predictor 330), an adder (adder 340), a filter (filter 350), and a memory (memory 360). The predictor 330 may include an inter predictor 331 and an intra predictor 332. The residual processor 320 may include a dequantizer (dequantizer 321) and an inverse transformer (inverse transformer 322).

[0072] The entropy decoding unit 310, residual processing unit 320, prediction unit 330, addition unit 340, and filtering unit 350 may be configured as a single hardware component (e.g., a decoding device chipset or processor) depending on the embodiment. Also, the memory 360 may include a decoded picture buffer (DPB) and may be configured as a digital storage medium. The hardware component may further include the memory 360 as an internal / external component.

[0073] When a bitstream including video / image information is input, the decoding apparatus 300 can reconstruct an image corresponding to the process by which the video / image information was processed by the encoding apparatus of FIG. 2. For example, the decoding apparatus 300 can derive units / blocks based on block division-related information obtained from the bitstream. The decoding apparatus 300 can perform decoding using the processing units applied by the encoding apparatus. Accordingly, the processing unit for decoding may be a coding unit, which may be divided from a coding tree unit or a maximum coding unit according to a quad tree structure, a binary tree structure, and / or a tertiary tree structure. One or more transform units may be derived from the coding unit. The reconstructed image signal decoded and output by the decoding apparatus 300 may be reproduced by a playback device.

[0074] The decoding device 300 may receive a signal output from the encoding device of FIG. 2 in the form of a bitstream, and the received signal may be decoded by the entropy decoding unit 310. For example, the entropy decoding unit 310 may parse the bitstream and derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information may further include information on various parameter sets, such as an adaptation parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). The video / image information may also include general constraint information. The decoding device may decode pictures based on the information on the parameter sets and / or the general constraint information. Signal / received information and / or syntax elements described later in this specification may be decoded by the decoding procedure and obtained from the bitstream. For example, the entropy decoding unit 310 may decode information in a bitstream based on a coding method such as Exponential-Golomb coding, CAVLC, or CABAC, and output values ​​of syntax elements required for image restoration and quantized values ​​of transform coefficients related to residuals. More specifically, the CABAC entropy decoding method receives bins corresponding to each syntax element in the bitstream, determines a context model using information on the syntax element to be decoded, decoding information on neighboring and current blocks, or information on symbols / bins decoded in previous steps, predicts the occurrence probability of the bins based on the determined context model, and generates symbols corresponding to the values ​​of each syntax element by performing arithmetic decoding of the bins. In this case, after determining the context model, the CABAC entropy decoding method may update the context model using information on the decoded symbols / bins for the context model of the next symbol / bin.Information related to prediction among the information decoded by the entropy decoding unit 310 may be provided to a prediction unit (inter prediction unit 332 and intra prediction unit 331), and residual values ​​entropy decoded by the entropy decoding unit 310, i.e., quantized transform coefficients and related parameter information, may be input to a residual processing unit 320. The residual processing unit 320 may derive a residual signal (residual block, residual sample, residual sample array). In addition, information related to filtering among the information decoded by the entropy decoding unit 310 may be provided to a filtering unit 350. Meanwhile, a receiving unit (not shown) that receives a signal output from the encoding apparatus may be further configured as an internal / external element of the decoding apparatus 300, or the receiving unit may be a component of the entropy decoding unit 310.

[0075] Meanwhile, the decoding apparatus according to the present specification may be referred to as a video / image / picture decoding apparatus, and the decoding apparatus may be divided into an information decoding apparatus (video / image / picture information decoding apparatus) and a sample decoding apparatus (video / image / picture sample decoding apparatus). The information decoding apparatus may include the entropy decoding unit 310, and the sample decoding apparatus may include at least one of the inverse quantization unit 321, the inverse transform unit 322, the addition unit 340, the filtering unit 350, the memory 360, the inter prediction unit 332, and the intra prediction unit 331.

[0076] The inverse quantization unit 321 can inverse quantize the quantized transform coefficients and output the transform coefficients. The inverse quantization unit 321 can rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement can be performed based on the coefficient scanning order performed in the encoding device. The inverse quantization unit 321 can inverse quantize the quantized transform coefficients using a quantization parameter (e.g., quantization step size information) to obtain transform coefficients.

[0077] The inverse transform unit 322 performs inverse transform on the transform coefficients to obtain a residual signal (residual block, residual sample array).

[0078] The prediction unit 320 may perform prediction on a current block and generate a predicted block including prediction samples for the current block. The prediction unit 320 may determine whether intra prediction or inter prediction is applied to the current block based on the prediction information output from the entropy decoding unit 310, and may determine a specific intra / inter prediction mode.

[0079] The prediction unit 320 may generate a prediction signal based on various prediction methods, which will be described later. For example, the prediction unit 320 may apply intra prediction or inter prediction for predicting a block, or may simultaneously apply intra prediction and inter prediction. This may be referred to as a combined inter and intra prediction (CIIP) mode. The prediction unit may also use an intra block copy (IBC) prediction mode or a palette mode for predicting a block. The IBC prediction mode or palette mode may be used for content image / video coding, such as games, such as screen content coding (SCC). IBC basically performs prediction within a current picture, but may be similar to inter prediction in that it derives a reference block within the current picture. That is, IBC may use at least one of the inter prediction techniques described herein. The palette mode may be considered an example of intra coding or intra prediction. When the palette mode is applied, information about a palette table and a palette index may be included in the video / image information and signaled.

[0080] The intra prediction unit 331 may predict a current block by referring to samples in a current picture. The referenced samples may be located in the neighborhood of the current block or at a certain distance from the current block depending on the prediction mode. In intra prediction, prediction modes may include one or more non-directional modes and multiple directional modes. The intra prediction unit 331 may determine a prediction mode to be applied to the current block using prediction modes applied to neighboring blocks.

[0081] The inter prediction unit 332 may derive a prediction block for a current block based on a reference block (reference sample array) identified by a motion vector in a reference picture. To reduce the amount of motion information transmitted in inter prediction mode, the motion information may be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information may include a motion vector and a reference picture index. The motion information may further include inter prediction direction information (e.g., L0 prediction, L1 prediction, Bi prediction, etc.). In the case of inter prediction, the neighboring blocks may include spatial neighboring blocks in the current picture and temporal neighboring blocks in the reference picture. For example, the inter prediction unit 332 may construct a motion information candidate list based on the neighboring blocks and derive a motion vector and / or a reference picture index for the current block based on received candidate selection information. Inter prediction may be performed based on various prediction modes, and the prediction information may include information indicating the inter prediction mode for the current block.

[0082] The adder 340 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the acquired residual signal to a prediction signal (prediction block, prediction sample array) output from a prediction unit (including the inter prediction unit 332 and / or the intra prediction unit 331). When there is no residual for the current block, such as when the skip mode is applied, the prediction block may be used as the reconstructed block.

[0083] The adder 340 may be referred to as a reconstruction unit or a reconstruction block generator. The generated reconstruction signal may be used for intra prediction of a next block to be processed in a current picture, may be output after filtering as described below, or may be used for inter prediction of a next picture. Meanwhile, luma mapping with chroma scaling (LMCS) may be applied during picture decoding.

[0084] The filtering unit 350 may apply filtering to the reconstructed signal to improve subjective / objective image quality. For example, the filtering unit 350 may apply various filtering methods to the reconstructed picture to generate a modified reconstructed picture, and may transmit the modified reconstructed picture to the memory 360, specifically, to the DPB of the memory 360. The various filtering methods may include deblocking filtering, sample adaptive offset, an adaptive loop filter, a bilateral filter, etc.

[0085] The (modified) reconstructed picture stored in the DPB of the memory 360 may be used as a reference picture in the inter predictor 332. The memory 360 may store motion information of a block from which motion information in the current picture is derived (or decoded) and / or motion information of a block in an already reconstructed picture. The stored motion information may be transmitted to the inter predictor 260 to be used as motion information of a spatially neighboring block or a temporally neighboring block. The memory 360 may store reconstructed samples of reconstructed blocks in the current picture and transmit them to the intra predictor 331.

[0086] In this specification, the embodiments described for the filtering unit 260, inter prediction unit 221, and intra prediction unit 222 of the encoding device 200 may also be applied identically or correspondingly to the filtering unit 350, inter prediction unit 332, and intra prediction unit 331 of the decoding device 300, respectively.

[0087] FIG. 4 is a diagram illustrating an image decoding method performed by a decoding device according to an embodiment of the present disclosure.

[0088] 4, transform coefficients of a current block can be derived from a bitstream (S400). That is, the bitstream may include residual information of the current block, and the transform coefficients of the current block can be derived by decoding the residual information.

[0089] Referring to FIG. 4, residual samples of the current block can be derived by performing at least one of dequantization and inverse transform on the transform coefficients of the current block (S410).

[0090] When an Adaptive Multiple Transform Selection (MTS) is applied, the inverse transform may be performed based on at least one of DCT-2, DST-7, or DCT-8, which may be referred to as a transform type, transform kernel, or transform core.

[0091] In the present disclosure, the inverse transform may refer to a separable transform. However, without being limited thereto, the inverse transform may also refer to a non-separable transform, and may be a concept including both separable and non-separable transforms. Furthermore, while the inverse transform in the present disclosure refers to a primary transform, it is not limited thereto and may be transformed into an identical / similar form and applied to a secondary transform.

[0092] For example, as a method for inverse transformation, only DCT-2 and a non-separable transform may be used, or a non-separable transform may be further used in addition to at least one of DCT-2, DST-7, and DCT-8, or a non-separable transform may be used instead of one or more transform kernels of DCT-2, DST-7, and DCT-8.

[0093] As a more specific example, when transform kernel candidates for a separable transform include (DCT-2, DCT-2), (DST-7, DST-7), (DCT-8, DST-7), (DST-7, DCT-8), and (DCT-8, DCT-8), a non-separable transform may replace or be added to one or more of the five transform kernel candidates. Here, a notation such as (Transform 1, Transform 2) indicates that transform 1 is applied in the horizontal direction and transform 2 is applied in the vertical direction. When a non-separable transform replaces some of the transform kernel candidates, the remaining transform kernel candidates except for (DCT-2, DCT-2) and (DST-7, DST-7) may be replaced with a non-separable transform. However, the above transform kernel candidates are merely examples, and other types of DCTs and / or DSTs may be included, and a transform skip may also be included as a transform kernel candidate.

[0094] A non-separable transform can refer to a transform or inverse transform based on a non-separable transform matrix. That is, unlike a separable transform, which separates vertical and horizontal transforms and performs horizontal and vertical transforms independently, a non-separable transform can perform horizontal and vertical transforms at the same time.

[0095] For example, when a non-separable transform is performed on a 4x4 block, the input data X to the non-separable transform is as follows:

[0096]

number

[0097] When the input data X is expressed in vector form, the vector X' may be expressed as follows:

[0098]

number

[0099] In this case, the non-separable transformation may be performed as shown in Equation 3 below.

[0100]

number

[0101] In Equation 3, F represents a transform coefficient vector, T represents a 16x16 non-separable transform matrix, and · represents matrix-vector multiplication.

[0102] A 16x1 transform coefficient vector F may be derived by Equation 3 above, and F may be rearranged into 4x4 blocks according to a predetermined scan order, which may be a horizontal scan, a vertical scan, a diagonal scan, a z scan, a raster scan, or a predefined scan.

[0103] The non-separable transform set and / or transform kernel for the non-separable transform may be configured differently based on at least one of the prediction mode (e.g., intra mode, inter mode, etc.), the width, height, or number of pixels of the current block, the position of the sub-block within the current block, explicitly signaled syntax elements, statistical characteristics of surrounding samples, whether or not a quadratic transform is used, or the quantization parameter (QP).

[0104] Specifically, in an intra mode, predefined intra prediction modes may be grouped to correspond to n non-separable transform sets, and each non-separable transform set may include k transform kernel candidates, where n and k may be any constants that follow rules (conditions) that are defined identically for the encoding device and the decoding device.

[0105] The number of non-separable transform sets and / or the number of transform kernel candidates included in the non-separable transform sets may be configured to vary depending on the width and / or height of the current block. For example, for a 4x4 block, n1 non-separable transform sets and k1 transform kernel candidates may be configured. For a 4x8 block, n2 non-separable transform sets and k2 transform kernel candidates may be configured. Furthermore, the number of non-separable transform sets and the number of transform kernel candidates included in each non-separable transform set may be configured to vary depending on the product of the width and height of the current block. For example, if the product of the width and height of the current block is equal to or greater than 256, n3 non-separable transform sets and k3 transform kernel candidates may be configured; otherwise, n4 non-separable transform sets and k4 transform kernel candidates may be configured. That is, since the degree of change in the statistical characteristics of the residual signal varies depending on the block size, the number of non-separable transform sets and the number of transform kernel candidates may be configured to vary to reflect this.

[0106] When a current block is divided into multiple sub-blocks, the statistical characteristics of the residual signals may differ for each sub-block, so the number of non-separable transform sets and transform kernel candidates may be configured to be different. For example, when a 4x8 or 8x4 block is divided into two 4x4 sub-blocks and a non-separable transform is applied to each sub-block, n5 non-separable transform sets and k5 transform kernel candidates may be configured for the top-left 4x4 sub-block, and n6 non-separable transform sets and k6 transform kernel candidates may be configured for the other 4x4 sub-blocks.

[0107] The number of non-separable transform sets and transform kernel candidates may be configured to differ from one another based on an explicitly signaled syntax element. Information indicating one of a plurality of non-separable transform configurations may be used as the syntax element. For example, if three types of non-separable transform configurations are supported (i.e., n7 non-separable transform sets and k7 transform kernel candidates, n8 non-separable transform sets and k8 transform kernel candidates, and n9 non-separable transform sets and k9 transform kernel candidates), the syntax element may have a value of 0, 1, or 2, and the non-separable transform configuration to be applied to the current block may be determined based on the value of the signaled syntax element.

[0108] The number of non-separable transformation sets and transformation kernel candidates may be configured differently depending on whether and / or what kind of secondary transformation is applied. For example, when no secondary transformation is applied, n 10 A set of non-separable transformations and k 10 A non-separable transformation configuration containing n candidate transformation kernels can be applied. 11 A set of non-separable transformations and k 11 A non-separable transform configuration including transform kernel candidates can be applied.

[0109] Different non-separable transform configurations may be applied according to the quantization parameter (QP) and / or range of QP values. For example, when the QP value has a small value, n 12 A set of non-separable transformations and k 12On the other hand, when the QP value is large, n 13 A set of non-separable transformations and k 13 A non-separable transform configuration including candidate transform kernels can be applied. If the QP value is equal to or less than a threshold (e.g., 32), the QP value can be classified as having a small value; otherwise, the QP value can be classified as having a large value. Alternatively, the QP value range can be divided into three or more ranges, and a different non-separable transform configuration can be applied to each range.

[0110] For relatively large blocks, instead of using a non-separable transform corresponding to the width and height of the block, the block can be divided into multiple sub-blocks and a non-separable transform corresponding to the width and height of the sub-block can be used. For example, when performing a non-separable transform on a 4x8 block, the 4x8 block can be divided into two 4x4 sub-blocks and a 4x4 block-based non-separable transform can be used for each 4x4 sub-block. Or, for an 8x16 block, it can be divided into two 8x8 sub-blocks and an 8x8 block-based non-separable transform can be used.

[0111] The non-separable transform set may be determined based on the intra prediction mode of the current block and a mapping table. The mapping table may define a mapping relationship between predefined intra prediction modes and the non-separable transform set. The predefined intra prediction modes may include two non-directional modes and 65 directional modes. Generally, a non-separable transform has a larger transform kernel size than a separable transform. This means that the computational complexity required for the transform process is high and a large memory is required for storing the transform kernel. Meanwhile, a separable transform can only consider statistical characteristics existing in the horizontal and / or vertical directions, while a non-separable transform can simultaneously consider statistical characteristics in two-dimensional space including the horizontal and vertical directions, thereby providing better compression efficiency. Since the statistical characteristics and diversity of residuals differ depending on the directionality of the intra prediction mode, a non-separable transform may be absolutely necessary. Alternatively, there may be an intra prediction mode that can sufficiently grasp the characteristics of the residuals through separable transform alone. Therefore, by predefining which transform to use according to the intra prediction mode in the encoding device and the decoding device, a transform process can be designed with optimized complexity and memory requirements. The non-directional modes may include a planar mode numbered 0 and a DC mode numbered 1, and the directional modes may include intra prediction modes numbered 2 to 66. However, this is merely an example, and the present disclosure may also be applied to cases where the number of predefined intra prediction modes is different.

[0112] By applying wide angle intra prediction (WAIP), the predefined intra prediction modes may further include intra prediction modes from -14 to -1 and intra prediction modes from 67 to 80.

[0113] FIG. 5 exemplarily illustrates intra prediction modes and their prediction directions according to the present disclosure. Referring to FIG. 5, modes -14 to -1 and modes 2 to 33 and modes 35 to 80 are symmetrical in terms of prediction direction around mode 34. For example, modes 10 and 58 are symmetrical around the direction corresponding to mode 34, and mode -1 is symmetrical with mode 67. Therefore, for vertical direction modes that are symmetrical to horizontal direction modes around mode 34, input data can be transposed for use. Transposing input data means that rows in MxN two-dimensional block input data become columns and columns become rows, forming NxM data.

[0114] For example, when a 4x4 block is used, 16 data pieces constituting the 4x4 block can be appropriately arranged to form a 16x1 one-dimensional vector for non-separable transformation. In this case, the one-dimensional vector can be arranged in row-major order or column-major order. The residual samples resulting from the non-separable transformation can be arranged in the above order to form a two-dimensional block.

[0115] If the data arrangement order for constructing the 16x1 input vector is row-major for modes -14 to -1 and modes 2 to 33, the input vector may be constructed in column-major order for modes 35 to 80.

[0116] Although the 34th mode can be considered neither a horizontal nor a vertical directional mode, this disclosure classifies it as belonging to the horizontal directional mode. That is, for modes -14 to -1 and 2 to 33, the input data sorting method for the horizontal directional mode, i.e., row-priority order, is used, and the input data can be transposed and used for the vertical directional mode, which is symmetrical around the 34th mode.

[0117] Non-square blocks cannot utilize symmetry in square blocks (i.e., the symmetry between the P mode and the (68-P) mode in an NxN block (2<=P<=33) or the symmetry between the Q mode and the (66-Q) mode (-14<=Q<=-1)). Therefore, in addition to the symmetry based only on the intra prediction mode, symmetry between block types that are in a mutually preceding relationship, i.e., the symmetry between the KxL block and the LxK block, can also be utilized. Specifically, a symmetrical relationship exists between the KxL block predicted with the P mode and the LxK block predicted with the (68-P) mode. Alternatively, a symmetrical relationship exists between the KxL block predicted with the Q mode and the LxK block predicted with the (66-Q) mode.

[0118] Since the KxL block having mode 2 and the LxK block having mode 66 are considered to be symmetrical to each other, the same transform kernel can be applied to the KxL block and the LxK block. If a non-separable transform set is mapped to the intra prediction mode of the KxL block, to apply a non-separable transform to the LxK block, the non-separable transform set can be derived using a mapping table corresponding to the KxL block based on mode (68-P) instead of mode P applied to the LxK block. Alternatively, the non-separable transform set can be derived using a mapping table corresponding to the KxL block based on mode (66-Q) instead of mode Q applied to the LxK block.

[0119] For example, to apply a non-separable transform to an LxK block, a non-separable transform set can be selected based on mode 2 instead of mode 66. For a KxL block, input data can be read in a predetermined order (e.g., row-major order or column-major order) to construct a one-dimensional vector, and then the non-separable transform can be applied. For an LxK block, input data can be read in a transposed order to construct a one-dimensional vector, and then the non-separable transform can be applied. That is, if the KxL block is read in row-major order, the LxK block can be read in column-major order. Conversely, if the KxL block is read in column-major order, the LxK block can be read in row-major order.

[0120] Also, when the 34th mode is applied to a KxL block, a non-separable transform set is determined based on the 34th mode, and the input data is read in a predetermined order to construct a one-dimensional vector, which can then be used to perform the non-separable transform. Similarly, when the 34th mode is applied to an LxK block, a non-separable transform set is determined based on the 34th mode, but the input data is read in a transposed order to construct a one-dimensional vector, which can then be used to perform the non-separable transform.

[0121] While this disclosure has described a method for determining a non-separable transform set and a method for configuring input data based on a KxL block, the non-separable transform can be performed by using the same symmetry described above for a KxL block based on an LxK block. Alternatively, a block whose width is greater than its height may be restricted to be used as a reference block. Alternatively, a non-square block may be restricted from utilizing symmetry. In this case, a non-square block may use a different number of non-separable transform sets and / or transform kernel candidates than a square block, and a different mapping table may be used to select a non-separable transform set.

[0122] An example of a mapping table for selecting a non-separable transformation set is as follows:

[0123] [Table 1]

[0124] Table 1 shows an example of allocating non-separable transform sets for each intra prediction mode when there are five non-separable transform sets. The value of predModeIntra indicates the value of the intra prediction mode taking WAIP into consideration, and TrSetIdx is an index indicating a specific non-separable transform set. It can be seen from Table 1 that the same non-separable transform set is applied to modes located in symmetric directions depending on the intra prediction mode. Table 1 is merely an example of using five non-separable transform sets and does not limit the total number of non-separable transform sets for non-separable transforms.

[0125] Alternatively, as shown in Table 2, non-separable transforms may not be applied to WAIP for compression performance reasons.

[0126] [Table 2]

[0127] Alternatively, as shown in Table 3, a separate non-separable transform set may not be configured for WAIP, and non-separable transform sets corresponding to adjacent intra prediction modes may be shared.

[0128] [Table 3]

[0129] The non-separable transform set may include a plurality of transform kernel candidates, and any one of the plurality of transform kernel candidates may be selectively used. To this end, an index signaled by a bitstream may be used. Alternatively, any one of the plurality of transform kernel candidates may be implicitly determined based on context information of the current block. Here, the context information may indicate the size of the current block or whether a non-separable transform is applied to a neighboring block. Here, the size of the current block may be defined as the width, height, maximum / minimum value of the width and height, the sum of the width and height, or the product of the width and height.

[0130] [Embodiment] A method for determining a transformation kernel for the inverse transformation of a current block will now be described in detail.

[0131] Example 1

[0132] As mentioned above, inverse transforms can be classified into separable transforms and non-separable transforms. A separable transform refers to a transform performed on a two-dimensional block in the horizontal and vertical directions, respectively, while a non-separable transform refers to a transform performed once on samples constituting the entire two-dimensional block or a partial region. A separable transform can be expressed as a pair of a horizontal transform and a vertical transform, and will be expressed as (horizontal transform, vertical transform) in this disclosure.

[0133] Multiple transform sets may be defined for the inverse transform of the current block, and each transform set may include one or more candidate transform kernels.

[0134] For example, any one of (DST-7, DST-7), (DCT-8, DST-7), (DST-7, DCT-8), or (DCT-8, DCT-8) may be applied as a separate transform, and these four transform kernel candidates may be considered as one transform set. Also, (DCT-2, DCT-2) may be considered as one transform set. A transform skip that does not apply a transform may also be considered as one transform set, and (DCT-2, DCT-2) and the transform skip may be considered as one transform set. In the present disclosure, a transform kernel may represent one transform (e.g., DCT-2, DST-7) or a pair of two transforms (e.g., (DCT-2, DCT-2)).

[0135] Another example of a transform set may be the non-separable transform set described above. In the present disclosure, a non-separable transform applied as a primary transform may be referred to as an NSPT (Non-Separable Primary Transform). In the NSPT, multiple non-separable transform sets may be configured, and each non-separable transform set may include one or more transform kernels as transform kernel candidates. In the case of the NSPT, one of the multiple non-separable transform sets is selected depending on the intra prediction mode, and the multiple non-separable transform sets for the NSPT may be referred to as an NSPT set list. This has been described above, and a detailed description thereof will be omitted here.

[0136] A group of one or more transform sets available for the current block may be configured from a plurality of predefined transform sets. The group of one or more transform sets may be configured for a predetermined region unit to which the current block belongs, and hereinafter referred to as a collection. Here, the predetermined region unit may be at least one of a picture, a slice, a coding tree unit row (CTU row), or a coding tree unit (CTU).

[0137] For example, a transformation set consisting of (DCT-2, DCT-2) is called S1, and a transformation set consisting of (DST-7, DST-7), (DCT-8, DST-7), (DST-7, DCT-8), and (DCT-8, DCT-8) is called S2. In addition, the above-mentioned NSPT set list may include N non-separable transformation sets, and the N non-separable transformation sets are called S 3,1 , S 3,2 , ..., S 3,N Here, N may be 35, but is not limited to this.

[0138] S depending on the intra prediction mode of the current block 3,13 is selected as the non-separable transformation set for NSPT, the transformation kernels applicable to the current block are S1, S2, or S 3,13 In this case, the collection in which the block is currently available is defined as {S1, S2, S 3,13} can be written as

[0139] As described above, a collection according to the present disclosure is a group of one or more transform sets available to a current block, and the collection may be configured to vary depending on the context of the current block. Here, the context may include at least one of a shape, a size, or an intra-prediction mode. If a total of K contexts are defined, K collections may be generated, and each collection may include C i (i=1, 2, ..., N). For example, if the block sizes to which NSPT can be applied are 4x4, 8x8, 16x16, and 32x32, and one of a total of 35 non-separable transform sets is selected depending on the intra prediction mode, if different transform kernels are applied for each block size, a total of 4x35=140 contexts may be defined.

[0140] A collection may be constructed based on the context of the current block. At this time, a process of selecting one of a plurality of transform sets belonging to the collection and selecting one of a plurality of transform kernel candidates belonging to the selected transform set may be performed. Here, the selection of the transform set and the transform kernel candidate may be performed implicitly based on the context of the current block or based on an explicitly signaled index. Alternatively, the process of selecting one of a plurality of transform sets belonging to the collection and the process of selecting one of a plurality of transform kernel candidates belonging to the selected transform set may be performed separately. For example, an index for selecting a transform set may be first signaled, and one of a plurality of transform sets belonging to the collection may be selected based on the index. Then, an index indicating one of a plurality of transform kernel candidates belonging to the transform set may be signaled, and one of the transform kernel candidates may be selected from the transform set based on the signaled index. A transform kernel for the current block may be determined based on the selected transform kernel candidate. Alternatively, the selection of any one transform set from the collection may be implicitly performed based on the context of the current block, and the selection of any one transform kernel candidate from the selected transform set may be performed based on a signaled index. Alternatively, the selection of any one transform set from the collection may be based on a signaled index, and the selection of any one transform kernel candidate from the selected transform set may be implicitly performed based on the context of the current block. Alternatively, the selection of any one transform set from the collection may be implicitly performed based on the context of the current block, and the selection of any one transform kernel candidate from the selected transform set may also be implicitly performed based on the context of the current block. Of course, if the number of transform sets belonging to the collection is one, an index for selecting a transform set does not need to be signaled. Similarly, if the number of transform kernel candidates belonging to the selected transform set is one, an index for indicating the transform kernel candidate does not need to be signaled.Alternatively, an index indicating any one of all transform kernel candidates currently belonging to the collection may be signaled. In this case, the process of selecting any one transform set from the collection may be omitted. In this case, all transform sets belonging to the collection may be rearranged in consideration of priority. For example, when assigning small-length binary codes to small-value indices, such as truncated unary codes, it may be advantageous to assign small-value indices to transform kernel candidates that are relatively advantageous in improving coding performance. When rearranging all transform kernel candidates belonging to a collection according to priority, different shuffling may be applied to each collection. Also, instead of rearranging all transform kernel candidates belonging to a collection, only some may be selectively rearranged.

[0141] Example 2

[0142] The transform kernel for the inverse transform of the current block may be determined based on multiple transform selection (MTS).

[0143] The MTS according to the present disclosure can use at least one of DST-7, DCT-8, DCT-5, DST-4, DST-1, or IDT (identity transform) as a transform kernel, and may further include a DCT-2 transform kernel.

[0144] In the present disclosure, multiple MTS sets for MTS may be defined. One of the multiple MTS sets may be determined based on the size of the current block and / or the intra prediction mode. For example, when determining one of the MTS sets, 16 transform block sizes may be considered, and for directional modes, symmetry between the transform block shape and the intra prediction mode may be considered. In the case of Wide Angle Intra Prediction (WAIP) modes (i.e., -1 to -14 (or -15), 67 to 80 (or 81)), the MTS set corresponding to mode 2 may be applied to modes -1 to -14 (or -15), and the MTS set corresponding to mode 66 may be applied to modes 67 to 80 (or 81). A separate MTS set may be assigned to a Matrix-based Intra Prediction (MIP) mode.

[0145] For example, the MTS set according to the transform block size and the intra prediction mode may be assigned / defined as shown in Table 4 below.

[0146] [Table 4]

[0147] Table 4 shows the allocation of MTS sets according to 16 transform block sizes and intra prediction modes. The number of predefined MTS sets is 80, and an index indicating one of the 80 MTS sets may range from 0 to 79 as shown in Table 4.

[0148] [Table 5] JPEG2026504649000010.jpg244143JPEG2026504649000011.jpg243142JPEG2026504649000012.jpg113142

[0149] Table 5 shows the transform kernel candidates included in each MTS set described in Table 4. Each MTS set may be composed of six transform kernel candidates. A transform kernel candidate index has a value of 0 to 5 and can indicate one of the six transform kernel candidates. Here, each transform kernel candidate may be a combination of a horizontal transform kernel and a vertical transform kernel for a separation transform, and 25 transform kernel candidates having indices of 0 to 24 may be defined.

[0150] [Table 6] JPEG2026504649000014.jpg52164

[0151] Table 6 shows an example of the 25 transform kernel candidates described in Table 5. Specifically, the horizontal transform and vertical transform of the transform kernel candidate are denoted as (horizontal transform, vertical transform). For each transform kernel candidate index, the horizontal / vertical transform when the intra prediction mode is less than 35 may be opposite to the horizontal / vertical transform when the intra prediction mode is 35 or greater. When the intra prediction mode value is 35 or greater, a mode symmetrical with respect to mode 34 may be derived, and an MTS set may be selected from Table 4 based on the derived mode. Symmetry of the block shape may also be taken into consideration. When an original transform block has a WxH size, the original transform block may be symmetrically considered to have an HxW size, and an MTS set may be selected from Table 4. Here, the value of the intra prediction mode may be the value of the modified intra prediction mode. That is, for the mode values ​​for WAIP, -14 (or -15) to -1 are modified to mode 2, 67 to 80 (or 81) are modified to mode 66, and the original intra prediction mode values ​​for the remaining modes can be set as modified intra prediction mode values. In this case, since the extended modes for WAIP are also configured symmetrically around mode 34, symmetry around mode 34 can be used for all directional modes except for planar mode and DC mode.

[0152] For example, when a 16x32 block is predicted as the 54th mode, the 14th mode (=68-54) is induced as a mode symmetric to the 54th mode, and the block size may be considered as 32x16. In this case, an MTS set having an index of 72 may be selected as defined in Table 4.

[0153] When the MIP mode is applied, the MTS set assigned to the MIP mode may be selected based on the size of the current block without considering the symmetry of the block type. Alternatively, when the MIP mode is applied, the MTS set assigned to the MIP mode may be selected based on the symmetric block size while considering the symmetry of the block type. For example, when the MIP mode is applied to an 8x16 block, the 8x16 block may be regarded as a 16x8 block symmetric to the 8x16 block, and the MTS set having an index of 49 may be selected as defined in Table 4. Alternatively, when the MIP mode is applied, the intra prediction mode may be regarded as the planar mode. In this case, the MTS set assigned to the MIP mode may be selected based on the size of the current block without considering the symmetry of the block type. Alternatively, the MTS set assigned to the MIP mode may be selected based on the symmetric block size while considering the symmetry of the block type.

[0154] In the case of the MIP mode, a flag indicating whether the MIP mode is applied as a transpose mode may be used. When the MIP mode is applied to an MxN current block and the flag indicates the application of the transpose mode, the intra prediction mode may be regarded as the planar mode, and the MxN current block may be regarded as an NxM block. That is, an MTS set corresponding to the NxM block size and the planar mode may be selected from Table 4. As described in Table 6, when the intra prediction mode value is 35 or greater, the horizontal transform and the vertical transform are interchanged. However, since the intra prediction mode of the current block is regarded as the planar mode, the horizontal transform and the vertical transform of the transform kernel candidate do not need to be interchanged. Alternatively, when the MIP mode is applied to an MxN current block and the flag indicates the application of the transpose mode, the intra prediction mode may not be regarded as the planar mode, and the MxN current block may be regarded as an NxM block. That is, an MTS set corresponding to the NxM block size and the MIP mode may be selected from Table 4.

[0155] In Table 5, a transform kernel candidate selected by a transform kernel candidate index may be set as the transform kernel for the current block. Alternatively, at least one of the horizontal transform or vertical transform of the selected transform kernel candidate may be changed to another transform kernel depending on the size of the current block. For example, if the transform kernel candidate index is 3 and both the width and height of the current block are 16 or less, at least one of the horizontal transform or vertical transform of the transform kernel candidate corresponding to the transform kernel candidate index of 3 may be changed to another transform kernel. In this case, the horizontal transform and the vertical transform may be changed independently of each other. If the difference (or the absolute value of the difference) between the intra prediction mode value and the horizontal mode value of the current block is equal to or less than a predetermined threshold, the vertical transform of the selected transform kernel candidate may be changed to IDT (identity transform). If the difference (or the absolute value of the difference) between the intra prediction mode value and the vertical mode value of the current block is equal to or less than a predetermined threshold, the horizontal transform of the selected transform kernel candidate may be changed to IDT (identity transform). Here, the threshold may be determined as shown in Table 7 below based on the width and height of the current block.

[0156] [Table 7]

[0157] Table 7 defines thresholds according to the size of the transform block for changing the horizontal transform and / or vertical transform of the transform kernel candidate selected by the transform kernel candidate index to another transform kernel.

[0158] Six transform kernel candidates constituting one MTS set may be distinguished by transform kernel candidate indices 0 to 5 as defined in Table 5. The transform kernel candidate indices may be signaled by a bitstream. A flag indicating whether an MTS set is available / applied (MTS enabled flag or MTS flag) may be signaled, and if the flag indicates that an MTS set is available / applied, the transform kernel candidate index may be signaled. The MTS flag may be configured as one bin, and one or more contexts for CABAC-based entropy coding (hereinafter referred to as CABAC contexts) may be assigned to the bin. For example, different CABAC contexts may be assigned to a non-MIP mode and an MIP mode, respectively.

[0159] The number of transform kernel candidates available to the current block may be set differently depending on the context of the current block. For example, the sum of absolute values ​​of all or some of the transform coefficients in the current block may be considered as the context of the current block. The sum of absolute values ​​of the transform coefficients is referred to as AbsSum. If AbsSum is less than or equal to T1, only one transform kernel candidate corresponding to a transform kernel candidate index of 0 may be available. If AbsSum is greater than T1 and less than or equal to T2, four transform kernel candidates corresponding to transform kernel candidate indexes of 0 to 3 may be available. If AbsSum is greater than T2, six transform kernel candidates corresponding to transform kernel candidate indexes of 0 to 5 may be available. Here, T1 may be 6 and T2 may be 32, but this is merely an example.

[0160] When AbsSum is less than or equal to T1, the number of transform kernel candidates available for the current block is one, and therefore, the transform kernel candidate corresponding to the transform kernel candidate index of 0 may be set as the transform kernel for the current block without signaling the transform kernel candidate index. When AbsSum is greater than T1 and less than or equal to T2, four transform kernel candidates are available, and therefore, one of the four transform kernel candidates may be selected based on the transform kernel candidate index having two bins. That is, the transform kernel candidate indexes 0 to 3 may be signaled as 00, 01, 10, and 11, respectively. For the two bins, the most significant bit (MSB) may be signaled first, and the least significant bit (LSB) may be signaled later. A different CABAC context may be assigned to each bin. For example, a CABAC context other than the CABAC context assigned for the MTS flag may be assigned to each of the two bins. Alternatively, no CABAC context may be assigned to two bins, and bypass coding may be applied. When AbsSum is greater than T2, the transform kernel candidate indexes have values ​​between 0 and 5, so the transform kernel candidate indexes cannot be expressed using only two bins. In this case, two or more bins may be assigned to express the transform kernel candidate indexes, as in truncated binary coding. A CABAC context may be assigned to each bin assigned by the truncated binary coding scheme, or bypass coding may be applied without assigning a CABAC context. Alternatively, a CABAC context may be assigned to some of the bins (e.g., the first bin or the first and second bins), and bypass coding may be applied to the remaining bins.

[0161] Example 3

[0162] A transformation kernel for a current block may be determined based on a transformation set including one or more transformation kernel candidates, and the transformation kernel for the current block may be derived from any one of the one or more transformation kernel candidates belonging to the transformation set.

[0163] The process of determining a transform kernel for the current block may include at least one of 1) determining a transform set for the current block, or 2) selecting one transform kernel candidate from the transform set for the current block. The process of determining the transform set may be a process of selecting one of a plurality of transform sets predefined identically for the encoding device and the decoding device. Alternatively, the process of determining the transform set may be a process of constructing one or more transform sets available for the current block from a plurality of transform sets predefined identically for the encoding device and the decoding device, and selecting one of the constructed transform sets. Alternatively, the process of determining the transform set may be a process of constructing one transform set based on a transform kernel candidate available for the current block from a plurality of transform kernel candidates predefined identically for the encoding device and the decoding device.

[0164] When a transform set of a current block includes multiple transform kernel candidates, a process of selecting one of the multiple transform kernel candidates for the current block may be performed. However, when the transform set of the current block includes one transform kernel candidate (i.e., when there is one transform kernel candidate available for the current block), the transform kernel of the current block may be set to the selected transform kernel candidate.

[0165] The transform set according to the present disclosure may refer to the (non-separable) transform set in the above-described first embodiment or the MTS set in the second embodiment. Alternatively, the transform set may be defined separately from the (non-separable) transform set in the first embodiment or the MTS set in the second embodiment. In this case, the transform set may include one or more specific transform kernels as transform kernel candidates. One specific transform kernel may be defined as a pair of a transform kernel for a horizontal transform and a transform kernel for a vertical transform, or as one transform kernel that is applied equally to both horizontal and vertical transforms.

[0166] In the embodiments of the present disclosure, a process for applying NSPT, a non-separable transform applied as a primary transform, will be described in detail. NSPT may be applied to all or a portion of a transform block. Based on forward NSPT, residual samples present in the region to which NSPT is applied may be input to a one-dimensional NSPT vector. That is, residual samples present in the entire transform block or a portion of the region (referred to as a Region of Interest, ROI, in the present disclosure) may be collected into a one-dimensional vector and configured as an input. Then, by applying forward NSPT, primary transform coefficients can be obtained. Conversely, by applying backward NSPT to the primary transform coefficients, a one-dimensional vector output can be obtained. By arranging each element value constituting the output vector at a specific position within the 2D transform block, residual samples for the ROI can be obtained.

[0167] The dimensions of the non-separable transformation kernel for NSPT may be determined by the size of the ROI. In this disclosure, the transformation kernel may be referred to as a transformation type or a transformation matrix, and the non-separable transformation kernel for NSPT may be referred to as an NSPT kernel. For example, if the current block is an MxN transformation block, the ROI is the entire region of the MxN transformation block, and square NSPT is applied, the dimensions of the transformation matrix may be MxMn. For example, if the ROI is the entire region of an 8x8 transformation block, the dimensions of the NSPT kernel may be 64x64.

[0168] According to an embodiment of the present disclosure, when NSPT is applied to a residual generated by intra prediction, the NSPT kernel may be adaptively determined according to the intra prediction mode. Because statistical characteristics of the residual block may vary depending on the intra prediction mode, adaptively determining the NSPT kernel according to the intra prediction mode may improve compression efficiency.

[0169] The NSPT kernels applied to one or more intra prediction modes may be configured to be shared. As described above, the non-separable transform set may be determined based on the intra prediction mode of the current block and a mapping table. The mapping table may define a mapping relationship between predefined intra prediction modes and the non-separable transform set. The predefined intra prediction modes may include two non-directional modes and 65 directional modes.

[0170] In one embodiment, intra prediction modes may be grouped into intra prediction mode groups. One NSPT kernel or multiple NSPT kernels may be assigned to an intra prediction mode group. In other words, a non-separable transform set (NSPT set) including one or more NSPT kernels may be assigned to an intra prediction mode group. The non-separable transform set may be mapped to an intra prediction mode, and one of N NSPT kernels included in the non-separable transform set may be selected.

[0171] For example, an intra prediction group may include adjacent prediction modes (e.g., modes 17, 18, and 19). An intra prediction group may also include symmetric modes. For example, directional modes may be symmetric with respect to the diagonal mode (i.e., intra prediction mode 34) in FIG. 5 . In this case, two symmetric modes may form one group (or pair). For example, modes 18 and 50 may be included in the same group because they are symmetric with respect to mode 34. However, for symmetric modes, a process of transposing a 2D input block and then constructing a one-dimensional input vector before applying a forward NSPT kernel may be added. For example, when the intra prediction mode is 34 or less, a one-dimensional input vector may be derived from the 2D input block in a row-first order without transposing the input block. If the intra prediction mode is greater than 34, a one-dimensional input vector can be constructed by first transposing the 2D input block and then reading the input block in row-major order, or by leaving the 2D input block as is and reading the input block in column-first order.

[0172] Table 8 below illustrates an example of an NSPT set allocation mapping table according to intra prediction modes. Referring to Table 8, a total of 35 NSPT sets from 0 to 34 may be defined. The extended WAIP mode (i.e., modes from -14 to -1 and modes from 67 to 80 in FIG. 5) may be assigned the NSPT set assigned to the nearest general directional mode. That is, the extended WAIP mode may be assigned the second NSPT set.

[0173] [Table 8]

[0174] An NSPT set may include one or more NSPT kernels (or kernel candidates). That is, an NSPT set may include N NSPT kernel candidates. As an example, N may be set to a value equal to or greater than 1, such as 1, 2, 3, or 4. A kernel to be applied to a current block among one or more NSPT kernels included in the NSPT set may be signaled using an index. In the present disclosure, the index may be referred to as an NSPT index. As an example, the NSPT index may have a value of 0, 1, 2, ..., N-1.

[0175] Also, as an example, when the number of NSPT kernel candidates is one, the NSPT index value may be fixed to 0. In this case, the NSPT index may be inferred without being separately signaled. Also, a flag indicating whether NSPT is applied may be signaled separately from the NSPT index. In the present disclosure, this flag may be referred to as an NSPT flag.

[0176] If the value of the NSPT flag is 1, NSPT may be applied. If the value of the NSPT flag is 0, NSPT may not be applied. If the NSPT flag is not signaled, the NSPT flag value may be inferred as 0. As an example, if the NSPT flag value is 1, an NSPT index may be signaled. Based on the signaled NSPT index, one of the N kernel candidates included in the NSPT set selected by the intra prediction mode may be identified.

[0177] In one embodiment, the entropy coding method for the NSPT index may be defined as various methods taking into account the number (N) of NSPT kernels included in the NSPT set. For example, truncated unary binarization, truncated binarization, or fixed-length binarization may be used as a method for mapping values ​​from 0 to N-1 to a bin string (i.e., binarization method).

[0178] For example, if the number N of kernel candidates constituting the NSPT set is 2, one of the two candidates can be identified in one bin. For example, 0 can indicate the first candidate and 1 can indicate the second candidate. Also, if N is 3 and truncated unary binarization is applied, candidates can be identified in two bins. For example, the first, second, and third candidates can be binarized to 0, 10, and 11, respectively, and signaled. As an example, the binarized bins can be coded using context coding or bypass coding.

[0179] This disclosure describes a reduced primary transform (RPT) method using a reduced-dimensional transform kernel as the primary transform. As described above, when forward NSPT is applied, samples belonging to a 2D residual block may be arranged (or rearranged) into a 1D vector in row-major order (or column-major order). Then, a transform matrix for NSPT may be multiplied by the arranged vector. When the 2D residual block is an MxN block (M is the width and N is the height), the length of the rearranged 1D vector may be M*N. That is, the 2D residual block can be represented by a column vector having dimensions M*Nx1. In this disclosure, M*N may be denoted as MN for convenience. In this case, the dimensions of the transform matrix may be MNxMN. In short, forward NSPT can operate by multiplying an MNx1 vector by the MNxMN transform matrix on the left side to obtain an MNx1 transform coefficient vector.

[0180] When RPT is applied, instead of multiplying by an MN x MN matrix as in the above-described forward NSPT transform matrix, an r x MN matrix can be multiplied to obtain r transform coefficients. Here, r represents the number of rows in the transform matrix, and MN represents the number of columns in the transform matrix. According to an embodiment of the present disclosure, the value of r may be set to be smaller than or equal to MN. That is, the existing forward NSPT transform matrix includes MN rows, each of which is a 1 x MN row vector and serves as a transform basis vector for the NSPT transform matrix. The transform coefficients may be obtained by multiplying each transform basis vector by an MN x 1 sample column vector.

[0181] Since the existing forward NSPT transform matrix is ​​composed of MN row vectors, MN transform coefficients (i.e., an MNx1 transform coefficient column vector) can be obtained by applying the forward NSPT. On the other hand, in the case of the forward RPT, the transform matrix may be composed of r transform basis vectors instead of MN transform basis vectors. As a result, when the forward RPT is applied, r transform coefficients (i.e., an rx1 transform coefficient column vector) can be obtained instead of MN.

[0182] An RPT kernel can be constructed by selecting r transform basis vectors, which are part of the transform basis vectors constituting the MNxMN forward NSPT kernel. In this disclosure, a transform kernel may be referred to as a transform type or a transform matrix, and a non-separable transform kernel for NSPT may be referred to as an RPT kernel. That is, when selecting r 1xMN row vectors from the MNxMN forward NSPT kernel, it may be advantageous to select the most important transform basis vectors in terms of coding performance. Specifically, in terms of energy compaction through transform, more energy may be concentrated in transform coefficients that appear earlier after multiplication by the forward NSPT transform matrix. In other words, the higher the transform basis vector located in the forward NSPT transform matrix, the more transform coefficients with greater energy can be generated. Taking this into consideration, an rxMN forward RPT kernel can be constructed (or derived) by selecting r transform basis vectors from the top of the forward NSPT kernel.

[0183] The RPT according to the present disclosure only extracts a portion (i.e., r) of the transform coefficients obtained by applying the existing NSPT, which may result in a loss of some of the energy contained in the original signal. That is, this process may cause distortion between the original signal and the original signal. Nevertheless, by applying the RPT, only r transform coefficients are generated instead of MN transform coefficients, thereby reducing the number of bits required to code the transform coefficients. Therefore, in the case of a signal in which a large amount of energy is concentrated in a small number of transform coefficients (e.g., an image residual signal), the gain obtained by reducing the signaling bits is significantly large, thereby improving coding performance.

[0184] The inverse NSPT is a transformation matrix, which may be the transpose matrix of the forward NSPT kernel described above. In this case, the input data may be a transform coefficient signal instead of a sample signal such as a residual signal. Specifically, if the forward NSPT transformation matrix is ​​G and the sample signal rearranged into a 1D vector is x, the transform coefficient vector obtained by multiplying the transformation matrix on the left side may be expressed as follows:

[0185]

number

[0186] Referring to Equation 4, x and y may be M N x 1 column vectors. G may have the form of an M N x M N matrix. The backward NSPT process may be expressed as Equation 5 using the same variables.

[0187]

number

[0188] In Equation 5, G Tmeans the transpose matrix of G. The forward RPT operation and the backward RPT operation according to the present disclosure may also be expressed by the above two equations. However, when RPT is applied, y is an rx1 column vector instead of an Mx1 column vector, and G is an rxMN matrix instead of an MxMN matrix. That is, the dimension of the sample signal (e.g., image residual signal) does not change even when RPT is applied instead of NSPT. This means that the original number of sample signals (i.e., MN sample signals) can be restored using only r transform coefficients through backward RPT. That is, the original MN sample signals can be restored by coding only r transform coefficients, which is less than MN, thereby improving coding performance.

[0189] In one embodiment of the present disclosure, we propose an RPT structure that defines an r value taking into account the statistical characteristics of a residual block and derives a residual block of an existing transform block size from a residual block of reduced size determined by the defined r value. If an additional transform (i.e., a secondary transform) is applied to predict the statistical distribution of primary transform coefficients, a quantization process is applied to the primary transform coefficients, resulting in a concentration of quantized non-zero coefficients in a relatively low frequency region. Therefore, a reduced secondary transform for the statistical distribution of primary transform coefficients can relatively easily define the statistical characteristics of the primary transform coefficients by setting an r value for a given low frequency region. However, the RPT according to the present disclosure is a technology for defining an r value taking into account the statistical characteristics of samples in a residual block, which have characteristics very different from the distribution of primary transform coefficients, and thus is fundamentally different from a reduced secondary transform. Various embodiments for determining an RPT kernel, which is a transform matrix of reduced dimensions, will be described below. In other words, a method for determining or defining the r value in an RPT will be described below.

[0190] In one embodiment of the present disclosure, the value of r in the RPT may be determined taking into account the worst-case complexity allowed by the conversion system. As an example, the worst-case complexity may be calculated based on the number of multiplications per sample. Applying RPT in both the forward and backward directions to an MxN block requires MN*r multiplications. Since a 2D block consists of a total of MN samples, the number of multiplications per sample may be calculated as (MN*r) / MN=r. Therefore, the value of r may be configured to be less than or equal to the maximum number of multiplications per sample allowed. For example, if the maximum number of multiplications per sample for a 16x16 block is set to 16, the value of r may be determined to be less than or equal to 16. That is, the forward RPT kernel may be set to 16x256.

[0191] In another embodiment, memory usage can be considered as a measure of worst-case complexity. As an example, the memory size allowed per kernel can be set. For example, if p bytes are required per kernel coefficient (in this disclosure, each element constituting a transform kernel is referred to as a kernel coefficient), and memory usage is set to q bytes or less per kernel, then the value of r can be set to q / (MN*p) or less. For example, if p is 1 byte for a forward RPT kernel for a 16x16 block, and memory usage is set to 8 KB or less per kernel (q=8KB=2 13 bytes), the r value may be set to 32 or less.

[0192] As another example, memory usage and / or the number of multiplications per sample may be considered as a measure of worst-case complexity. For example, if the maximum possible number of multiplications per sample for a 16x16 block is set to 16 and the memory usage is set to 8 KB or less per kernel (kernel coefficients are represented in 1 byte), then the r value may be set to 16 or less.

[0193] In one embodiment, the r value constituting the RPT kernel may be determined based on specific information. In other words, the r value constituting the RPT kernel may be determined based on predefined coding parameters. For example, the r value may be determined based on the size of a block. In other words, the RPT kernel may be variably determined based on the size of a block. Here, the block may be at least one of a coding block, a transform block, and a prediction block. Furthermore, the r value may be determined based on prediction information. Here, the prediction information may include information regarding inter / intra prediction, intra prediction mode information, etc. Furthermore, the r value may be determined based on signaled information (syntax element values). For example, the r value may be variably determined based on a quantization parameter value. Furthermore, in terms of complexity improvement, a predefined fixed value may be used as the r value, and the predefined fixed value may be determined based on the signaled information.

[0194] By multiplying the sample signal by the RPT kernel rxMN, r transform coefficients can be obtained. The obtained r transform coefficients may be arranged according to a predefined scan order of transform coefficients (e.g., forward / reverse zig-zag scan order, forward / reverse horizontal scan order, forward / reverse vertical scan order, forward / reverse diagonal scan order, scan order determined based on an intra prediction mode, etc.). When the transform coefficients obtained by applying the forward RPT are arranged according to such a scan order (e.g., a scan order based on coefficient groups (CGs) may also be applied), if the value of r is smaller than MN, the r transform coefficients may not fill the entire MxN block, resulting in empty spaces. In an embodiment of the present disclosure, the above-mentioned empty spaces may be predicted in the following manner, taking into account characteristics of the residual signal.

[0195] - Values ​​of available surrounding pixels can be used to fill in the empty spaces.

[0196] The values ​​of the empty spaces can be filled based on the values ​​of available surrounding pixels and the intra prediction mode, for example, intra prediction can be performed to predict the values ​​of the empty spaces based on the values ​​of available surrounding pixels and the intra prediction mode.

[0197] - You can fill empty spaces with a predefined fixed value (e.g., 0).

[0198] A predetermined intra prediction mode (eg, planar mode) can be used to fill the empty space values ​​from available surrounding pixels.

[0199] In the present disclosure, among the examples described above, filling empty spaces with zeros may be referred to as a zero-out process. When filling empty spaces with zeros, the following embodiment may be applied. When a non-zero transform coefficient is detected (or parsed) in a corresponding empty space during passing of transform coefficients at the decoding device side, it may be considered (or inferred) that RPT is not applied. In other words, when a non-zero transform coefficient exists in a predefined region representing the corresponding empty space, it may be considered that RPT is not applied. In this case, a flag indicating whether RPT is applied and / or an index specifying one of multiple RPT kernel candidates is not signaled (or parsed). For example, when a non-zero transform coefficient exists in the predefined region representing the corresponding empty space, a predefined variable value may be updated, and it may be inferred that RPT is not applied based on the updated variable value.

[0200] In one embodiment of the present disclosure, whether or not to apply RPT may be determined depending on the size and / or shape of a block. Furthermore, the RPT kernel may be variably determined depending on the size and / or shape of a block. Because the r value may differ depending on the size and / or shape of a block (i.e., for each MxN block), the free space may vary depending on the size and / or shape of the block. Accordingly, the region for checking whether a non-zero transform coefficient is detected may be defined differently depending on the size and / or shape of the block. In other words, the zero-out region may be variably determined.

[0201] For example, when a 16x64 matrix is ​​applied as a forward RPT matrix for an 8x8 block, the r value may be 16. In this case, when the CG is a 4x4 sub-block, non-zero RPT transform coefficients may be filled only in the upper left 4x4 block, and zero values ​​may be filled in the remaining three 4x4 sub-blocks (i.e., the upper right, lower left, and lower right sub-blocks) that are empty spaces. In this case, if non-zero transform coefficients are detected in the remaining three 4x4 sub-block areas during the decoding process, it can be determined that RPT is not applied. Also, as described above, a flag indicating whether RPT is applied or an index specifying one of multiple RPT kernel candidates may not be signaled.

[0202] As another example, when a 32x128 matrix is ​​applied as the forward RPT matrix for a 16x8 block (i.e., r is 32) and the CGs are 4x4 sub-blocks, non-zero RPT coefficients may be embedded in only two CGs in the scan order. For example, the RPT coefficients may be embedded in the top-left 4x4 sub-block and the 4x4 sub-block adjacent to the bottom of the top-left sub-block. The remaining area other than the two 4x4 sub-blocks may be determined as the empty space to be filled with zeros. The RPT kernel may be variably determined depending on the size and / or shape of the block, and as described above, the empty space may be determined differently for 8x8 blocks and 16x8 blocks.

[0203] In one embodiment, when the r value is a multiple of the CG size and transform coefficients are scanned in CG units, if a non-zero transform coefficient is detected in a CG belonging to an empty space, a flag and / or index related to the RPT may not be signaled. That is, for each CG, the CG internal transform coefficients may be scanned in a specified order, and then the next CG may be moved to according to the scan order for the CG unit, and the CG internal transform coefficients may be scanned in the same way. In conventional image compression techniques, a flag indicating whether a non-zero transform coefficient exists in each CG is first signaled, so that whether or not an RPT is applied may be determined based on this information alone, thereby reducing signaling overhead and associated implementation complexity.

[0204] As described above, when RPT is applied, if a non-zero transform coefficient is detected in an empty spatial region to be filled with zeros, RPT may not be applied. In this case, signaling of information related to RPT may be omitted. However, if a non-zero transform coefficient is not detected in the corresponding empty spatial region, it is not possible to determine whether RPT is applied. Therefore, after parsing (or signaling) the relevant transform coefficient, a flag indicating whether RPT is applied can be parsed to finally determine whether RPT is applied.

[0205] As an example, a forward quadratic transform may be further applied to the transform coefficients generated by applying the RPT. Alternatively, a forward quadratic transform may be further applied to a region within an MxN block where the generated transform coefficients are located. In this disclosure, the region or a portion of the region may be referred to as a ROI in terms of the forward quadratic transform. For the inverse direction, an inverse quadratic transform may be first applied, and then the inverse RPT may be applied. Specifically, a region or a portion of the region where the r transform coefficients generated by applying the forward RPT are located may be set as an ROI, and a forward quadratic transform may be applied to the region. In this case, when a 16x64 forward RPT transform matrix is ​​applied to an 8x8 region, the generated 16 transform coefficients may be located in the upper left 4x4 sub-block, and the sub-block region may be set as an ROI, and a forward quadratic transform may be applied to the ROI.

[0206] Furthermore, the coefficient values ​​of the RPT kernel may be adjusted taking into account operations such as integer arithmetic or fixed-point arithmetic. That is, the RPT kernel can be configured to appropriately scale the kernel coefficients belonging to the kernel, rather than a theoretical orthogonal or non-orthogonal transform (here, orthogonal transform and non-orthogonal transform refer to transforms in which the norm of each transform basis vector is 1), to perform transforms using integer arithmetic (or fixed-point arithmetic) in an actual codec system. The scaling factor applied when applying a separable transform in existing image compression technology can be reflected in the same way when applying RPT. In this case, a separable transform or non-separable transform (including RPT) can be performed while maintaining processes other than the transform (e.g., quantization and inverse quantization processes).

[0207] The integerized coefficients of the RPT kernel can be obtained by multiplying the transformation basis vectors by the scaling factor. In one embodiment, multiplying by the scaling factor may involve applying an operation such as rounding, rounding down, or rounding up to each kernel coefficient. That is, the integerized RPT kernel obtained by the above method may be defined and used in the transformation / inverse transformation process. As described above, by obtaining the scaled integer kernel coefficients through operations such as rounding, rounding down, or rounding up, maximum and minimum values ​​can be determined for all kernel coefficients. Therefore, the number of bits required to represent all kernel coefficients can be obtained from the maximum and minimum values. For example, if the maximum value is 127 or less and the minimum value is -128 or greater, all integer kernel coefficients can be represented using 8 bits (e.g., using two's complement representation).

[0208] In general, the maximum value is (2 (N-1) -1), and the minimum value is -2 (N-1) If the maximum value is (2 (N-1) -1) or the minimum is -2 (N-1)If the number of bits is smaller than N, it may not be possible to represent all integer kernel coefficients with N bits. In such cases, 1) all kernel coefficients can be further multiplied by a scaling factor to fit within the N-bit range, or 2) the number of bits required to represent the kernel coefficients can be increased (i.e., N+1 bits or more). Suppose all kernel coefficients are multiplied by 2 to be represented with N bits. -p If only p >= 1, then we integrate it into the existing encoding / decoding process. p As an example, p The multiplication may be implemented by further shifting the data to the left by p bits, or by reducing the amount of right shift applied in the quantization or dequantization process by p bits.

[0209] Using the method described above, all kernel coefficients can be expressed in 8 bits, 9 bits, 10 bits, etc., and of course, the scaling values ​​of the kernel coefficients may be set differently for each block size or kernel, and the number of bits for expressing the kernel coefficients may be set differently.

[0210] The above-mentioned NSPT may be applied based on at least one of the size, tree type, or component type of the current block. For example, whether or not to apply NSPT may be determined based on at least one of the size, tree type, or component type of the current block. An NSPT index may be signaled based on at least one of the size, tree type, or component type of the current block. An NSPT set or an NSPT kernel may be derived based on at least one of the size, tree type, or component type of the current block.

[0211] Allowed transform block sizes predefined in the decoding device can be roughly divided into two groups. One of the two groups (hereinafter referred to as the first group) may refer to a set of block sizes to which NSPT can be applied. The first group may consist of any one of the allowable transform block sizes, or may consist of two or more of the allowable block sizes. The block sizes to which NSPT can be applied may be defined as block sizes in which at least one of width and height is smaller than or equal to a predetermined threshold. Alternatively, the block sizes to which NSPT can be applied may be defined as block sizes in which the product of width and height is smaller than or equal to a predetermined threshold. Alternatively, the block sizes to which NSPT can be applied may be defined as block sizes in which the maximum value of width and height is smaller than or equal to a predetermined threshold. The threshold may be an integer of 4, 8, 16, 32, 64, 128, or more.

[0212] The other of the two groups (hereinafter referred to as the second group) may refer to a set of block sizes to which NSPT is not applied. The aforementioned separable linear transform may be applied to the block sizes belonging to the second group. Also, a non-separable secondary transform may be applied to all or some of the block sizes belonging to the second group.

[0213] For example, if the size of the current block belongs to the first group, an inverse NSPT may be applied to the (dequantized) transform coefficients of the current block. If the size of the current block belongs to the second group, an inverse separable linear transform may be applied to the (dequantized) transform coefficients of the current block. Alternatively, if the size of the current block belongs to the second group, an inverse non-separable linear transform (e.g., low frequency non-separable transform (LFNST)) may be first applied to the (dequantized) transform coefficients of the current block, and then an inverse separable linear transform (e.g., DCT-2) may be applied to the resulting transform coefficients.

[0214] As an example, a first group, which is a set of block sizes to which NSPT can be applied, may be defined as 4x4, 4x8, 8x4, and 8x8. Alternatively, the first group may be defined as 4x8, 8x4, and 8x8. Alternatively, the first group may be defined as 4x8 and 8x4. Alternatively, the first group may be defined as 4x4, 4x8, 4x16, 8x4, 8x8, and 16x4. Alternatively, the first group may be defined as 4x8, 4x16, 8x4, 8x8, and 16x4. Alternatively, the first group may be defined as 4x8, 4x16, 8x4, and 16x4. Alternatively, the first group may be defined as 4x8, 4x16, 8x4, and 16x4. Alternatively, the first group may be defined as 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, and 16x16. Alternatively, the first group may be defined as a set of 4x4, 4x8, 8x4, 8x8, 8x16, and 16x8. Alternatively, the first group may be defined as a set of 4x8, 8x4, 8x8, 8x16, and 16x8. Alternatively, the first group may be defined as a set of 4x8, 8x4, 8x16, and 16x8. Alternatively, the first group may be defined as a set of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, 32x16, and 32x32. Alternatively, the first group may be defined as a set of 4x4, 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as a set of 4x8, 8x4, 8x8, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as a set of 4x8, 8x4, 8x16, 16x8, 16x16, 16x32, and 32x16. Alternatively, the first group may be defined as a set of 4x8, 8x4, 8x16, 16x8, 16x32, and 32x16. Alternatively, the first group may be defined as a set of 4x4, 4x8, 4x16, 8x4, and 16x4. Alternatively, the first group may be defined as a set of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as a set of 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as a set of 4x4, 4x8, 4x16, 8x4, 8x16, 16x4, and 16x8.Alternatively, the first group may be defined as a set of 4x8, 4x16, 8x4, 8x16, 16x4, and 16x8. Alternatively, the first group may be defined as a set of 4x4, 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as a set of 4x8, 4x16, 8x4, 8x8, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as a set of 4x4, 4x8, 4x16, 8x4, 8x16, 16x4, 16x8, and 16x16. Alternatively, the first group may be defined as a set of 4x8, 4x16, 8x4, 8x16, 16x4, 16x8, and 16x16.

[0215] As in the above example, NSPT may be applied to non-square MxN and NxM blocks. For example, NSPT may be applied to 4x8 and 8x4 blocks. Alternatively, NSPT may be applied to 4x16 and 16x4 blocks, 8x16 and 16x8 blocks, or 16x32 and 32x16 blocks.

[0216] Applying NSPT to specific block sizes belonging to the first group can improve coding performance by performing more accurate transformations. When applying forward LFNST, linearly transformed transform coefficients of remaining regions other than the region to which LFNST is applied (i.e., the Region-Of-Interest, ROI) may be zeroed out. Furthermore, LFNST may be configured with a small number of transform basis vectors. In such cases, applying a separable linear transform such as DCT-2 and a non-separable quadratic transform such as LFNST instead of NSPT to the block size may result in performance degradation. In such cases, applying NSPT instead of LFNST omits the zero-out process, thereby improving coding performance compared to applying LFNST. Performance improvement can also be expected depending on the method of applying NSPT. NSPT or LFNST may be applied using symmetry, as described below. Here, LFNST performs a transpose operation on the input block using symmetry only for the ROI region. In contrast, NSPT performs a transposition operation using symmetry across the entire block. Therefore, NSPT can train and apply the NSPT kernel using the symmetry of a more accurate method, which is expected to improve performance.

[0217] Furthermore, when applying LFNST instead of NSPT to an 8x8 block, a 32x64 transform matrix can be applied instead of a 16x64 transform matrix in terms of forward transform. Here, the 16x64 transform matrix may be constructed by sampling the top 16 rows of a 32x64 transform matrix. When applying LFNST based on a 16x64 transform matrix to an 8x8 block, 16 multiplications per sample are required to apply LFNST, whereas when using a 32x64 transform matrix, 32 multiplications per sample are required to apply LFNST. However, using a 32x64 transform matrix in this way is expected to improve coding performance.

[0218] If the tree type of the current block is a single tree, NSPT is applied to the luma component of the current block, but NSPT does not need to be applied to the chroma component of the current block. If the tree type of the current block is a dual tree, NSPT can be applied to the luma and chroma components of the current block.

[0219] Alternatively, regardless of whether the tree type of the current block is a single tree, NSPT may be applied to the luma component of the current block, but not to the chroma component of the current block. Alternatively, regardless of whether the tree type of the current block is a single tree, NSPT may be applied to both the luma component and the chroma component of the current block.

[0220] For example, if the tree type of the current block is single tree, NSPT is allowed for the luma component and the chroma component, and the size of the current block belongs to the first group, one NSPT index may be signaled, and the luma component and the chroma component of the current block may share the NSPT index. Here, the NSPT index may be an index for selecting one of the transform kernel candidates for NSPT. If the sizes of the luma block and the chroma block of the current block belong to the first group, the transform kernel candidate selected by the same NSPT index may be applied to the luma component and the chroma component. If the tree type of the current block is single tree and NSPT is applied only to the luma component, LFNST may not be applied to the chroma component of the current block, and a separate transform may be applied. Alternatively, if the tree type of the current block is single tree and NSPT is applied only to the luma component, LFNST may be applied to the chroma component of the current block.

[0221] In the case of a single tree, the luma component and the chroma component may have a high correlation characteristic. In this case, unnecessary signaling can be reduced and compression efficiency can be improved by applying NSPT only to the luma component or by applying a transform kernel candidate selected by one NSPT index to both the luma component and the chroma component. On the other hand, in the case of a non-single tree, the luma component and the chroma component each have their own division and coding structure. In this case, signaling an NSPT index for each component can reflect the characteristics of each component and improve compression efficiency.

[0222] The NSPT kernel for the NSPT may be derived based on at least one of symmetry between intra-prediction modes or symmetry between block types. For example, the NSPT kernel may be derived as an NSPT kernel corresponding to at least one of a mode symmetrical to the intra-prediction mode of the current block or a block type symmetrical to the block type of the current block. Alternatively, the NSPT kernel may be derived based on an NSPT set including one or more NSPT kernel candidates, where the NSPT set may be derived as an NSPT set corresponding to at least one of a mode symmetrical to the intra-prediction mode of the current block or a block type symmetrical to the block type of the current block. Any one of the one or more NSPT kernel candidates belonging to the NSPT set may be set as the NSPT kernel for the current block. For this purpose, an NSPT index identifying any one of the one or more NSPT kernel candidates belonging to the NSPT set may be used. The NSPT index may be signaled in the bitstream or may be derived based on the above-described symmetry.

[0223] Symmetry may exist between at least two of the intra prediction modes predefined in the decoding device. Hereinafter, for convenience of explanation, symmetry will be described around the top left diagonal mode (i.e., mode 34). Referring to FIG. 5, symmetry exists between the directional modes. All modes have a prediction direction except for planar mode 0 and DC mode 1. Modes 2 to 6 are named normal directional modes (which can be expressed as [2,66]), and modes -14 to -1 (which can be expressed as [-14,-1]) and modes 67 to 80 (which can be expressed as [67,80]) are named wide directional modes. The wide directional modes may include at least one of a mode having a value less than -14 or a mode having a value greater than 80. Referring to FIG. 5, all modes except for modes 0 and 1 are symmetric around mode 34. Specifically, the x-th and (68-x)-th modes are symmetric with respect to the [2,66] mode, and the x-th and (66-x)-th modes are symmetric with respect to the [-14,-1] and [67,80] modes. A similar symmetry relationship can be established between the [N,-1] and [67,66-N] modes, where N can be an integer less than or equal to -14.

[0224] Meanwhile, regarding the symmetry between the block shapes, an MxN block and an NxM block may be defined as blocks having symmetry with each other. Here, M and N may be the same or different. Alternatively, if the width-to-height ratio (M1 / N1) of an M1xN1 block and the height-to-width ratio (N2 / M2) of an M2xN2 block are the same, the M1xN1 block and the M2xN2 block may be defined as blocks having symmetry with each other. Alternatively, if the width-to-height ratio (M1 / N1) of an M1xN1 block and the width-to-height ratio (M2 / N2) of an M2xN2 block are the same, the M1xN1 block and the M2xN2 block may be defined as blocks having symmetry with each other.

[0225] In a square block, modes that are symmetric to each other may share at least one of an NSPT set, an NSPT index, or an NSPT kernel, i.e., at least one of an NSPT set, an NSPT index, or an NSPT kernel for one of the symmetric modes may be applied identically to any other of the symmetric modes.

[0226] For example, two symmetrical modes may share a single NSPT kernel. However, for one of the symmetrical modes, the NSPT kernel may be applied to the input data, and for the other, the NSPT kernel may be applied after applying a transpose operation to the input data. Specifically, if the xth mode belongs to the [2,33] mode, for the xth mode, a one-dimensional vector (1D vector) may be constructed for the MxM block of input data in row-major order, and the NSPT kernel may be applied to the one-dimensional vector. Here, constructing a one-dimensional vector in column-major order may involve reading the input data column by column from the MxM block of input data, obtaining M columns, and sequentially arranging these columns to construct a one-dimensional vector. On the other hand, for the (68-x)th mode, which is symmetrical to the xth mode, a one-dimensional vector may be constructed in column-major order, and the same NSPT kernel may be applied to the one-dimensional vector. Here, constructing a one-dimensional vector in row-major order may involve reading input data row by row from an MxM block of input data, obtaining M rows, and sequentially arranging the M rows to construct a one-dimensional vector. If the xth mode belongs to the [N,-1] mode (N≦−14), a one-dimensional vector may be constructed in row-major order for the (66-x) mode symmetrical to the xth mode, and the same NSPT kernel as for the xth mode may be applied to the one-dimensional vector. Column-major order or row-major order may be applied to the 0th and 1st modes, and column-major order or row-major order may also be applied to the 34th mode. Furthermore, row-major order may be applied to intra prediction modes belonging to the [2,33] mode, and column-major order may be applied to modes symmetrical to the intra prediction mode. Row-major order may be applied to intra prediction modes belonging to the [N,-1] mode, and column-major order may be applied to modes symmetrical to the intra prediction mode.

[0227] For non-square blocks, in addition to symmetry between intra prediction modes, symmetry between block shapes may also be considered. A non-square block with width and height M and N, respectively, may be considered to have a symmetric relationship with a non-square block with width and height N and M, respectively. For example, in a [2,66] mode, symmetry may exist between the xth mode of an MxN block and the (68-x)th mode of an NxM block. Similarly, if the xth mode of an MxN block belongs to the [N,-1] mode (N≦-14), symmetry may exist between the xth mode of an MxN block and the (66-x)th mode of an NxM block.

[0228] The method of constructing a one-dimensional vector from an input data block is as described above. That is, if column-major order is applied to the xth mode, row-major order may be applied to the symmetrical mode. Alternatively, if row-major order is applied to the xth mode, column-major order may be applied to the symmetrical mode. Specifically, if column-major order is applied to the xth mode, input data may be read column-by-column from an MxN block of input data to obtain M columns, which may be sequentially arranged to construct a one-dimensional vector. Here, each column may have a length of N. For a mode symmetrical to the xth mode, input data may be read row-by-row from an MxN block of input data to obtain N rows, which may be sequentially arranged to construct a one-dimensional vector. Here, each row may have a length of M. Alternatively, if row-major order is applied to the xth mode, input data may be read row-by-row from an MxN block of input data to obtain N rows, which may be sequentially arranged to construct a one-dimensional vector. Here, each row may have a length of M. For a mode symmetric to the xth mode, input data is read column by column from an MxN block of input data to obtain M columns, which can then be sequentially arranged to form a one-dimensional vector, where each column may have a length of N.

[0229] When the current block is an MxN block having an xth mode and the symmetry is used for the current block, the NSPT set and / or NSPT kernel for the current block may be determined based on at least one of an intra prediction mode symmetrical to the xth mode or an NxM block size symmetrical to the MxN block size. Here, the NSPT kernel may be set as an NSPT kernel for an NxM block rather than an NSPT kernel for an MxN block. That is, when symmetry is used for the current block, the NSPT sets and / or NSPT kernels for blocks symmetrical to the current block may be used in the same way. As described above, a one-dimensional vector may be constructed from an input data block according to a predetermined priority order, which may correspond to the input of the NSPT kernel.

[0230] In addition, the symmetry may be limited to be used only when the intra prediction mode value of the current block is greater than 34. That is, when the intra prediction mode value of the current block is greater than 34, a transpose operation may be applied when constructing a one-dimensional vector from the input data block, and an NSPT set or NSPT kernel corresponding to a mode and / or block type that is symmetric with the current block may be used. Specifically, when the intra prediction mode of the current block belongs to the [N, -1] mode or the [2, 34] mode, symmetry may not be used for the current block. On the other hand, when the intra prediction mode of the current block belongs to the [35, 66] mode or the [67, 66-N] mode, symmetry may be used for the current block. Here, N may be an integer less than or equal to -14.

[0231] The symmetry-based derivation of the NSPT set or NSPT kernel may be adaptively performed based on the size of the current block. For example, the symmetry-based derivation of the NSPT set or NSPT kernel may be performed for a 4x4 block and an 8x8 block, and the symmetry-based derivation of the NSPT set or NSPT kernel may not be performed for a 4x8 block and an 8x4 block.

[0232] The number of available NSPT sets may vary depending on whether symmetry is used. For example, when symmetry is used, the number of available NSPT sets may be 35, and when symmetry is not used, the number of available NSPT sets may be 67.

[0233] Table 9 below relates to an example in which an NSPT set is determined using symmetry, and shows the mapping relationship between intra prediction modes and NSPT sets when the number of available NSPT sets is 35.

[0234] [Table 9]

[0235] Referring to Table 9, if the value (X) of the intra prediction mode of the current block is less than 0, the NSPT set of the current block may be determined to be the NSPT set having an NSPT set index of 2 among the 35 NSPT sets. If the value (X) of the intra prediction mode of the current block is greater than or equal to 0 and less than or equal to 34, the NSPT set of the current block may be determined to be the NSPT set having an NSPT set index of X among the 35 NSPT sets. If the value (X) of the intra prediction mode of the current block is greater than or equal to 35 and less than or equal to 66, the NSPT set of the current block may be determined to be the NSPT set having an NSPT set index of (68-X) among the 35 NSPT sets. If the value (X) of the intra prediction mode of the current block is greater than or equal to 35 and less than or equal to 66, the NSPT set of the current block may be the same as the NSPT set corresponding to the value (68-X) of the mode symmetric to the intra prediction mode of the current block. Similarly, if the value (X) of the intra prediction mode of the current block is greater than 66, the NSPT set of the current block may be determined to be an NSPT set having an NSPT set index of 2 among the 35 NSPT sets. If the value (X) of the intra prediction mode of the current block is greater than 66, the NSPT set of the current block may be the same as the NSPT set corresponding to the mode symmetrical to the intra prediction mode of the current block.

[0236] Table 10 below relates to an example in which the NSPT set is determined without using symmetry, and shows the mapping relationship between the intra prediction modes and the NSPT sets when the number of available NSPT sets is 67.

[0237] [Table 10]

[0238] Referring to Table 10, if the value (X) of the intra prediction mode of the current block is less than 0, the NSPT set of the current block may be determined to be the NSPT set having an NSPT set index of 2 among the 67 NSPT sets. If the value (X) of the intra prediction mode of the current block is greater than or equal to 0 and less than or equal to 66, the NSPT set of the current block may be determined to be the NSPT set having an NSPT set index of X among the 67 NSPT sets. If the value (X) of the intra prediction mode of the current block is greater than 66, the NSPT set of the current block may be determined to be the NSPT set having an NSPT set index of 66 among the 67 NSPT sets.

[0239] By utilizing the symmetry, the memory size required to store the transformation kernels can be reduced while maintaining the performance of the transformation application. For example, if 35 NSPT sets are used instead of 67 NSPT sets by utilizing the symmetry, the memory size required to store the NSPT kernels can be significantly reduced.

[0240] The number of available NSPT sets and / or the number of NSPT kernel candidates belonging to an NSPT set may vary depending on the block size. For example, the number of available NSPT sets for a 4x4 block may be 35, the number of available NSPT sets for a 4x8 block and an 8x4 block may be 19, and the number of available NSPT sets for an 8x8 block may be 10. The NSPT set for a 4x4 block may be composed of three NSPT kernel candidates, the NSPT sets for a 4x8 block and an 8x4 block may be composed of three or two NSPT kernel candidates, and the NSPT set for an 8x8 block may be composed of one NSPT kernel candidate.

[0241] As the block size increases, the size of the transform kernel may increase. Therefore, by reducing the number of available NSPT sets and / or the number of NSPT kernel candidates belonging to the NSPT set, the memory size required to store the transform kernel can be reduced. In addition, as the block size increases, the residual signal characteristics within the block tend to become more generalized. Therefore, reducing the number of available NSPT sets and / or the number of NSPT kernel candidates belonging to the NSPT set reflects such statistical characteristics and helps maintain compression efficiency while reducing implementation complexity.

[0242] The NSPT kernel may be configured with 8-bit precision. The range of coefficients in the NSPT kernel may be greater than or equal to -128 and less than or equal to 127. If the precision is increased beyond 8 bits, the result value obtained by matrix multiplication can be shifted right by the increased precision. For example, if the value obtained after matrix multiplication based on an NSPT kernel with 8-bit precision is shifted right by S bits and stored in a buffer, if the kernel coefficients are configured with N-bit precision, they can be shifted right by (S + (N - 8)) bits and stored in the buffer.

[0243] When the NSPT kernel is configured with 8-bit precision, excessive increase in internal precision within the codec performing the conversion can be prevented, and the implementation complexity in terms of memory requirements and computational complexity can be reduced while minimizing the decline in compression efficiency.

[0244] When applying backward NSPT to a current block of size MxN, the size of the NSPT kernel (or NSPT matrix) may be expressed as M×r. Here, M may refer to the product of the width and height of the current block. This may refer to the output length of the NSPT or the number of residual samples generated by the NSPT. Also, r may refer to the input length of the NSPT or the number of (dequantized) transform coefficients to which the NSPT is applied. r may be an integer greater than or equal to 0 and less than or equal to M. The following is an example of an M×r NSPT matrix depending on the block size.

[0245] The NSPT matrix for 4x4 blocks may be a 16x16 matrix. The NSPT matrix for 4x8 and 8x4 blocks may be a 32x20, 32x16, 32x24, 32x28, or 32x32 matrix. The NSPT matrix for 8x8 blocks may be a 64x16, 64x24, 64x32, 64x40, 64x48, 64x56, or 64x64 matrix. The NSPT matrix for 4x16 and 16x4 blocks may be a 64x16, 64x24, 64x32, 64x40, 64x48, 64x56, or 64x64 matrix. The NSPT matrix for 8x16 and 16x8 blocks may be a 128x96 matrix, a 128x64 matrix, a 128x48 matrix, or a 128x32 matrix. The NSPT matrix for 16x16 blocks may be a 256x128 matrix, a 256x96 matrix, or a 256x64 matrix. The NSPT matrix for 16x32 and 32x16 blocks may be a 512x256 matrix or a 512x128 matrix. The NSPT matrix for 32x32 blocks may be a 1024x512 matrix, a 1024x256 matrix, or a 1024x128 matrix.

[0246] Alternatively, a 16x16 matrix may be applied to 4xN and Nx4 blocks, where N may be an integer greater than or equal to 4; a 64x16 matrix may be applied to 8x8 blocks; a 64x32 matrix may be applied to 8xN and Nx8 blocks, where N may be an integer greater than or equal to 16; and a 96x32 matrix may be applied to 16xN and Nx16 blocks, where N may be an integer greater than or equal to 16.

[0247] Alternatively, the value of r in the MNxr NSPT matrix can be determined according to predetermined criteria, such as (1) ensuring that the sum of the computational complexity for the linear transformation and the computational complexity for the secondary transformation is below a certain level, and (2) ensuring that the number of multiplications per sample required for the NSPT calculation is below a certain number.

[0248] When performing a separate linear transform on an MxN block using matrix multiplication based on the inverse transform, (M+N) multiplications are required per sample to perform the linear transform. Also, when applying LFNST to a certain ROI (Region-Of-Interest) region, assuming that the LFNST matrix for the inverse transform is a PxQ matrix, (P*Q) / (M*N) multiplications are required per sample. Here, a PxQ matrix can refer to a matrix with P columns and Q rows.

[0249] When applying NSPT to an MxN block instead of applying a DCT-2 transform (or a separate transform such as KLT) and an LFNST, the value of r that makes the number of multiplications per sample when applying the NSPT smaller than or equal to the number of multiplications per sample when applying a DCT-2 transform and an LFNST may be determined as follows:

[0250]

number

[0251] If the above equation 6 is satisfied and the value of r is set to the maximum value (i.e., r=M+N+(P*Q) / (M*N)), the value of r in the NSPT matrix for each block size may be set as follows:

[0252] For NSPT on a 4x4 block, the value of r is 24 (r=4+4+((16x16) / (4x4))), but the value of r must be less than or equal to 16, so the value of r may be set to 16.

[0253] For NSPT on 4x8 and 8x4 blocks, the value of r is 20 (r=4+8+((16x16) / (4x8))), so the value of r may be set to 20.

[0254] For NSPT on an 8x8 block, the value of r is 32 (r=8+8+((64x16) / (8x8))), so the value of r may be set to 32.

[0255] For NSPT on 4x16 blocks and 16x4 blocks, the value of r is 24 (r=4+16+((16x16) / (4x16))), so the value of r may be set to 24.

[0256] For NSPT on 8x16 and 16x8 blocks, the value of r is 40 (r=8+16+((64x32) / (8x16))), so the value of r may be set to 40.

[0257] For NSPT on 16x16 blocks, the value of r is 44 (r=16+16+((96x32) / (16x16))), so the value of r may be set to 44.

[0258] For NSPT on 16x32 blocks and 32x16 blocks, the value of r is 54 (r=16+32+((96x32) / (16x32))), so the value of r may be set to 54.

[0259] For NSPT on a 32x32 block, the value of r is 67 (r=32+32+((96x32) / (32x32))), so the value of r may be set to 67.

[0260] The value of r according to the above-mentioned predetermined criterion is for the case where zero-out is not taken into consideration. That is, when the forward LFNST is applied, the linearly transformed coefficients of the remaining regions other than the region to which the LFNST is applied are zero-out, so the actual amount of calculation required to apply the DCT-2 and LFNST may be less than the above-mentioned amount of calculation. Therefore, when the zero-out is taken into consideration, the value of r may be set to a value smaller than the value of r according to the above-mentioned predetermined criterion.

[0261] Since no zero-out is performed for 4x4 blocks, the value of r may be set to a value less than or equal to 16.

[0262] For a 4x8 block, zero-out may be performed on the remaining area except for the upper left 4x4 block based on the forward transform, and a 16x16 matrix, which is the forward LFNST matrix, may be applied to the upper left 4x4 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 8 (((4x4x8) + (4x8x4)) / (4x8) = 8), and the number of multiplications per sample required in the LFNST is 8 ((16x16) / 32 = 8). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 16, which is the sum of the number of multiplications per sample in the separable linear transform and the number of multiplications per sample in the LFNST. Since the same amount of calculation is required when applying the inverse separable linear transform and LFNST, the value of r may be set to a value less than or equal to 16.

[0263] For an 8x4 block, zero-out may be performed on the remaining region except for the upper left 4x4 block based on the forward transform, and a 16x16 matrix, which is the forward LFNST matrix, may be applied to the upper left 4x4 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 6 ((4x8x4) + (4x4x4) / (8x4) = 6), and the number of multiplications per sample required in the LFNST is 8 ((16x16) / 32 = 8). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 14, which is the sum of the number of multiplications per sample in the separable linear transform and the number of multiplications per sample in the LFNST. Since the same amount of calculation is required when applying the inverse separable linear transform and LFNST, the value of r may be set to a value less than or equal to 14.

[0264] For 8x8 blocks, no zero-out may be performed on the separate linear transform, in which case the value of r may be set to a value less than or equal to 32.

[0265] For an 8x16 block, zero-out may be performed on the remaining area except for the upper-left 8x8 block based on the forward transform, and a 64x32 matrix, which is a forward LFNST matrix, may be applied to the upper-left 8x8 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 16 ((8x8x16) + (8x16x8) / (8x16) = 16), and the number of multiplications per sample required in the LFNST is 16 ((64x32) / 128 = 16). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 32, which is the sum of the number of multiplications per sample in the separable linear transform and the number of multiplications per sample in the LFNST. Since applying the inverse separable linear transform and the LFNST requires the same amount of computation, the value of r may be set to a value less than or equal to 32.

[0266] For a 16x8 block, zero-out may be performed on the remaining area except for the upper-left 8x8 block based on the forward transform, and a 64x32 matrix, which is a forward LFNST matrix, may be applied to the upper-left 8x8 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 12 ((8x16x8) + (8x8x8) / (16x8) = 12), and the number of multiplications per sample required in the LFNST is 16 ((64x32) / 128 = 16). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 28, which is the sum of the number of multiplications per sample in the separable linear transform and the number of multiplications per sample in the LFNST. Since applying the inverse separable linear transform and the LFNST requires the same amount of computation, the value of r may be set to a value less than or equal to 28.

[0267] For a 16x16 block, zero-out may be performed on the remaining area except for the upper-left 12x12 block based on the forward transform, and a 96x32 matrix, which is a forward LFNST matrix, may be applied to the upper-left 12x12 block. When such zero-out is performed, the number of multiplications per sample required in the forward separable linear transform is 21 ((12x16x16) + (12x16x12) / (16x16) = 21), and the number of multiplications per sample required in the LFNST is 12 ((96x32) / 256 = 12). Therefore, when replacing the separable linear transform and LFNST with NSPT, the value of r may be set to a value less than or equal to 33, which is the sum of the number of multiplications per sample in the separable linear transform and the number of multiplications per sample in the LFNST. Since applying the inverse separable linear transform and the LFNST requires the same amount of computation, the value of r may be set to a value less than or equal to 33.

[0268] As mentioned above, the value of r in the NSPT matrix for an MxN block may be different from the value of r in the NSPT matrix for an NxM block. For example, the inverse NSPT matrix for a 4x8 block may be a 32x16 matrix, and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix. In this case, the NSPT matrix may be determined by utilizing the symmetry between the MxN block and the NxM block.

[0269] Assume that the current block is an MxN block having an xth mode. When utilizing the above-described symmetry for the current block, instead of applying an NSPT matrix corresponding to the xth mode or the MxN block size, an NSPT matrix corresponding to at least one of a mode symmetrical to the xth mode or an NxM block size symmetrical to the MxN block size may be applied. In this case, the NSPT matrix corresponding to the NxM block size may be applied to the current block as is. Alternatively, the NSPT matrix corresponding to the NxM block size may be applied, but the value of r in the NSPT matrix corresponding to the MxN block size may be used.

[0270] For example, the inverse NSPT matrix for a 4x8 block may be a 32x16 matrix (i.e., the value of r in the NSPT matrix is ​​16), and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix (i.e., the value of r in the NSPT matrix is ​​14). If the current block is an 8x4 block having an xth mode, an NSPT matrix for at least one of the xth mode and the symmetrical mode of the current block or the 4x8 block symmetrical to the 8x4 block may be applied. In this case, the 32x16 matrix that is the inverse NSPT matrix for the 4x8 block may be used as is, or a 32x14 matrix having the r value of the inverse NSPT matrix for the 8x4 block may be used. Here, the 32x14 matrix may be derived by sampling 14 rows from the left of the 32x16 matrix. In this way, applying the 32x14 matrix to a current block having an 8x4 block size satisfies the above-mentioned predetermined criteria.

[0271] Conversely, if the current block is a 4x8 block having an xth mode, an NSPT matrix for at least one of a mode symmetrical to the xth mode or an 8x4 block symmetrical to the 4x8 block may be applied to the current block. In this case, a 32x14 matrix for an 8x4 block may be applied to the current block instead of a 32x16 matrix for a 4x8 block. This allows NSPT to be performed on the 4x8 block using fewer multiplications than the allowed number of multiplications.

[0272] If the values ​​of r that satisfy the above-mentioned predetermined conditions for NSPT for MxN blocks and NxM blocks are r1 and r2, respectively, the reverse NSPT matrix for MxN blocks and NxM blocks may be set to MNxmax(r1,r2).

[0273] For example, the inverse NSPT matrix for a 4x8 block may be a 32x16 matrix (i.e., the value of r in the NSPT matrix is ​​16), and the inverse NSPT matrix for an 8x4 block may be a 32x14 matrix (i.e., the value of r in the NSPT matrix is ​​14). If the current block is a 4x8 block with an xth mode, the NSPT matrix for at least one of the xth mode and the 8x4 block symmetrical to the 4x8 block may be applied. In this case, a 32x16 matrix may be used as the inverse NSPT matrix for the 8x4 block. If the inverse NSPT matrix is ​​not configured as M×max(r1,r2), a 32x14 matrix would be used as the NSPT matrix for the 8x4 block. However, if the NSPT matrix for the 4x8 block and the NSPT matrix for the 8x4 block are configured as a 32xmax(16,14) matrix, the 32x16 matrix may be fully applied.

[0274] Conversely, if the current block is an 8x4 block having an xth mode, an NSPT matrix for at least one of a mode symmetrical to the xth mode or a 4x8 block symmetrical to the 8x4 block may be applied to the current block. In this case, a 32x16 matrix, which is an inverse NSPT matrix for the 4x8 block, or a 32x14 matrix may be used. Here, the 32x14 matrix may be derived by sampling 14 rows from the left of the 32x16 matrix.

[0275] As described above, when constructing an NSPT matrix, a transform consisting of the maximum number of transform basis vectors can be applied while satisfying the predetermined conditions, thereby maximizing coding performance.

[0276] In the above-described embodiment, the value of r may be set to be a multiple of 16. For example, in the case of inverse NSPT for a 4x8 block and an 8x4 block, a 32x16 matrix may be applied instead of a 32x20 matrix. Transform coefficients of a transform block may be coded in units of a predetermined coefficient group (CG). Here, a CG may be defined as a group of 16 transform coefficients. For example, a CG may be a sub-block of a size such as 4x4, 2x8, or 8x2. A CG may not contain any non-zero transform coefficients. In this case, the coding process of the transform coefficients for that CG may be skipped. Therefore, setting the value of r to a multiple of 16 has the advantage of reducing implementation complexity.

[0277] Transform coefficients may be derived by applying forward NSPT to residual samples of an MxN block. In this case, the number of derived transform coefficients may be less than or equal to the value of (M*N) due to zero-out. That is, a forward NSPT matrix may be defined as an rx(M*N) matrix, where r represents the output length of the NSPT or the number of transform coefficients derived by the NSPT, and (M*N) represents the input length of the NSPT or the number of residual samples to which the NSPT is applied.

[0278] The derived transform coefficients may be arranged within an MxN block according to a predetermined scanning order, and areas where no transform coefficients are filled may be filled with 0 (i.e., zero-out). Therefore, in the process of scanning transform coefficients in a decoding device, if a non-zero transform coefficient is found in an area that would have been filled with 0 if NSPT had been applied (or if the scan position of the last significant coefficient in the MxN block is greater than or equal to r), it is considered that NSPT has not been applied to the MxN block, and an NSPT index need not be signaled.

[0279] One or more available r values ​​may be defined for block sizes to which NSPT can be applied. For example, one or more r values ​​may be defined for each block size to which NSPT can be applied. Alternatively, one r value may be defined for each block size to which NSPT can be applied, and the r value for one of the block sizes to which NSPT can be applied may be different from the r value for any other size. Alternatively, one r value may be defined for some of the block sizes to which NSPT can be applied, and two or more r values ​​may be defined for the remaining block sizes.

[0280] When multiple values ​​of r are available, an index specifying one of the multiple values ​​of r or r itself may be signaled. The index may be signaled in a high-level syntax (HLS) such as VPS, SPS, PPS, PH, or SH, or at a block level such as CTU, CU, or TU. When the value of r is within a specific range, bits that can include the range may be assigned and signaled. For example, when the value of r is within the range of 1 to 256, 8 bits may be specified as a fixed length and signaled.

[0281] The transformation kernel of the current block may be determined based on any one of the above-described embodiments 1 to 3. Alternatively, the transformation kernel of the current block may be determined based on a combination of at least two of the above-described embodiments 1 to 3, as long as the inventions according to the above-described embodiments 1 to 3 do not conflict with each other.

[0282] A transform index for the inverse transform of the current block may be signaled. Here, the transform index may identify one or more transform kernels (or transform matrices) belonging to a transform set. Here, the transform index may refer to an NSPT index identifying one or more NSPT kernels belonging to an NSPT set. Alternatively, the transform index may refer to an LFNST index identifying one or more LFNST kernels belonging to an LFNST set.

[0283] Whether the transform index corresponds to an NSPT index may be determined based on whether the size of the current block is one of the block sizes belonging to the first group. This assumes that block sizes to which NSPT can be applied and block sizes to which LFNST can be applied are distinguished from each other. In this case, if the size of the current block belongs to the first group, the transform index signaled for the current block corresponds to an NSPT index, and an NSPT kernel may be determined from the NSPT set based on the transform index. On the other hand, if the size of the current block does not belong to the first group, the transform index signaled for the current block corresponds to an LFNST index, and an LFNST kernel may be determined from the LFNST set based on the transform index. If the size of the current block does not belong to the first group, this may mean that the size of the current block belongs to the second group. Alternatively, if the size of the current block does not belong to the first group, this may mean that the size of the current block corresponds to a block size to which LFNST can be applied among block sizes belonging to the second group. In this way, the NSPT index and the LFNST index may be configured in a single integrated syntax rather than being configured in separate syntaxes.

[0284] As an example, assume that the block sizes to which NSPT can be applied that belong to the first group are 4x4, 4x8, 8x4, and 8x8. NSPT may be applied instead of LFNST for the block sizes that belong to the first group. Specifically, NSPT may be applied instead of a combination of a separable linear transform (e.g., DCT-2 or separable KLT) and LFNST. NSPT indexes may be signaled for the four block sizes that belong to the first group, and LFNST indexes may be signaled for the remaining block sizes (for which LFNST is allowed).

[0285] In this way, when NSPT indexes and LFNST indexes are signaled using a single unified syntax, the amount of information to be coded can be reduced. Also, by applying at least one of binarization, CABAC context, or initial value for entropy coding to the NSPT / LFNST indexes in the same way, the implementation complexity can be reduced.

[0286] Alternatively, the NSPT index and the LFNST index can be signaled as separate syntaxes, which may increase the implementation complexity but improve the compression performance by performing optimized entropy coding for each index.

[0287] When the number of LFNST kernel candidates in the LFNST set is the same as the number of NSPT kernel candidates in the NSPT set, the same binarization may be applied to the LFNST index and the NSPT index. The same CABAC context (or CABAC context increment) may be assigned to the bins of the LFNST index and the NSPT index.

[0288] Different binarization and / or CABAC contexts may be used for the LFNST index and the NSPT index. Different CABAC initial values ​​may be assigned to the LFNST index and the NSPT index. For example, one of the LFNST index and the NSPT index may be binarized based on fixed-length binarization, and the other may be binarized based on truncated unary binarization. Even if the binarization of the LFNST index and the NSPT index is the same, different CABAC contexts and / or CABAC initial values ​​may be assigned to them. When the number of LFNST kernel candidates belonging to the LFNST set and the number of NSPT kernel candidates belonging to the NSPT set are different from each other, different binarization and / or CABAC contexts may be used for the LFNST index and the NSPT index.

[0289] The number of NSPT kernel candidates belonging to the NSPT set may be set to be different for each block size. Alternatively, the block sizes belonging to the first group may be divided into a plurality of subgroups. In this case, the number of NSPT kernel candidates belonging to the NSPT set may be set to be different for each of the plurality of subgroups. Here, at least one of the plurality of subgroups may include a plurality of block sizes that are different from each other.

[0290] Depending on the number of NSPT kernel candidates belonging to the NSPT set, the binarization applied to the NSPT index may differ.

[0291] For example, if the number of NSPT kernel candidates in the NSPT set for a specific block size is three, the NSPT index may have any one of values ​​0 to 3. An NSPT index value of 0 may indicate that NSPT is not applied to the current block. An NSPT index value other than 0 may indicate the NSPT kernel candidate corresponding to the NSPT index among the three NSPT kernel candidates. A bin may be assigned to distinguish between cases where NSPT is applied and cases where it is not. A bin value of 0 may correspond to a case where the NSPT index value is 0. On the other hand, a bin value of 1 may correspond to a case where the NSPT index value is 1, 2, or 3. In this case, truncated unary binarization may be applied to distinguish the three NSPT kernel candidates. That is, two bins may be assigned to distinguish three NSPT kernel candidates, 0, 10, and 11.

[0292] When the number of NSPT kernel candidates in the NSPT set for a specific block size is two, the NSPT index may have a value between 0 and 2. When the value of the NSPT index is 0, it may indicate that NSPT is not applied to the current block. When the value of the NSPT index is not 0, it may indicate the NSPT kernel candidate corresponding to the NSPT index among the two NSPT kernel candidates. One bin may be assigned to distinguish between cases where NSPT is applied and cases where it is not. Two NSPT kernel candidates may be distinguished by assigning one bin indicating one of the two NSPT kernel candidates.

[0293] When the number of NSPT kernel candidates in the NSPT set for a specific block size is one, the NSPT index may have a value of either 0 or 1. When the value of the NSPT index is 0, this may indicate that NSPT is not applied to the current block. When the value of the NSPT index is 1, this may indicate one NSPT kernel candidate. In this case, it is possible to identify whether NSPT is applied and the NSPT kernel candidate with just one bin.

[0294] The inverse transform of the current block may be a separate linear transform and / or an LFNST-based inverse transform. That is, an inverse LFNST may be applied to all or some of the (dequantized) transform coefficients of the current block, and then an inverse separate linear transform may be applied to the transform coefficients derived by the LFNST to derive residual samples. For example, an inverse LFNST may be applied to the (dequantized) transform coefficients belonging to a certain region of the current block. Here, the certain region refers to a region to which a forward LFNST is applied, and is hereinafter referred to as a region of interest (ROI). The transform coefficients derived by the LFNST may be arranged in an ROI region according to a predetermined scan order. The predetermined scan order may be row-first order or column-first order. An inverse separate linear transform may be applied to the transform coefficients derived by the LFNST and the transform coefficients belonging to the remaining region of the current block excluding the ROI region. Alternatively, if zero-out is performed on the remaining region excluding the ROI region in the current block during the forward transform process (i.e., the transform coefficients in the remaining region are set to 0), an inverse separate linear transform may be applied to the transform coefficients derived by the LFNST.

[0295] An LFNST defined for a 4x8 block (hereinafter referred to as LFNST4x8) or an LFNST defined for an 8x4 block (hereinafter referred to as LFNST8x4) may be applied to the current block. When LFNST4x8 is applied to the current block, the ROI region of the current block may be a 4x8 region within the current block. Here, the 4x8 region may refer to a region including the top left sample of the current block, with a width and height of 4 and 8, respectively. When LFNST8x4 is applied to the current block, the ROI region of the current block may be an 8x4 region within the current block. Here, the 8x4 region may refer to a region including the top left sample of the current block, with a width and height of 8 and 4, respectively.

[0296] The LFNST4x8 may be applied only when the current block is a 4x8 block. Alternatively, the LFNST4x8 may be applied when the current block is 4xN and N is greater than or equal to 8 (e.g., when the current block is a 4x8, 4x16, 4x32, or larger block). Alternatively, the LFNST4x8 may be applied when the current block is 4xN and N is greater than or equal to 8 and less than or equal to a predetermined threshold. As an example, when the threshold is 16, the LFNST4x8 may be applied when the current block is a 4x8 or 4x16 block.

[0297] The LFNST8x4 may be applied only when the current block is an 8x4 block. Alternatively, the LFNST8x4 may be applied when the current block is an Nx4 block and N is greater than or equal to 8 (e.g., when the current block is an 8x4, 16x4, 32x4, or larger block). Alternatively, the LFNST8x4 may be applied when the current block is an Nx4 block and N is greater than or equal to 8 and less than or equal to a predetermined threshold. As an example, when the threshold is 16, the LFNST8x4 may be applied when the current block is an 8x4 or 16x4 block.

[0298] Specifically, when LFNST4x8 or LFNST8x4 is applied to a current block, an LFNST having an input length less than or equal to 32 and an output length of 32 (hereinafter referred to as LFNST-32) may be used. The input length may refer to the number of transform coefficients input to the LFNST. Here, the transform coefficients input to the LFNST may refer to all transform coefficients derived based on residual information of the current block, or may refer to transform coefficients to which a forward LFNST is applied among the derived transform coefficients. The output length may refer to the number of transform coefficients derived by the LFNST.

[0299] For example, if the size of the current block is 4xN, LFNST-32 may be applied to all or some of the transform coefficients belonging to a 4x8 region, which is a ROI region within the current block. The transform coefficients derived by LFNST-32 may be arranged in a predetermined scan order within the 4x8 region within the current block. Alternatively, if the size of the current block is Nx4, LFNST-32 may be applied to all or some of the transform coefficients belonging to an 8x4 region, which is a ROI region within the current block. The transform coefficients derived by LFNST-32 may be arranged in a predetermined scan order within the 8x4 region within the current block.

[0300] However, as described above, LFNST4x8 and LFNST8x4 may be adaptively applied based on the size of the current block. If LFNST4x8 and LFNST8x4 are not applicable to the current block, LFNST with an input length less than or equal to 16 and an output length of 16 (hereinafter referred to as LFNST16) may be used. LFNST16 may be applied to all or some of the transform coefficients belonging to a 4x4 region within the current block. A 4x4 region within the current block may refer to a region including the upper left sample of the current block, with a width and height of 4 and 4, respectively. The transform coefficients derived by LFNST16 may be arranged in the 4x4 region within the current block according to a predetermined scan order.

[0301] From the perspective of forward transform, transform coefficients can be derived by applying a separate linear transform to residual samples of the current block. Forward LFNST may be applied to all of the derived transform coefficients or only to transform coefficients belonging to a certain region (i.e., ROI region) within the current block. The transform coefficients derived by LFNST may be arranged in the ROI region of the current block according to a predetermined scan order. If the number of transform coefficients output from LFNST is less than the number of transform coefficients input to LFNST, there may be areas within the ROI region that are not filled with transform coefficients derived by LFNST. These areas may be filled with transform coefficients set to zero. Furthermore, the remaining regions of the current block excluding the ROI region (or regions to which LFNST is not applied) may be filled with transform coefficients derived by the separate linear transform, or may be filled with transform coefficients set to zero by zero-out.

[0302] Specifically, when LFNST4x8 or LFNST8x4 is applied to a current block or an ROI region of the current block, an LFNST having an input length of 32 and an output length less than or equal to 32 (hereinafter referred to as LFNST-32) may be used. Here, the input length may refer to the number of transform coefficients input to the LFNST. The input transform coefficients may be coefficients derived by a separate linear transform. The input length may refer to the number of transform coefficients belonging to the ROI region of the current block. The output length may refer to the number of transform coefficients derived by the LFNST.

[0303] For example, if the size of the current block is 4xN, LFNST-32 may be applied to transform coefficients belonging to a 4x8 region, which is a ROI region within the current block. The transform coefficients derived by LFNST-32 may be arranged in the 4x8 region within the current block according to a predetermined scan order. Alternatively, if the size of the current block is Nx4, LFNST-32 may be applied to transform coefficients belonging to an 8x4 region, which is a ROI region within the current block. The transform coefficients derived by LFNST-32 may be arranged in the 8x4 region within the current block according to a predetermined scan order.

[0304] However, as described above, LFNST4x8 and LFNST8x4 may be adaptively applied based on the size of the current block. If LFNST4x8 and LFNST8x4 are not applicable to the current block, LFNST (hereinafter referred to as LFNST16), whose input length and output length are 16 and 16, respectively, may be used. LFNST16 may be applied to transform coefficients belonging to a 4x4 region within the current block. The 4x4 region within the current block may refer to a region including the upper left sample of the current block, with a width and height of 4 and 4, respectively. The transform coefficients derived by LFNST16 may be arranged in the 4x4 region within the current block according to a predetermined scan order.

[0305] When applying LFNST4x8 or LFNST8x4 to the current block, the symmetry between the block types described above may be utilized. For example, the LFNST set and / or LFNST kernel for the LFNST of the current block may be determined depending on the symmetry between a 4xN block and an Nx4 block. A 4xN block (or an Nx4 block) may use the same LFNST set and / or LFNST kernel corresponding to the Nx4 block (or a 4xN block) that has symmetry with it. For blocks where N is 8, the above-described symmetry may be utilized as is, and for blocks where N is greater than or equal to 16, LFNST may be applied to an ROI region (i.e., a 4x8 region or an 8x4 region) within the current block using the above-described symmetry.

[0306] In addition, LFNST for 8x16 blocks (hereinafter referred to as LFNST8x16) and LFNST for 16x8 blocks (hereinafter referred to as LFNST16x8) may be defined. When LFNST8x16 is applied to a current block, the ROI region of the current block may be a 4x8, 4x16, 8x8, or 8x16 region within the current block. Here, the ROI region may refer to a region including the top left sample of the current block and a block having the corresponding width and height. When LFNST16x8 is applied to a current block, the ROI region of the current block may be an 8x4, 16x4, 8x8, or 16x8 region within the current block. Here, the ROI region may refer to a region including the top left sample of the current block and a block having the corresponding width and height. The ROI region may be preset identically in the encoding device and the decoding device for LFNST8x16 and LFNST16x8, respectively. The same ROI region may be set for blocks whose width and height are 8 and N, respectively. Alternatively, at least one of the blocks whose width and height are 8 and N, respectively, may have a different ROI region from the other blocks. Similarly, the same ROI region may be set for blocks whose width and height are N and 8, respectively. Alternatively, at least one of the blocks whose width and height are N and 8, respectively, may have a different ROI region from the other blocks.

[0307] The LFNST8x16 may be applied only when the current block is an 8x16 block. Alternatively, the LFNST8x16 may be applied when the current block is 8xN and N is greater than or equal to 16 (e.g., when the current block is an 8x16, 8x32, 8x64, or larger block). Alternatively, the LFNST8x16 may be applied when the current block is 8xN and N is greater than or equal to 16 and less than or equal to a predetermined threshold. As an example, when the threshold is 32, the LFNST8x16 may be applied when the current block is an 8x16 or 8x32 block.

[0308] The LFNST16x8 may be applied only when the current block is a 16x8 block, or when the current block is Nx8 and N is greater than or equal to 16 (e.g., when the current block is a 16x8, 32x8, 64x8, or larger block), or when the current block is Nx8 and N is greater than or equal to 16 and less than or equal to a predetermined threshold. As an example, when the threshold is 32, the LFNST16x8 may be applied when the current block is a 16x8 or 32x8 block.

[0309] Specifically, when LFNST8x16 or LFNST16x8 is applied to a current block, an LFNST having an input length less than or equal to K and an output length of K (hereinafter referred to as LFNST-K) may be used. As described above, the input length may refer to the number of transform coefficients input to the LFNST. Here, the transform coefficients input to the LFNST may refer to all transform coefficients derived based on residual information of the current block, or may refer to transform coefficients to which a forward LFNST is applied among the derived transform coefficients. The output length may refer to the number of transform coefficients derived by the LFNST.

[0310] The K may be determined based on the size of the ROI region of the current block. For example, if the ROI region is 4x8 or 8x4, an LFNST whose input length is less than or equal to 32 and whose output length is 32 (i.e., LFNST-32) may be used. If the ROI region is 4x16, 16x4, or 8x8, an LFNST whose input length is less than or equal to 64 and whose output length is 64 (i.e., LFNST-64) may be used. If the ROI region is an 8x16 or 16x8 region, an LFNST whose input length is less than or equal to 128 and whose output length is 128 (i.e., LFNST-128) may be used. That is, if LFNST8x16 or LFNST16x8 is applied to an 8xN or Nx8 current block, LFNST-K may be applied to the ROI region within the current block. Here, K may be 32, 64, or 128, but this is merely an example, and K may be an integer greater than 128.

[0311] If the size of the current block is 8xN, LFNST-K may be applied to all or a portion of the transform coefficients belonging to an ROI region within the current block. The transform coefficients derived by LFNST-K may be arranged in a predetermined scan order within the ROI region of the current block. Alternatively, if the size of the current block is Nx8, LFNST-K may be applied to all or a portion of the transform coefficients belonging to an ROI region within the current block. The transform coefficients derived by LFNST-K may be arranged in a predetermined scan order within the ROI region of the current block.

[0312] However, as described above, LFNST8x16 and LFNST16x8 may be adaptively applied based on the size of the current block. If LFNST8x16 and LFNST16x8 are not applicable to the current block, LFNST (hereinafter referred to as LFNST48), which has an input length of 16 and an output length of 48, may be used. LFNST48 may be applied to transform coefficients belonging to a 4x4 region within the current block. The 4x4 region within the current block may be a region including the top-left sample of the current block, and may refer to a block with a width and height of 4. The transform coefficients derived by LFNST48 may be arranged in three 4x4 regions within the current block according to a predetermined scanning order. The three 4x4 regions may include a 14x4 region including the top-left sample of the current block, a 24x4 region adjacent to the right of the 14x4 region, and a 34x4 region adjacent to the bottom of the 14x4 region.

[0313] From the perspective of forward transform, transform coefficients can be derived by applying a separate linear transform to residual samples of the current block. Forward LFNST may be applied to all of the derived transform coefficients or only to transform coefficients belonging to a certain region (i.e., ROI region) within the current block. The transform coefficients derived by LFNST may be arranged in the ROI region of the current block according to a predetermined scan order. If the number of transform coefficients output from LFNST is less than the number of transform coefficients input to LFNST, there may be areas within the ROI region that are not filled with transform coefficients derived by LFNST. These areas may be filled with transform coefficients of zero. Furthermore, the remaining regions of the current block excluding the ROI region (or regions to which LFNST is not applied) may be filled with transform coefficients derived by the separate linear transform, or may be filled with transform coefficients of zero by zero-out.

[0314] Specifically, when LFNST8x16 or LFNST16x8 is applied to a current block or an ROI region of the current block, an LFNST having an input length of K and an output length less than or equal to K (i.e., referred to as LFNST-K) may be used. Here, the input length may refer to the number of transform coefficients input to the LFNST. The input transform coefficients may be coefficients derived by a separate linear transform. The input length may refer to the number of transform coefficients belonging to the ROI region of the current block. The output length may refer to the number of transform coefficients derived by the LFNST.

[0315] For example, if the size of the current block is 8xN, where N is greater than or equal to 16, LFNST-K may be applied to transform coefficients belonging to an ROI region within the current block. The transform coefficients derived by LFNST-K may be arranged in the ROI region within the current block according to a predetermined scan order. Alternatively, if the size of the current block is Nx8, where N is greater than or equal to 16, LFNST-K may be applied to transform coefficients belonging to an ROI region within the current block. The transform coefficients derived by LFNST-K may be arranged in the ROI region within the current block according to a predetermined scan order.

[0316] However, as described above, LFNST8x16 and LFNST16x8 may be adaptively applied based on the size of the current block. If LFNST8x16 and LFNST16x8 are not applicable to the current block, LFNST (i.e., LFNST48) with input and output lengths of 48 and 16, respectively, may be used. LFNST48 may be applied to transform coefficients belonging to three 4x4 regions within the current block. The three 4x4 regions may consist of a 14x4 region including the upper left sample of the current block, a 24x4 region adjacent to the right of the 14x4 region, and a 34x4 region adjacent to the bottom of the 14x4 region. The transform coefficients derived by LFNST48 may be arranged in the 4x4 regions within the current block according to a predetermined scan order.

[0317] When applying LFNST8x16 or LFNST16x8 to the current block, the symmetry between the block types described above may be utilized. For example, the LFNST set and / or LFNST kernel for LFNST-K of the current block may be determined depending on the symmetry between an 8xN block and an Nx8 block. An 8xN block (or an Nx8 block) may use the same LFNST set and / or LFNST kernel corresponding to its symmetric Nx8 block (or an 8xN block). For blocks where N is 16, the symmetry described above may be utilized, and for blocks where N is greater than or equal to 32, LFNST may be applied to an ROI region (i.e., an 8x16 region or a 16x8 region) within the current block using the symmetry described above.

[0318] As described above, LFNSTMx2M for an Mx2M block and LFNST2MxM for a 2MxM block may be defined. LFNSTMx2M may be applied to an MxN block, and LFNST2MxM may be applied to an NxM block, where N may be greater than or equal to (2*M). ROI regions may be defined for LFNSTMx2M and LFNST2MxM, respectively, and LFNSTs with predetermined input and output lengths may be defined / used. The size of the ROI region may be determined based on the minimum of the width and height of the current block (i.e., M). The size of the ROI region may vary depending on the value of M. At least one of the input and output lengths of the LFNST may be determined based on at least one of the minimum of the width and height of the current block or the size of the ROI region.

[0319] Referring to FIG. 4, the current block can be reconstructed based on the residual samples of the current block (S420).

[0320] A prediction sample of the current block can be derived based on the intra prediction mode of the current block, and a reconstructed sample of the current block can be generated based on the prediction sample and residual sample of the current block.

[0321] FIG. 6 shows a schematic configuration of a decoding device 300 that performs the image decoding method according to the present disclosure.

[0322] 6, a decoding apparatus 300 according to the present disclosure may include a transform coefficient derivation unit 600, a residual sample derivation unit 610, and a reconstructed block generation unit 620. The transform coefficient derivation unit 600 may be configured in the entropy decoding unit 310 of FIG. 3, the residual sample derivation unit 610 may be configured in the residual processing unit 320 of FIG. 3, and the reconstructed block generation unit 620 may be configured in the adder 340 of FIG. 3.

[0323] The transform coefficient deriving unit 600 can obtain residual information of the current block from the bitstream and decode it to derive transform coefficients of the current block.

[0324] The residual sample derivation unit 610 may derive residual samples of the current block by performing at least one of inverse quantization and inverse transformation on the transform coefficients of the current block.

[0325] The residual sample deriving unit 610 determines a transform kernel for inverse transform of the current block using a predetermined transform kernel determination method, and derives a residual sample of the current block based on the determined transform kernel, as described with reference to FIG. 4, and detailed description thereof will be omitted here.

[0326] The reconstructed block generator 620 can reconstruct the current block based on the residual samples of the current block.

[0327] FIG. 7 is a diagram illustrating an image encoding method performed by an encoding device 200 according to an embodiment of the present disclosure.

[0328] Referring to FIG. 7, residual samples of the current block can be derived (S700).

[0329] Residual samples of the current block may be derived by subtracting prediction samples from original samples of the current block, where the prediction samples may be derived based on a predetermined intra prediction mode.

[0330] Referring to FIG. 7, transform coefficients of the current block can be derived by performing at least one of transformation and quantization on residual samples of the current block (S710).

[0331] The transformation method according to the present disclosure may be understood as the inverse process of the inverse transformation described with reference to Fig. 4. The method for determining the transformation kernel for the transformation is the same as that described with reference to Fig. 4, and a detailed description thereof will be omitted here.

[0332] For example, one or more transform sets for transforming the current block may be defined / configured, and each transform set may include one or more transform kernel candidates. In this case, one of the plurality of transform sets may be selected as the transform set for the current block. One of the plurality of transform kernel candidates belonging to the transform set for the current block may be selected. The selection may be made implicitly based on the context of the current block. Alternatively, the optimal transform set and / or transform kernel candidate for the current block may be selected, and an index indicating this may be signaled.

[0333] Alternatively, the transform kernel for the current block may be determined based on an MTS set. One of a plurality of MTS sets may be selected based on at least one of the size of the current block or the intra prediction mode. The selected MTS set may include one or more transform kernel candidates. One of the one or more transform kernel candidates may be selected, and the transform kernel for the current block may be determined based on the selected transform kernel candidate. The selection of the transform kernel candidate may be performed using a transform kernel candidate index derived based on the context of the current block. Alternatively, the optimal transform kernel candidate for the current block may be selected, and a transform kernel candidate index indicating the selected transform kernel candidate may be signaled.

[0334] Alternatively, the transform kernel of the current block may be determined based on a non-separable linear transform (NSPT) kernel. If the size of the current block belongs to a first group, which is a set of block sizes to which NSPT can be applied, a forward NSPT may be applied to the current block. If the size of the current block belongs to a second group, a forward NSPT may not be applied to the current block. If the size of the current block belongs to the second group, a forward separable linear transform (e.g., DCT-2) may be applied to residual samples of the current block to derive transform coefficients. A forward LFNST may further be applied to all or some of the transform coefficients derived by the separable linear transform.

[0335] Furthermore, the NSPT may be applied based on at least one of the tree type or component type of the current block. The NSPT kernel (or NSPT matrix) for the NSPT may be determined using symmetry between intra prediction modes or symmetry between block types. When forward NSPT is applied to an MxN current block, the NSPT kernel may be expressed as rxMN. Here, r represents the output length of the NSPT or the number of transform coefficients generated by the NSPT, and MN is the product of the width and height of the current block and may represent the input length of the NSPT or the number of residual samples to which the NSPT is applied. A method for determining the size of the NSPT kernel has been described with reference to FIG. 4.

[0336] The LFNST index and / or NSPT index for the transform may be coded into a single unified syntax, or the LFNST index and the NSPT index may be coded separately and inserted into the bitstream. The binarization of the LFNST index and the NSPT index, and the assignment of the CABAC context and initial value are as described with reference to FIG. 4.

[0337] Referring to FIG. 7, the transform coefficients of the current block can be coded to generate a bitstream (S720).

[0338] Residual information about the transform coefficients may be generated based on the transform coefficients of the current block, and the residual information may be coded to generate a bitstream.

[0339] FIG. 8 shows a schematic configuration of an encoding device 200 that performs the image encoding method according to the present disclosure.

[0340] 8, the encoding apparatus 200 according to the present disclosure may include a residual sample derivation unit 800, a transform coefficient derivation unit 810, and a transform coefficient encoding unit 820. The residual sample derivation unit 800 and the transform coefficient derivation unit 810 may be configured in the residual processing unit 230 of FIG. 2, and the transform coefficient encoding unit 820 may be configured in the entropy encoding unit 240 of FIG. 2.

[0341] The residual sample deriving unit 800 may derive residual samples of the current block by subtracting predicted samples from original samples of the current block, where the predicted samples may be derived based on a predetermined intra prediction mode.

[0342] The transform coefficient deriving unit 810 may derive transform coefficients of the current block by performing at least one of transform and quantization on residual samples of the current block. The transform coefficient deriving unit 810 may determine a transform kernel for the current block based on at least one of the first to third embodiments and derive transform coefficients by applying the transform kernel to residual samples of the current block.

[0343] The transform coefficient encoding unit 820 may encode the transform coefficients of the current block to generate a bitstream.

[0344] In the above-described embodiments, the method is described based on a flowchart with a series of steps or blocks, but the embodiment is not limited to the order of the steps, and some steps may occur in a different order or simultaneously with other steps than those described above. Furthermore, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, and other steps may be included, or one or more steps of the flowchart may be deleted without affecting the scope of the embodiments of this document.

[0345] The methods according to the embodiments of the present document described above may be implemented in the form of software, and the encoding device and / or decoding device according to the present document may be included in an image processing device such as a TV, a computer, a smartphone, a set-top box, or a display device.

[0346] When embodiments in this document are embodied as software, the methods described above may be embodied as modules (processes, functions, etc.) that perform the functions described above. The modules may be stored in memory and executed by a processor. The memory may be internal or external to the processor and may be coupled to the processor by various known means. The processor may include an application-specific integrated circuit (ASIC), other chipsets, logic circuits, and / or data processing devices. The memory may include read-only memory (ROM), random access memory (RAM), flash memory, a memory card, a storage medium, and / or other storage devices. That is, the embodiments described herein may be embodied and executed on a processor, microprocessor, controller, or chip. For example, the functional units illustrated in the figures may be embodied and executed on a computer, processor, microprocessor, controller, or chip. In this case, information (e.g., information on instructions) or algorithms for the implementation may be stored on a digital storage medium.

[0347] In addition, the decoding device and encoding device to which the embodiments of the present specification are applied may be included in a multimedia broadcast transmitting / receiving device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, a custom video (VoD) service providing device, an over-the-top (OTT) video (over-the-top) device, an internet streaming service providing device, a three-dimensional (3D) video device, a virtual reality (VR) device, an augmented reality (AR) device, an image telephone video device, a vehicle terminal (e.g., a vehicle terminal (including an autonomous vehicle), an airplane terminal, a ship terminal, etc.), a medical video device, etc., and may be used to process video signals or data signals. For example, over-the-top (OTT) video (over-the-top) video devices may include a game console, a Blu-ray player, an internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), etc.

[0348] In addition, a processing method to which the embodiments of the present specification are applied may be produced in the form of a program executed by a computer and stored in a computer-readable recording medium. Multimedia data having a data structure according to the embodiments of the present specification may also be stored in a computer-readable recording medium. The computer-readable recording medium may include any type of storage device or distributed storage device in which computer-readable data is stored. The computer-readable recording medium may include, for example, a Blu-ray Disc (BD), a Universal Serial Bus (USB), a ROM, a PROM, an EPROM, an EEPROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device. The computer-readable recording medium may also include media embodied in the form of a carrier wave (e.g., transmission via the Internet). In addition, a bitstream generated by the encoding method may be stored in a computer-readable recording medium or transmitted via a wired or wireless communication network.

[0349] Furthermore, the embodiments of the present specification may be embodied as a computer program product using program code, which may be executed by a computer according to the embodiments of the present specification. The program code may be stored on a computer-readable carrier.

[0350] FIG. 9 illustrates an example of a content streaming system to which the embodiments of the present disclosure can be applied.

[0351] Referring to FIG. 9, a content streaming system to which the embodiments of the present specification are applied may broadly include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0352] The encoding server compresses content input from a multimedia input device such as a smartphone, camera, camcorder, etc. into digital data to generate a bitstream and transmits the bitstream to the streaming server. As another example, if a multimedia input device such as a smartphone, camera, camcorder, etc. directly generates a bitstream, the encoding server may be omitted.

[0353] The bitstream may be generated by an encoding method or a bitstream generation method to which the embodiments of this specification are applied, and the streaming server may temporarily store the bitstream during the process of transmitting or receiving the bitstream.

[0354] The streaming server transmits multimedia data to a user device based on a user request via a web server, and the web server acts as an intermediary to inform the user of available services. When a user requests a desired service from the web server, the web server transmits the request to the streaming server, which then transmits the multimedia data to the user. In this case, the content streaming system may include a separate control server, which controls commands and responses between devices in the content streaming system.

[0355] The streaming server can receive content from a media storage and / or encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, the streaming server can store the bitstream for a certain period of time to provide a smooth streaming service.

[0356] Examples of the user devices include mobile phones, smartphones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, slate PCs, tablet PCs, ultrabooks, wearable devices (e.g., smartwatches, smart glasses, and head-mounted displays (HMDs)), digital TVs, desktop computers, and digital signage.

[0357] Each server in the content streaming system may be operated as a distributed server, in which case data received by each server may be processed in a distributed manner.

[0358] The claims described herein may be combined in various ways. For example, the technical features of the method claims herein may be combined and embodied as an apparatus, and the technical features of the apparatus claims herein may be combined and embodied as a method. Furthermore, the technical features of the method claims herein and the technical features of the apparatus claims herein may be combined and embodied as an apparatus, and the technical features of the method claims herein and the technical features of the apparatus claims herein may be combined and embodied as a method.

[0359] [Claims at the time of international application] [Claim 1] 1. A method for decoding an image, comprising: obtaining residual information from the bitstream; deriving transform coefficients of a current block based on the residual information; determining a transform kernel for the current block based on a transform index for an inverse transform of the current block; applying the inverse transform to at least one of the transform coefficients based on the transform kernel to derive a residual sample; reconstructing the current block based on residual samples of the current block. [Claim 2] The transformation kernel is determined from a non-separable linear transform (NSPT) set including one or more candidate transformation kernels based on the size of the current block belonging to a first group, which is a set of block sizes to which a non-separable linear transform (NSPT) can be applied; 2. The image decoding method of claim 1, wherein the transform kernel is determined from a low-frequency non-separable transform (LFNST) set including one or more LFNST kernel candidates based on the size of the current block not belonging to the first group. [Claim 3] The image decoding method of claim 1 , wherein the transform index indicates one of a non-separable linear transform (NSPT) index or a low-frequency non-separable transform (LFNST) index. [Claim 4] The image decoding method of claim 3 , wherein the NSPT index is coded based on at least one of a binarization method different from that of the LFNST index or a CABAC context. [Claim 5] the inverse transformation is based on a non-separable linear transform (NSPT); The image decoding method of claim 1 , wherein an NSPT set for the NSPT includes one or more NSPT kernel candidates. [Claim 6] The image decoding method of claim 5 , wherein the number of the one or more NSPT kernel candidates belonging to the NSPT set is determined based on a size of the current block. [Claim 7] the inverse transform is based on a low frequency non-separable transform (LFNST); If the size of the current block is MxN or NxM, the number of transform coefficients output by the LFNST is (2*M*M), 2. The image decoding method of claim 1, wherein N is an integer greater than or equal to (2*M). [Claim 8] 8. The image decoding method of claim 7, wherein the size of the current block is 4x8, 8x4, 8x16, or 16x8. [Claim 9] The image decoding method of claim 7, wherein the number of transform coefficients output by the LFNST is 16 or 48 depending on whether the size of the current block is MxN or NxM. [Claim 10] 1. An image encoding method, comprising: deriving a residual sample of the current block; applying a transform to residual samples of the current block to derive transform coefficients of the current block; encoding the transform coefficients of the current block to generate a bitstream; the bitstream further includes a transform index for the transform; An image encoding method, wherein the transform index is a non-separable linear transform (NSPT) index indicating one or more NSPT kernel candidates belonging to an NSPT set, or a low-frequency non-separable transform (LFNST) index indicating one or more LFNST kernel candidates belonging to an LFNST set. [Claim 11] A computer-readable recording medium, comprising: A computer-readable recording medium storing a bitstream generated by the image encoding method of claim 10. [Claim 12] 1. A data transmission method, comprising: obtaining a bitstream for image information; the bitstream is generated by deriving residual samples of a current block, applying a transform to the residual samples of the current block to derive transform coefficients of the current block, and encoding the transform coefficients of the current block; transmitting data containing said bitstream; the bitstream further includes a transform index for the transform; A data transmission method, wherein the transform index is an NSPT index indicating any one of one or more NSPT kernel candidates belonging to a non-separable linear transform (NSPT) set, or an LFNST index indicating any one of one or more LFNST kernel candidates belonging to a low-frequency non-separable transform (LFNST) set.

Claims

1. 1. A method for decoding an image, comprising: obtaining residual information from the bitstream; deriving transform coefficients of a current block based on the residual information; determining a transform kernel for the current block based on a transform index for an inverse transform of the current block; applying the inverse transform to at least one of the transform coefficients based on the transform kernel to derive a residual sample; reconstructing the current block based on residual samples of the current block.

2. The transformation kernel is determined from a non-separable linear transform (NSPT) set including one or more candidate transformation kernels based on the size of the current block belonging to a first group, which is a set of block sizes to which a non-separable linear transform (NSPT) can be applied; 2. The image decoding method of claim 1, wherein the transform kernel is determined from a low frequency non-separable transform (LFNST) set including one or more LFNST kernel candidates based on the size of the current block not belonging to the first group.

3. 2. The image decoding method of claim 1, wherein the transform index represents one of a non-separable linear transform (NSPT) index or a low frequency non-separable transform (LFNST) index.

4. The image decoding method of claim 3 , wherein the NSPT index is encoded based on at least one of a binarization method or a CABAC context different from that of the LFNST index.

5. the inverse transformation is based on a non-separable linear transform (NSPT); The image decoding method of claim 1 , wherein the NSPT set for the NSPT includes one or more NSPT kernel candidates.

6. The image decoding method of claim 5 , wherein the number of the one or more NSPT kernel candidates belonging to the NSPT set is determined based on a size of the current block.

7. the inverse transform is based on a low frequency non-separable transform (LFNST); Based on the size of the current block being MxN or NxM, the number of transform coefficients output by the LFNST is (2*M*M), 2. The image decoding method of claim 1, wherein N is an integer greater than or equal to (2*M).

8. The image decoding method of claim 7 , wherein the size of the current block is 4×8, 8×4, 8×16, or 16×8.

9. The image decoding method of claim 7 , wherein the number of transform coefficients output by the LFNST is 16 or 48 depending on whether the size of the current block is M×N or N×M.

10. 1. An image encoding method, comprising: deriving a residual sample of the current block; applying a transform to residual samples of the current block to derive transform coefficients of the current block; encoding the transform coefficients of the current block to generate a bitstream; the bitstream further includes a transform index for the transform; An image encoding method, wherein the transform index is a non-separable linear transform (NSPT) index indicating any one of one or more NSPT kernel candidates belonging to an NSPT set, or a low frequency non-separable transform (LFNST) index indicating any one of one or more LFNST kernel candidates belonging to an LFNST set.

11. A computer-readable recording medium, comprising: A computer-readable recording medium storing a bitstream generated by the image encoding method of claim 10.

12. 1. A data transmission method, comprising: obtaining a bitstream for image information; the bitstream is generated by deriving residual samples of a current block, applying a transform to the residual samples of the current block to derive transform coefficients of the current block, and encoding the transform coefficients of the current block; transmitting data containing the bitstream; the bitstream further includes a transform index for the transform; A data transmission method, wherein the transform index is an NSPT index indicating any one of one or more NSPT kernel candidates belonging to a non-separable primary transform (NSPT) set, or an LFNST index indicating any one of one or more LFNST kernel candidates belonging to a low frequency non-separable transform (LFNST) set.

Images