Method and apparatus for designing low frequency non-separable transforms

Abstract

The image decoding method described in this document includes the following steps: deriving the LFNST matrix for the current block based on the LFNST set index and the LFNST index derived from the LFNST index information; deriving modified transform coefficients based on the transform coefficients and the LFNST matrix; and generating residual samples for the current block based on the modified transform coefficients. The LFNST matrix can be derived as a 96×32 dimensional matrix when the width or height of the current block is 16, or when both the width and height are 16 or greater. Therefore, the coding performance achievable with LFNST can be maximized within the implementation complexity allowed by the upcoming standard.

Description

Technical Field

[0001] This document relates to video / image coding technology, and more specifically, to image coding methods and apparatus based on Low Frequency Inseparable Transform (LFNST) in video or image coding systems. Background Technology

[0002] There has been a growing demand for high-resolution, high-quality images and videos, such as ultra-high-definition (HUD) images and 4K or 8K or higher video, across various fields. As image and video data becomes higher resolution and higher quality, the relative amount of information or bits transmitted increases compared to existing image and video data. Therefore, transmission and storage costs increase if media such as existing wired or wireless broadband lines are used to transmit image data or if existing storage media are used to store image and video data.

[0003] Furthermore, there has been a growing interest in and demand for immersive media such as virtual reality (VR), artificial reality (AR) content, or holograms. The broadcasting of images and videos, such as game graphics, whose image characteristics differ from those of real-world images, is also on the rise.

[0004] Therefore, in order to effectively compress and transmit or store and play back information of high-resolution and high-quality images and videos with such various characteristics, efficient image and video compression technologies are needed. Summary of the Invention

[0005] Technical issues

[0006] The technical problem this document addresses is to provide methods and devices for improving image coding efficiency.

[0007] Another technical problem this document addresses is providing image coding methods and devices for applying LFNST under various conditions.

[0008] Another technical problem this document addresses is to provide an image coding method and apparatus that sets up the LFNST kernel by taking into account the computational complexity of the samples.

[0009] Another technical problem this document addresses is providing an image encoding method and apparatus using LFNST, which improves encoding performance and minimizes complexity.

[0010] Technical solution

[0011] According to the embodiments in this document, an image decoding method performed by a decoding device is provided. The method includes the following steps: deriving an LFNST matrix for the current block based on an LFNST set index and an LFNST index derived from LFNST index information; deriving modified transform coefficients based on transform coefficients and the LFNST matrix; and generating residual samples for the current block based on the modified transform coefficients, wherein the LFNST matrix can be derived as a 96×32-dimensional matrix based on the current block's width or height being equal to 16, and both the width and height being equal to or greater than 16.

[0012] According to the embodiments in this document, an image encoding method performed by an encoding device is provided. The method includes the following steps: deriving modified transform coefficients based on transform coefficients and an LFNST matrix; generating residual information based on the modified transform coefficients; generating LFNST index information indicating the LFNST matrix in an LFNST set indexed by the LFNST set index; and encoding image information including information related to intra-frame prediction modes, residual information, and LFNST index information, wherein the LFNST matrix can be derived as a 32×96-dimensional matrix based on the current block's width or height being equal to 16, and the width and height both being equal to or greater than 16.

[0013] According to the embodiments described in this document, an encoding device for performing video / image encoding is provided.

[0014] According to another embodiment of this document, a digital storage medium for storing image data, including bit streams and / or encoded image information generated based on an image encoding method performed by an encoding device, and a method for transmitting such image information and / or bit streams, can be provided.

[0015] According to another embodiment of this document, a digital storage medium for storing image data, including encoded image information and / or bitstreams for an image decoding method to be performed by a decoding device, and a method for transmitting such image information and / or bitstreams, can be provided.

[0016] Beneficial effects

[0017] The implementation methods described in this document can improve overall video / image compression efficiency.

[0018] According to the implementation method described in this document, LFNST can be applied based on various conditions.

[0019] According to the implementation method described in this document, the LFNST set index can be efficiently derived based on the intra-frame prediction mode.

[0020] According to this document, the LFNST kernel can be set by taking into account the computational complexity of the samples.

[0021] According to this document, an image encoding method and apparatus can be provided, and the application of LFNST can improve encoding performance and minimize complexity.

[0022] The effects achievable through the specific examples in this specification are not limited to those listed above. For example, various technical effects may exist that can be understood or derived from this specification by one of ordinary skill in the art. Therefore, the specific effects of this specification are not limited to those explicitly described in this specification, but may include various effects that can be understood or derived from the technical features of this specification. Attached Figure Description

[0023] Figure 1 Examples of video / image coding systems to which exemplary embodiments of this disclosure are applicable are illustrated schematically.

[0024] Figure 2 This is a diagram illustrating, schematically, the configuration of a video / image encoding device to which exemplary embodiments of this disclosure are applicable.

[0025] Figure 3 This is a diagram illustrating, schematically, the configuration of a video / image decoding device to which exemplary embodiments of this disclosure are applicable.

[0026] Figure 4 An example is provided illustrating intra-frame orientation modes with 65 prediction directions.

[0027] Figure 5 Empirical examples illustrate variations of the techniques according to embodiments of the present disclosure.

[0028] Figure 6 This is a diagram used to illustrate an embodiment of the RST according to the present disclosure.

[0029] Figure 7 This is a diagram illustrating a forward LFNST input area according to an embodiment of the present disclosure.

[0030] Figure 8 This is a diagram illustrating the input sequence of input data according to an embodiment of the present disclosure.

[0031] Figure 9 This is a diagram illustrating an input sequence of input data according to another embodiment of the present disclosure.

[0032] Figure 10 This is a diagram illustrating a scan sequence of transformation coefficients according to an embodiment of the present disclosure.

[0033] Figure 11An example of a video / image coding method according to an embodiment of the present disclosure is illustrated schematically.

[0034] Figure 12 An example of a video / image decoding method according to an embodiment of the present disclosure is illustrated schematically.

[0035] Figure 13 Examples of content streaming systems to which the embodiments disclosed in this disclosure can be applied are illustrated. Detailed Implementation

[0036] This document can be modified in various ways and can have various embodiments, and specific embodiments will be illustrated and described in detail in the accompanying drawings. However, this is not intended to limit the document to the specific embodiments. The terminology generally used in this specification is used to describe specific embodiments and not to limit the technical spirit of the document. Unless otherwise expressly indicated in the context, singular expressions include plural expressions. Terms such as "comprising" or "having" in this specification should be understood to indicate the presence of the features, numbers, steps, operations, elements, components or combinations thereof described in this specification, without excluding the possibility of the presence or addition of one or more features, numbers, steps, operations, elements, components or combinations thereof.

[0037] Furthermore, for ease of description in relation to different features and functions, the elements in the accompanying drawings described in this document are illustrated independently. This does not imply that the individual elements are implemented as separate hardware or separate software. For example, at least two elements may be combined to form a single element, or a single element may be divided into multiple elements. Embodiments in which elements are combined and / or separated are also included within the scope of the claims of this document, unless they depart from the spirit of this document.

[0038] Hereinafter, preferred embodiments of the document are described in more detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used for the same elements, and redundant descriptions of the same elements may be omitted.

[0039] Figure 1 Examples of video / image coding systems to which exemplary embodiments of this disclosure are applicable are illustrated schematically.

[0040] refer to Figure 1 A video / image encoding system may include a first device (source device) and a second device (receiving device). The source device may transmit encoded video / image information or data to the receiving device in the form of a file or stream via a digital storage medium or network.

[0041] The source device may include a video source, an encoding device, and a transmitter. The receiving device may include a receiver, a decoding device, and a renderer. The encoding device may be referred to as a video / image encoding device, and the decoding device may be referred to as a video / image decoding device. The transmitter may be included in the encoding device. The receiver may be included in the decoding device. The renderer may include a display, and the display may be configured as a separate device or an external component.

[0042] Video sources can be obtained through processes that capture, synthesize, or generate video / images. Video sources may include video / image capture devices and / or video / image generation devices. Video / image capture devices may include, for example, one or more cameras, video / image archives including previously captured video / images, etc. Video / image generation devices may include, for example, computers, tablets, and smartphones, and can generate video / images (electronically). For example, virtual video / images can be generated by computers, etc. In this case, the video / image capture process can be replaced by a process that generates related data.

[0043] Encoding devices can encode input video / images. They can perform a series of processes such as prediction, transformation, and quantization for compression and encoding efficiency. The encoded data (encoded video / image information) can be output as a bitstream.

[0044] A transmitter can send encoded video / image information or data, output as a bitstream, to a receiver in a receiving device via a digital storage medium or network, either as a file or a stream. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. The transmitter can include elements for generating media files according to a predetermined file format and may include elements for transmission over a broadcast / communication network. The receiver can receive / extract the bitstream and send the received bitstream to a decoding device.

[0045] Decoding devices can decode video / images by performing a series of processes such as dequantization, inverse transform, and prediction, which correspond to the operations of encoding devices.

[0046] The renderer can render decoded video / images. The rendered video / images can then be displayed on a monitor.

[0047] This disclosure relates to video / image coding. For example, the methods / exemplary implementations disclosed in this disclosure are applicable to methods disclosed in the Universal Video Coding (VVC) standard. Furthermore, the methods / exemplary implementations disclosed in this disclosure are applicable to methods disclosed in the Basic Video Coding (EVC) standard, the AOMedia Video 1 (AV1) standard, the Audio Video Coding 2 (AVS2) standard, or next-generation video / image coding standards (e.g., H.267 or H.268).

[0048] This disclosure presents various implementations of video / image encoding, and unless otherwise stated, these implementations can be combined with each other.

[0049] In this disclosure, video can refer to a series of images over time. An image generally refers to a unit representing an image within a specific time period, and slices and tiles are units that constitute part of an image during encoding. A slice / tile may include one or more Code Tree Units (CTUs). An image may consist of one or more slices / tiles. A tile is a rectangular region of a CTU within a specific tile column and a specific tile row in an image. A tile column is a rectangular region of a CTU having a height equal to the height of the image and a width specified by a syntax element in the image parameter set. A tile row is a rectangular region of a CTU having a height specified by a syntax element in the image parameter set and a width equal to the width of the image. A tile scan is a specific ordering of CTUs that partition an image, where CTUs are ordered consecutively by CTU raster scan within a tile, and tiles within an image are ordered consecutively by tile raster scan within the image. A slice comprises an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of an image that may be exclusively contained within a single NAL unit.

[0050] Furthermore, an image can be divided into two or more sub-images. A sub-image can be a rectangular region of one or more slices within the image.

[0051] A pixel or cell (pel) can refer to the smallest unit that makes up a picture (or image). Additionally, "sample" can be used as the term corresponding to a pixel. A sample can typically represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component.

[0052] A unit can represent the basic unit of image processing. A unit may include a specific region of an image and at least one of the information associated with that region. A unit may include a luminance block and two chrominance (e.g., cb, cr) blocks. In some cases, the term "unit" may be used interchangeably with terms such as "block" or "region". In general, an M×N block may include a set (or array) of samples (or transform coefficients) with M columns and N rows.

[0053] In this disclosure, "A or B" may mean "A only", "B only", or "both A and B". In other words, in this disclosure, "A or B" can be interpreted as "A and / or B". For example, in this disclosure, "A, B or C" may mean "A only", "B only", "C only", or "any combination of A, B and C".

[0054] The forward slash ( / ) or comma used in this disclosure can mean "and / or". For example, "A / B" can mean "A and / or B". Therefore, "A / B" can mean "A only", "B only", or "both A and B". For example, "A, B, C" can mean "A, B, or C".

[0055] In this disclosure, "at least one of A and B" can mean "only A", "only B" or "both A and B". Furthermore, in this disclosure, the expression "at least one of A or B" or "at least one of A and / or B" can be interpreted as the same as "at least one of A and B".

[0056] Additionally, in this disclosure, "at least one of A, B, and C" may mean "only A," "only B," "only C," or "any combination of A, B, and C." Furthermore, "at least one of A, B, or C" or "at least one of A, B, and / or C" may mean "at least one of A, B, and C."

[0057] Additionally, the brackets used in this disclosure may mean "for example". Specifically, if "prediction (intra-frame prediction)" is indicated, "intra-frame prediction" may be proposed as an example of "prediction". In other words, "prediction" in this disclosure may not be limited to "intra-frame prediction", and "intra-frame prediction" may be proposed as an example of "prediction". Furthermore, even if "prediction (i.e., intra-frame prediction)" is indicated, "intra-frame prediction" may be proposed as an example of "prediction".

[0058] In this disclosure, a technical feature described individually in a single figure may be implemented individually or simultaneously.

[0059] Figure 2 This diagram is used to schematically illustrate the configuration of a video / image encoding apparatus to which exemplary embodiments of this disclosure are applicable. Hereinafter, the encoding apparatus may include an image encoding apparatus and / or a video encoding apparatus.

[0060] refer to Figure 2The encoding device 200 includes an image partitioner 210, a predictor 220, a residual processor 230, an entropy encoder 240, an adder 250, a filter 260, and a memory 270. The predictor 220 may include an inter-frame predictor 221 and an intra-frame predictor 222. The residual processor 230 may include a transformer 232, a quantizer 233, a dequantizer 234, and an inverse transformer 235. The residual processor 230 may also include a subtractor 231. The adder 250 may be referred to as a reconstructor or a reconstruction block generator. According to embodiments, the image partitioner 210, predictor 220, residual processor 230, entropy encoder 240, adder 250, and filter 260 may be configured by at least one hardware component (e.g., an encoder chipset or processor). Additionally, the memory 270 may include a decoded picture buffer (DPB) or may be configured by a digital storage medium. The hardware component may also include the memory 270 as an internal / external component.

[0061] Image partitioner 210 can partition an input image (or picture or frame) input to encoding device 200 into one or more processing units. For example, a processing unit may be referred to as a coding unit (CU). In this case, the coding unit can be recursively partitioned from coding tree unit (CTU) or maximum coding unit (LCU) according to a quadtree-binary-trinary tree (QTBTTT) structure. For example, a coding unit can be partitioned into multiple deeper coding units based on a quadtree structure, a binary tree structure, and / or a ternary structure. In this case, for example, a quadtree structure can be applied first, and a binary tree structure and / or a ternary structure can be applied later. Alternatively, a binary tree structure can be applied first. The encoding process according to this disclosure can be performed based on the final coding unit that is no longer partitioned. In this case, based on the image characteristics and encoding efficiency, the maximum coding unit can be used as the final coding unit, or if necessary, the coding unit can be recursively partitioned into deeper coding units and the coding unit with the optimal size can be used as the final coding unit. Here, the encoding process may include prediction, transformation, and reconstruction processes (described later). As another example, the processor may also include a prediction unit (PU) or a transform unit (TU). In this case, the prediction unit and the transform unit can be split or partitioned from the final coding unit described above. The prediction unit may be a unit for predicting samples, and the transform unit may be a unit for deriving transform coefficients and / or a unit for deriving residual signals from transform coefficients.

[0062] In some cases, a unit can be used interchangeably with terms such as a block or region. Generally, an M×N block can represent a set of samples or transform coefficients consisting of M columns and N rows. A sample can typically represent a pixel or pixel value, and can represent only the pixel / pixel value of the luminance component or only the pixel / pixel value of the chrominance component. A sample can be used as a term corresponding to a picture (or image) of pixels or cells.

[0063] In the encoding device 200, a residual signal (residual block, residual sample array) is generated by subtracting the prediction signal (prediction block, prediction sample array) output from the inter-frame predictor 221 or the intra-frame predictor 222 from the input image signal (original block, original sample array), and the generated residual signal is sent to the converter 232. In this case, as shown, the unit in the encoder 200 that subtracts the prediction signal (prediction block, prediction sample array) from the input image signal (original block, original sample array) can be called the subtractor 231. The predictor can perform prediction on the block to be processed (hereinafter referred to as the current block) and generate a prediction block that includes the prediction samples of the current block. The predictor can determine whether to apply intra-frame prediction or inter-frame prediction based on the current block or CU. As described later in the description of the various prediction modes, the predictor can generate various types of information related to the prediction (such as prediction mode information) and send the generated information to the entropy encoder 240. The information about the prediction can be encoded in the entropy encoder 240 and output as a bitstream.

[0064] Intra-predictor 222 can predict the current block by referencing samples in the current image. Depending on the prediction mode, the referenced samples can be located near or separated from the current block. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. For example, non-directional modes can include DC mode and planar mode. For example, depending on the level of detail in the prediction direction, directional modes can include 33 or 65 directional prediction modes. However, this is just an example, and more or fewer directional prediction modes can be used depending on the settings. Intra-predictor 222 can determine the prediction mode applied to the current block by using prediction modes applied to neighboring blocks.

[0065] Inter-frame predictor 221 can derive the predicted block of the current block based on the reference block (reference sample array) specified by the motion vector on the reference image. Here, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation between the motion information of neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. The reference image including the reference block and the reference image including the temporally neighboring block may be the same or different. The temporally neighboring block may be referred to as a juxtaposed reference block, a juxtaposed CU (colCU), etc., and the reference image including the temporally neighboring block may be referred to as a juxtaposed image (colPic). For example, inter-frame predictor 221 can configure a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference image index of the current block. Inter-frame prediction can be performed based on various prediction modes. For example, in skip mode and merge mode, the inter-frame predictor 221 can use motion information from neighboring blocks as motion information for the current block. In skip mode, unlike merge mode, residual signals may not be sent. In motion vector prediction (MVP) mode, motion vectors from neighboring blocks can be used as motion vector prediction terms, and the motion vector of the current block can be identified by signaling the motion vector difference.

[0066] Predictor 220 can generate prediction signals based on various prediction methods described below. For example, the predictor can apply not only intra-frame prediction or inter-frame prediction to predict a block, but also both intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined inter-frame and intra-frame prediction (CIIP). Alternatively, the predictor can predict blocks based on intra-block copy (IBC) prediction mode or palette mode. IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction in the current picture, but can be performed similarly to inter-frame prediction, such that a reference block is derived in the current picture. That is, IBC can use at least one of the inter-frame prediction techniques described in this disclosure. Palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When applying palette mode, sample values ​​within the picture can be signaled based on information about the palette table and palette index.

[0067] The predicted signal generated by the predictor (including inter-frame predictor 221 and / or intra-frame predictor 222) can be used to generate a reconstructed signal or a residual signal. Transformer 232 can generate transform coefficients by applying transform techniques to the residual signal. For example, the transform technique can include at least one of Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), Graphical Based Transform (GBT), or Conditional Nonlinear Transform (CNT). Here, when the relationship information between pixels is represented graphically, GBT refers to a transform obtained from a graphic. CNT refers to a transform generated based on the predicted signal generated using all previously reconstructed pixels. Furthermore, the transform processing can be applied to square pixel blocks of the same size or to blocks of variable size other than square.

[0068] Quantizer 233 can quantize the transform coefficients and send them to entropy encoder 240, which can encode the quantized signal (information about the quantized transform coefficients) and output a bitstream. The information about the quantized transform coefficients can be referred to as residual information. Quantizer 233 can rearrange the block-type quantized transform coefficients into a one-dimensional vector form based on the coefficient scan order, and generate information about the quantized transform coefficients based on the one-dimensional vector form. Entropy encoder 240 can perform various encoding methods such as exponential Golomb, context-adaptive variable-length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). Entropy encoder 240 can encode information required for video / image reconstruction other than the quantized transform coefficients (e.g., values ​​of syntax elements, etc.) together or separately. The encoded information (e.g., encoded video / image information) can be sent or stored in NAL (Network Abstraction Layer) units in the form of a bitstream. The video / image information may also include information about various parameter sets, such as Adaptive Parameter Set (APS), Picture Parameter Set (PPS), Sequence Parameter Set (SPS), or Video Parameter Set (VPS). Additionally, the video / image information may include general constraint information. In this disclosure, information and / or syntax elements transmitted from the encoding device / notified by signal to the decoding device may be included in the video / image information. The video / image information may be encoded by the encoding process described above and included in a bitstream. The bitstream may be transmitted over a network or stored in a digital storage medium. The network may include broadcast networks and / or communication networks, and the digital storage medium may include various storage media such as USB, SD, CD, DVD, Blu-ray, HDD, SSD, etc. A transmitter (not shown) that transmits the signal output from the entropy encoder 240 and / or a storage unit (not shown) that stores the signal may be included as an internal / external element of the encoding device 200, and alternatively, the transmitter may be included in the entropy encoder 240.

[0069] The quantized transform coefficients output from quantizer 233 can be used to generate a prediction signal. For example, the residual signal (residual block or residual sample) can be reconstructed by applying dequantization and inverse transform to the quantized transform coefficients via dequantizer 234 and inverse transformer 235. Adder 250 adds the reconstructed residual signal to the prediction signal output from inter-frame predictor 221 or intra-frame predictor 222 to generate a reconstructed signal (reconstructed image, reconstructed block, reconstructed sample array). If the block to be processed does not have a residual (such as in the case of applying a skip mode), the prediction block can be used as a reconstructed block. Adder 250 can be referred to as a reconstructor or reconstructed block generator. As described below, the generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image and can be filtered for inter-frame prediction of the next image.

[0070] In addition, Luminance Mapping and Chromatography Scaling (LMCS) can be applied during image encoding and / or reconstruction.

[0071] Filter 260 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 260 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 270 (specifically, the DPB of memory 270). Various filtering methods may include deblocking filtering, sample adaptive offset, adaptive loop filtering, bilateral filtering, etc. Filter 260 can generate various types of filtering-related information and send the generated information to entropy encoder 240, as described later in the description of the various filtering methods. The filtering-related information can be encoded by entropy encoder 240 and output as a bitstream.

[0072] The modified reconstructed image sent to memory 270 can be used as a reference image in inter-frame predictor 221. When inter-frame prediction is applied via the encoding device, prediction mismatch between the encoding device 200 and the decoding device can be avoided, and encoding efficiency can be improved.

[0073] The DPB of memory 270 can store a reconstructed image modified for use as a reference image in inter-frame predictor 221. Memory 270 can store motion information of blocks from which motion information in the current image is derived (or encoded) and / or motion information of already reconstructed blocks in the image. The stored motion information can be sent to inter-frame predictor 221 and used as motion information for spatially or temporally neighboring blocks. Memory 270 can store reconstructed samples of reconstructed blocks in the current image and can transmit these reconstructed samples to intra-frame predictor 222.

[0074] Figure 3This diagram schematically illustrates the configuration of a video / image decoding device to which exemplary embodiments of this disclosure are applicable. Hereinafter, the decoding device may include an image decoding device and / or a video decoding device.

[0075] refer to Figure 3 The decoding device 300 may include an entropy decoder 310, a residual processor 320, a predictor 330, an adder 340, a filter 350, and a memory 360. The predictor 330 may include an inter-frame predictor 331 and an intra-frame predictor 332. The residual processor 320 may include a dequantizer 321 and an inverse transformer 321. According to embodiments, the entropy decoder 310, residual processor 320, predictor 330, adder 340, and filter 350 may be configured by hardware components (e.g., a decoder chipset or processor). Additionally, the memory 360 may include a decoded picture buffer (DPB) and may be configured by a digital storage medium. The hardware components may also include the memory 360 as an internal / external component.

[0076] When the input includes a bitstream containing video / image information, the decoding device 300 can interact with... Figure 2 The processing of video / image information in the encoding device correspondingly reconstructs the image. For example, the decoding device 300 can derive units / blocks based on block partitioning information obtained from the bitstream. The decoding device 300 can use a processor applied in the encoding device to perform decoding. Thus, for example, the processor for decoding can be an encoding unit, and the encoding unit can be partitioned from encoding tree units or maximum encoding units according to a quadtree structure, binary tree structure, and / or ternary tree structure. One or more transform units can be derived from the encoding unit. The reconstructed image signal decoded and output by the decoding device 300 can be reproduced by a reproduction device.

[0077] Decoding device 300 can receive from Figure 2The encoding device outputs a signal in the form of a bitstream, and the received signal can be decoded by the entropy decoder 310. For example, the entropy decoder 310 can parse the bitstream to derive information (e.g., video / image information) required for image reconstruction (or picture reconstruction). The video / image information may also include information about various parameter sets, such as adaptive parameter sets (APS), picture parameter sets (PPS), sequence parameter sets (SPS), or video parameter sets (VPS). In addition, the video / image information may also include general constraint information. The decoding device can also decode the picture based on the information about the parameter sets and / or general constraint information. The signaling / receiving information and / or syntax elements described later in this disclosure can be decoded and obtained from the bitstream through the decoding process. For example, the entropy decoder 310 decodes the information in the bitstream based on encoding methods such as exponential Golomb coding, CAVLC, or CABAC, and outputs the quantized values ​​of the syntax elements and transform coefficients of the residuals required for image reconstruction. More specifically, the CABAC entropy decoding method can receive bins corresponding to each syntax element in the bitstream, determine the context model using information about the target syntax element, decoding information about the target block, or information about symbols / bins decoded in a previous stage, and perform arithmetic decoding on the cells by predicting the probability of bin occurrence based on the determined context model, generating symbols corresponding to the values ​​of each syntax element. In this case, the CABAC entropy decoding method can update the context model after determining the context model by using the information of the decoded symbols / bins for the context model of the next symbol / bin. The prediction-related information in the information decoded by the entropy decoder 310 can be provided to the predictors (inter-frame predictor 332 and intra-frame predictor 331), and the residual values ​​(i.e., quantized transform coefficients and related parameter information) from the entropy decoder 310 can be input to the residual processor 320. The residual processor 320 can derive residual signals (residual blocks, residual samples, residual sample arrays). Additionally, the filtering information in the information decoded by the entropy decoder 310 can be provided to the filter 350. Furthermore, the receiver (not shown) for receiving the signal output from the encoding device can also be configured as an internal / external element of the decoding device 300, or the receiver can be a component of the entropy decoder 310. Additionally, the decoding device according to this disclosure can be referred to as a video / image / picture decoding device, and the decoding device can be classified as an information decoder (video / image / picture information decoder) and a sample decoder (video / image / picture sample decoder). The information decoder may include the entropy decoder 310, and the sample decoder may include at least one of a dequantizer 321, an inverse transformer 322, an adder 340, a filter 350, a memory 360, an inter-frame predictor 332, and an intra-frame predictor 331.

[0078] Dequantizer 321 can dequantize the quantized transform coefficients and output the transform coefficients. Dequantizer 321 can rearrange the quantized transform coefficients in a two-dimensional block format. In this case, the rearrangement can be performed based on the coefficient scan order performed in the encoding device. Dequantizer 321 can perform dequantization on the quantized transform coefficients using quantization parameters (e.g., quantization step size information) and obtain the transform coefficients.

[0079] The inverse transformer 322 performs an inverse transformation on the transformation coefficients to obtain the residual signal (residual block, residual sample array).

[0080] Predictor 330 can perform prediction on the current block and generate a prediction block that includes prediction samples of the current block. Predictor 330 can determine whether to apply intra-frame prediction or inter-frame prediction to the current block based on information about the prediction output from entropy decoder 310, and can determine a specific intra-frame / inter-frame prediction mode.

[0081] Predictor 330 can generate prediction signals based on various prediction methods described below. For example, predictor 330 can not only apply intra-frame prediction or inter-frame prediction to predict a block, but can also apply intra-frame prediction and inter-frame prediction simultaneously. This can be referred to as combined intra-frame and inter-frame prediction (CIIP). Additionally, predictor 330 can predict blocks based on intra-frame block copy (IBC) prediction mode or palette mode. IBC prediction mode or palette mode can be used for content image / video coding such as screen content coding (SCC) in games, etc. IBC essentially performs prediction in the current frame, but can be performed similarly to inter-frame prediction, such that a reference block is derived in the current frame. That is, IBC can use at least one of the inter-frame prediction techniques described in this disclosure. Palette mode can be considered as an example of intra-frame coding or intra-frame prediction. When applying palette mode, sample values ​​within the frame can be signaled based on information about the palette table and palette index.

[0082] Intra-predictor 331 can predict the current block by referencing samples in the current image. Depending on the prediction mode, the referenced samples can be located in the neighbors of the current block or far from the current block. In intra-prediction, the prediction mode can include multiple non-directional modes and multiple directional modes. Intra-predictor 331 can determine the prediction mode applied to the current block by using the prediction modes applied to neighboring blocks.

[0083] Inter-frame predictor 332 can derive the predicted block of the current block based on the reference block (reference sample array) specified by the motion vector on the reference image. In this case, to reduce the amount of motion information transmitted in inter-frame prediction mode, motion information can be predicted on a block, sub-block, or sample basis based on the correlation of motion information between neighboring blocks and the current block. Motion information may include motion vectors and reference image indices. Motion information may also include inter-frame prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter-frame prediction, neighboring blocks may include spatially neighboring blocks existing in the current image and temporally neighboring blocks existing in the reference image. For example, inter-frame predictor 332 can configure a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference image index of the current block based on the received candidate selection information. Inter-frame prediction can be performed based on various prediction modes, and the information about the prediction may include information indicating the inter-frame prediction mode of the current block.

[0084] Adder 340 can generate a reconstruction signal (reconstructed image, reconstruction block, reconstruction sample array) by adding the obtained residual signal to the prediction signal (prediction block, prediction sample array) output from the predictor (including inter-frame predictor 332 and / or intra-frame predictor 331). If the block to be processed does not have a residual (such as when a skip mode is applied), the prediction block can be used as a reconstruction block.

[0085] Adder 340 can be referred to as a reconstructor or reconstruction block generator. The generated reconstructed signal can be used for intra-frame prediction of the next block to be processed in the current image, or it can be filtered and output as described below, or it can be used for inter-frame prediction of the next image.

[0086] In addition, Luminance Mapping and Chromaticity Scaling (LMCS) can be applied in image decoding processing.

[0087] Filter 350 can improve subjective / objective image quality by applying filtering to the reconstructed signal. For example, filter 350 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image and store the modified reconstructed image in memory 360 (specifically, the DPB of memory 360). For example, various filtering methods may include deblocking filtering, sample adaptive shifting, adaptive loop filtering, bilateral filtering, etc.

[0088] The (modified) reconstructed image stored in the DPB of memory 360 can be used as a reference image in inter-frame predictor 332. Memory 360 can store motion information of blocks from which motion information in the current image is derived (or decoded) and / or motion information of already reconstructed blocks in the image. The stored motion information can be sent to inter-frame predictor 332 as motion information for spatially or temporally neighboring blocks. Memory 360 can store reconstructed samples of reconstructed blocks in the current image and transmit the reconstructed samples to intra-frame predictor 331.

[0089] In this disclosure, the embodiments described in the filter 260, inter-frame predictor 221, and intra-frame predictor 222 of the encoding device 200 can be applied in the same way as or respectively corresponding to the filter 350, inter-frame predictor 332, and intra-frame predictor 331 of the decoding device 300. This can also be applied to the inter-frame predictor 332 and the intra-frame predictor 331.

[0090] As described above, prediction is performed during video encoding to improve compression efficiency. A prediction block, comprising predicted samples for the current block (i.e., the target coding block), can be generated through prediction. In this case, the prediction block includes predicted samples in the spatial domain (or pixel domain). The prediction block is derived identically from both the encoding and decoding devices. The encoding device can improve image coding efficiency by signaling the decoding device with information about the residual (residual information) between the original block and the prediction block, rather than the original sample values ​​of the original block itself. The decoding device can derive a residual block including residual samples based on the residual information, generate a reconstructed block including reconstructed samples by adding the residual block to the prediction block, and generate a reconstructed image including the reconstructed block.

[0091] Residual information can be generated through transform and quantization processes. For example, an encoding device can derive a residual block between the original block and the prediction block, derive transform coefficients by performing a transform process on the residual samples (residual sample array) included in the residual block, derive quantized transform coefficients by performing a quantization process on the transform coefficients, and signal the relevant residual information (via bitstream) to the decoding device. In this case, the residual information may include information such as the values ​​of the quantized transform coefficients, their positions, the transform scheme, the transform kernel, and the quantization parameters. The decoding device can perform dequantization / inverse transform processes based on the residual information and derive residual samples (or residual blocks). The decoding device can generate a reconstructed image based on the prediction block and the residual block. Furthermore, the encoding device can derive residual blocks by performing dequantization / inverse transform on the quantized transform coefficients for inter-frame prediction reference in subsequent images and can generate a reconstructed image.

[0092] In this disclosure, at least one of quantization / dequantization and / or transformation / inverse transformation may be omitted. If quantization / dequantization is omitted, the transformation coefficients of the quantization may be referred to as transformation coefficients. If transformation / inverse transformation is omitted, the transformation coefficients may also be referred to as coefficients or residual coefficients, or, for consistency of representation, may still be referred to as transformation coefficients.

[0093] Furthermore, in this disclosure, the quantized transform coefficients and the transform coefficients can be referred to as transform coefficients and scaling transform coefficients, respectively. In this case, the residual information can include information about the transform coefficients, and this information can be signaled via residual coding syntax. The transform coefficients can be derived based on the residual information (or the information about the transform coefficients), and the scaling transform coefficients can be derived via the inverse transform (scaling) of the transform coefficients. Residual samples can be derived based on the inverse transform (scaling) of the scaling transform coefficients. This can also be applied to / expressed in other parts of this disclosure.

[0094] Intra-frame prediction can be used to identify predictions for generating prediction samples for the current block based on reference samples within the image to which the current block belongs (hereinafter referred to as the current image). When intra-frame prediction is applied to the current block, neighboring reference samples to be used for intra-frame prediction of the current block can be derived. The neighboring reference samples of the current block include a total of 2×nH samples adjacent to the left boundary and the lower left neighbor of the current block of size nW×nH, a total of 2×nW samples adjacent to the top boundary and the upper right neighbor of the current block, and 1 sample adjacent to the upper left neighbor of the current block. Alternatively, the neighboring reference samples of the current block may include multiple columns of upper neighbor samples and multiple rows of left neighbor samples. In addition, the neighboring reference samples of the current block include a total of nH samples adjacent to the right boundary of the current block of size nW×nH, a total of nW samples adjacent to the lower boundary of the current block, and 1 sample adjacent to the lower right neighbor of the current block.

[0095] However, some neighboring reference samples of the current block may not have been decoded or may not be available. In this case, the decoder can construct neighboring reference samples for prediction by replacing unavailable samples with available samples. Alternatively, neighboring reference samples to be used for prediction can be constructed by interpolation of available samples.

[0096] When deriving neighboring reference samples, (i) the predicted sample can be derived based on the average or interpolation of the neighboring reference samples of the current block, or (ii) the predicted sample can be derived based on a reference sample that exists in a specific (predictive) direction relative to the predicted sample among the neighboring reference samples of the current block. Case (i) can be referred to as non-directional mode or non-angular mode, and case (ii) can be referred to as directional mode or angular mode.

[0097] Alternatively, prediction samples can be generated by interpolation between the first neighboring sample in the prediction direction of the intra-prediction mode of the current block relative to the prediction sample in the neighboring reference samples and the second neighboring sample in the opposite direction to the prediction direction. This can be called linear interpolation intra-prediction (LIP). Furthermore, chroma prediction samples can be generated based on luminance samples using a linear model (LM). This can be called LM mode or chroma component LM (CCLM) mode.

[0098] Additionally, a temporary prediction sample for the current block is derived based on filtered neighboring reference samples. This prediction sample can then be derived by weighted summing of the temporary prediction sample and at least one reference sample derived from the existing neighboring reference samples (i.e., unfiltered neighboring reference samples) according to the intra-prediction mode. This process can be referred to as Position-Related Intra-Prediction (PDPC).

[0099] Alternatively, intra-frame prediction coding can be performed by selecting the reference sample line with the highest prediction accuracy among multiple reference sample lines adjacent to the current block, deriving the prediction sample using the reference sample in the prediction direction of the corresponding line, and at this time, indicating (signaling) the reference sample line used to the decoding device. This can be referred to as multi-reference line intra-frame prediction or MRL-based intra-frame prediction.

[0100] Alternatively, intra-prediction can be performed based on the same intra-prediction mode by dividing the current block into vertical or horizontal sub-partitions, but neighboring reference samples can be derived and used on a sub-partition basis. That is, in this case, the intra-prediction mode for the current block is applied equally to the sub-partitions, but in some cases, intra-prediction performance can be improved by deriving and using neighboring reference samples on a sub-partition basis. This prediction method can be called intra-prediction based on intra-partition sub-partitions (ISP).

[0101] The intra-prediction methods described above can be referred to as intra-prediction types to distinguish them from intra-prediction modes. Intra-prediction types can be referred to by various terms such as intra-prediction techniques or additional intra-prediction modes. For example, an intra-prediction type (or additional intra-prediction mode, etc.) can include at least one of LIP, PDPC, MRL, or ISP as described above. General intra-prediction methods other than specific intra-prediction types such as LIP, PDPC, MRL, and ISP can be referred to as normal intra-prediction types. When not applying the specific intra-prediction types described above, normal intra-prediction types can generally be applied, and prediction can be performed based on the intra-prediction modes described above. Furthermore, post-processing filtering can be applied to the derived prediction samples as needed.

[0102] Furthermore, as described above, the intra-prediction modes may include two non-directional intra-prediction modes and 65 directional prediction modes. The non-directional intra-prediction modes may include a planar intra-prediction mode and a DC intra-prediction mode, and the directional intra-prediction modes may include intra-prediction modes numbered 2 to 66.

[0103] Figure 4 An example is provided illustrating intra-frame orientation modes with 65 prediction directions.

[0104] refer to Figure 4 The system can distinguish between intra-prediction modes with horizontal orientation and intra-prediction modes with vertical orientation centered on intra-prediction mode 34, which has a top-left diagonal prediction direction. That is, intra-prediction modes 2 to 33 are horizontal, and intra-prediction modes 34 to 66 are vertical. Intra-prediction modes 18 and 50 represent horizontal and vertical intra-prediction modes, respectively. Intra-prediction mode 2 can represent a bottom-left diagonal intra-prediction mode, intra-prediction mode 34 can represent a top-left diagonal intra-prediction mode, and intra-prediction mode 66 can represent a top-left diagonal intra-prediction mode. Non-directional intra-prediction modes can include the planar intra-prediction mode (mode number 0) and the DC intra-prediction mode (mode number 1).

[0105] As mentioned above, the prediction direction of intra-frame prediction can be defined as 45 degrees to -135 degrees clockwise. However, when the current block is a non-square block, some existing directional intra-frame prediction modes can be adaptively replaced by the Wide Angle Intra-Frame Prediction (WAIP) mode.

[0106] According to the example, in addition to the 65 directional intra-prediction modes mentioned above, intra-prediction modes may also include wide-angle intra-prediction modes.

[0107] When applying wide-angle intra-prediction mode, information about existing intra-prediction modes is signaled, and after parsing the information about intra-prediction modes, this information is remapped to the index of the wide-angle intra-prediction mode. Therefore, the total number of intra-prediction modes for a specific block (e.g., a non-square block of a specific size) can remain unchanged; that is, the total number of intra-prediction modes is 67, and the encoding of the intra-prediction modes for a specific block can remain unchanged.

[0108] Figure 5 The variations of the techniques according to the embodiments of the present disclosure are illustrated schematically.

[0109] refer to Figure 5 The converter can correspond to the above. Figure 2 The converter in the encoding device, and the inverse converter can correspond to the above. Figure 2The inverse converter in the encoding device, or corresponding to Figure 3 The inverse converter in the decoding device.

[0110] The transformer can derive (primary) transform coefficients (S510) by performing a primary transform based on residual samples (residual sample array) in the residual block. This primary transform can be referred to as the core transform. In this paper, the primary transform can be based on multiple transform selection (MTS), and when the multiple transform is applied as the primary transform, it can be referred to as a multi-core transform.

[0111] Multi-core transform can represent a method of transforming a signal by additionally using Discrete Cosine Transform (DCT) Type 2 and Discrete Sine Transform (DST) Type 7, DCT Type 8, and / or DST Type 1. In other words, multi-core transform can represent a method of transforming a spatial domain residual signal (or residual block) into frequency domain transform coefficients (or primary transform coefficients) based on multiple transform kernels selected from DCT Type 2, DST Type 7, DCT Type 8, and DST Type 1. In this paper, DCT Type 2, DST Type 7, DCT Type 8, and DST Type 1 can be referred to as transform types, transform kernels, or transform cores. These DCT / DST transform types can be defined based on basis functions.

[0112] When performing multi-core transformation, a vertical transformation kernel and a horizontal transformation kernel can be selected from the transformation kernels for the target block. A vertical transformation can be performed on the target block based on the vertical transformation kernel, and a horizontal transformation can be performed on the target block based on the horizontal transformation kernel. Here, the horizontal transformation can represent the transformation of the horizontal component of the target block, while the vertical transformation can represent the transformation of the vertical component of the target block.

[0113] As an example, if a primary transform is performed by applying an MTS, the mapping of the transform kernels can be set by setting specific basis functions to predetermined values ​​and combining the basis functions to be applied in the vertical or horizontal transform. For example, when the horizontal transform kernel is expressed as trTypeHor and the vertical transform kernel as trTypeVer, the value of trTypeHor or trTypeVer of 0 can be set to DCT2, the value of trTypeHor or trTypeVer of 1 can be set to DST-7, and the value of trTypeHor or trTypeVer of 2 can be set to DCT-8.

[0114] In this scenario, the MTS index information can be encoded and signaled to the decoding device to identify any one of the multiple transform kernel sets. For example, MTS index 0 can indicate that both trTypeHor and trTypeVer values ​​are 0, MTS index 1 can indicate that both trTypeHor and trTypeVer values ​​are 1, MTS index 2 can indicate that trTypeHor value is 2 and trTypeVer value is 1, MTS index 3 can indicate that trTypeHor value is 1 and trTypeVer value is 2, and MTS index 4 can indicate that both trTypeHor and trTypeVer values ​​are 2.

[0115] In one example, the set of transformation kernels based on MTS index information is illustrated in the table below.

[0116] [Table 1]

[0117] The transformer can derive modified (secondary) transform coefficients by performing a secondary transform based on the (primary) transform coefficients (S520). The primary transform is a transform from the spatial domain to the frequency domain, while the secondary transform refers to a transform performed using the correlations existing between the (primary) transform coefficients to achieve a more compressed expression. Secondary transforms can include non-separable transforms. In this case, the secondary transform can be referred to as a non-separable secondary transform (NSST) or a mode-dependent non-separable secondary transform (MDNSST).

[0118] An inseparable secondary transform can refer to a transform that generates modified transform coefficients (or secondary transform coefficients) of the residual signal by performing a secondary transform on the (first-order) transform coefficients derived from the primary transform based on an inseparable transform matrix. Here, based on the inseparable transform matrix, the transform can be applied to the (primary) transform coefficients in one step without separately applying vertical and horizontal transforms (or independently applying horizontal and vertical transforms). For example, after rearranging a two-dimensional signal (transform coefficients) into a one-dimensional signal through a specific predetermined direction (e.g., row-priority direction or column-priority direction), the inseparable secondary transform can derive the modified transform coefficients (or secondary transform coefficients) based on matrix operations between a one-dimensional vector and the inseparable transform matrix.

[0119] For example, row priority arranges the M×N blocks in the order of row 1, row 2, ..., row N, while column priority arranges the M×N blocks in the order of column 1, column 2, ..., column M. In other words, for inseparable secondary transformations, the transformation coefficients (derived from the primary transformation) can be arranged into a one-dimensional vector according to the row-principal direction, and then matrix operations can be performed; alternatively, the transformation coefficients can be arranged into a one-dimensional vector according to the column-principal direction, and then matrix operations can be performed.

[0120] Inseparable secondary transformations can be applied to the upper-left region of a block composed of (primary) transformation coefficients (hereinafter referred to as a transformation coefficient block or transformation block). For example, when the width (W) and height (H) of the transformation coefficient block are both 8 or greater, an 8×8 inseparable secondary transformation can be applied to the upper-left 8×8 region of the transformation coefficient block. Alternatively, when the width (W) and height (H) of the transformation coefficient block are both 4 or greater, and the width (W) or height (H) of the transformation coefficient block is less than 8, a 4×4 inseparable secondary transformation can be applied to the upper-left min(8,W)×min(8,H) region of the transformation coefficient block. However, the embodiments are not limited to this; for example, even if only the condition that the width (W) or height (H) of the transformation coefficient block is 4 or greater is met, a 4×4 inseparable secondary transformation can be applied to the upper-left min(8,W)×min(8,H) region of the transformation coefficient block. In summary, depending on the size of the transformation block, inseparable secondary transformations can be applied to the upper-left 4×4 or 8×8 region of the transformation block. Based on the example, a transformation on the top-left 4×4 region can be called a 4×4 transformation, and a transformation on the top-left 8×8 region can be called an 8×8 transformation.

[0121] Here, to select the transform kernels, two inseparable secondary transform kernels can be configured per transform set for both 8×8 and 4×4 transforms, and four transform sets can exist. That is, four transform sets can be configured for 8×8 transforms, and four transform sets can be configured for 4×4 transforms. In this case, each transform set in the four transform sets for 8×8 transforms can include two 8×8 transform kernels, and each transform set in the four transform sets for 4×4 transforms can include two 4×4 transform kernels.

[0122] However, as the size of the transformation (i.e., the size of the region to which the transformation is applied) can be, for example, a size other than 8×8 or 4×4, the number of sets can be n, and the number of transformation kernels in each set can be k.

[0123] The transform set can be referred to as the NSST set or the LFNST set. A specific set of transforms can be selected, for example, based on the intra-prediction mode of the current block (CU or sub-block). The Low-Frequency Inseparable Transform (LFNST) can be an example of a reduced inseparable transform, which will be described later, and can represent an inseparable transform for low-frequency components.

[0124] As an example, the mapping of four transform sets according to the intra-prediction mode can be represented, for example, as shown in Table 2 below.

[0125] [Table 2]

[0126] As shown in Table 2, the intra-frame prediction mode can be mapped to any of the four transform sets, that is, lfnstTrSetIdx can be mapped to any of 0 to 3.

[0127] On the other hand, when a specific set is determined to be used for an inseparable transform, one of the k transform kernels in the specific set can be selected using the inseparable secondary transform index. The encoding device can derive the inseparable secondary transform index identifying the specific transform kernel based on rate-distortion (RD) check and can signal the inseparable secondary transform index to the decoding device. The decoding device can select one of the k transform kernels in the specific set based on the inseparable secondary transform index. For example, lfnst index 0 can refer to the first inseparable secondary transform kernel, lfnst index 1 can refer to the second inseparable secondary transform kernel, and lfnst index 2 can refer to the third inseparable secondary transform kernel. Alternatively, lfnst index 0 can indicate that the first inseparable secondary transform is not applied to the target block, and lfnst indexes 1 to 3 can indicate the three transform kernels.

[0128] The converter can perform an inseparable secondary transformation based on the selected transform core and obtain modified (secondary) transform coefficients. As described above, the modified transform coefficients can be derived as transform coefficients quantized by a quantizer, and can be encoded and sent as a signal to the decoding device, and transmitted to the dequantizer / inverse converter in the encoding device.

[0129] Furthermore, as described above, if the secondary transformation is omitted, the (primary) transform coefficients, which are the output of the primary (separable) transform, can be derived as transform coefficients quantized by the quantizer as described above, and can be encoded and sent to the decoding device as a signal, and transmitted to the dequantizer / inverse transformer in the encoding device.

[0130] The inverse transformer can perform a series of processes in the reverse order of those already executed in the aforementioned transformers. The inverse transformer can receive (dequantized) transform coefficients and derive (primary) transform coefficients by performing a secondary (inverse) transform (S550), and obtain residual blocks (residual samples) by performing a primary (inverse) transform on the (primary) transform coefficients (S560). In this regard, from the perspective of the inverse transformer, the primary transform coefficients can be referred to as modified transform coefficients. As described above, the encoding and decoding devices can generate reconstructed blocks based on the residual blocks and prediction blocks, and can generate reconstructed images based on the reconstructed blocks.

[0131] The inverse transformer derives modified transform coefficients by applying a transform kernel matrix to the (inverse-quantized) transform coefficients of the transform, which are arranged in a specific order (e.g., a diagonal scan order, specifically, a diagonal scan order starting from the upper left of the transform block and proceeding in the lower right direction). Depending on the direction in which the transform coefficients are read for the secondary transform in the transform unit (i.e., row-major or column-major), the modified transform coefficients can be arranged in a two-dimensional manner in the upper left region of the transform block. When the transform unit performs a 4×4 transform, the inverse transformer can arrange the modified transform coefficients in the 4×4 region of the transform block in two dimensions; when the transform unit performs an 8×8 transform, the inverse transformer can align the modified transform coefficients in the 8×8 region of the transform block in two dimensions.

[0132] Furthermore, the inverse secondary transform can be NSST, reduced secondary transform (RST), or LFNST, and whether to apply the inverse secondary transform can be determined based on the secondary transform flag parsed from the bitstream. As another example, the application of the inverse secondary transform can be determined based on the transform coefficients of the residual block.

[0133] The secondary inverse transform (i.e., the transform kernel, transform matrix, or transform kernel matrix) can be determined based on the LFNST (NSST or RST) transform set specified according to the intra-prediction mode. Furthermore, as an example, the secondary transform determination method can be determined based on the primary transform determination method. Depending on the intra-prediction mode, various combinations of the primary and secondary transforms can be determined. Additionally, for example, the region where the secondary inverse transform is applied can be determined based on the size of the current block.

[0134] Furthermore, as described above, when the secondary (inverse) transform is omitted, the residual block (residual sample) can be obtained by receiving the (inverse-quantized) transform coefficients and performing a first-order (separated) inverse transform. As mentioned above, the encoding and decoding devices can generate a reconstructed block based on the residual block and the prediction block, and generate a reconstructed image based on the residual block.

[0135] Furthermore, in this document, the concept of Reduced Secondary Transform (RST) (where the size of the transformation matrix (kernel) is reduced) can be applied to the NSST concept to reduce the computational and storage requirements of the non-separable secondary transformation. Additionally, since RST is primarily performed in the low-frequency region of the transform block, which includes non-zero coefficients, it can be called LFNST (Low-Frequency Non-Separable Transform). The transform index can be named the LFNST index.

[0136] In this specification, LFNST can represent the transformations performed on residual samples of a target block based on a transformation matrix of reduced size. When performing simplified transformations, the computational cost of the transformation can be reduced due to the reduction in the size of the transformation matrix. In other words, LFNST can be used to solve computational complexity problems that arise when transforming large blocks or non-separable transformations.

[0137] Furthermore, when the secondary inverse transform is based on LFNST, the inverse transform unit 235 of the encoding device 200 and the inverse transform unit 322 of the decoding device 300 may include an inverse RST unit for deriving modified transform coefficients based on the inverse RST of the transform coefficients and an inverse primary transform unit for deriving residual samples of the target block based on the inverse primary transform for the modified transform coefficients. The inverse primary transform refers to the inverse transform applied to the primary transform of the residual. In this document, deriving transform coefficients based on a transform can mean deriving the transform coefficients by applying the corresponding transform.

[0138] Figure 6 This is a diagram used to illustrate the application of RST to RST or LFNST according to the implementation method described in this document.

[0139] In this disclosure, "target block" may refer to the current block, residual block, or transform block to be encoded.

[0140] In the example RST, an N-dimensional vector can be mapped to an R-dimensional vector in another space, such that a reduced transformation matrix can be determined, where R is less than N. N can refer to the square of the length of the side of the block to which the transformation is applied, or the total number of transformation coefficients corresponding to the block to which the transformation is applied, and the reduction factor can refer to the R / N value. The reduction factor can be called a reduction factor, shrinkage factor, simplification factor, or other various terms. Furthermore, R can be called a reduction coefficient, but depending on the situation, the reduction factor can refer to R. Additionally, depending on the situation, the reduction factor can refer to the N / R value.

[0141] The size of the reduced transformation matrix, as shown in the example, can be less than N×N (the size of the regular transformation matrix) and can be R×N, as defined in Equation 1 below.

[0142] [Formula 1]

[0143] Figure 6 The matrix T in the reduced transformation block shown in (a) can refer to the matrix T in Equation 1. R×N .like Figure 6 As shown in (a), when the reduced transformation matrix T R×N By multiplying by the residual samples for the target block, the transformation coefficients for the target block can be derived.

[0144] In the example, if the size of the block to which the transformation is applied is 8×8 and R=16 (i.e., R / N=16 / 64=1 / 4), then according to Figure 6 The RST of (a) can be represented as the matrix operation shown in Equation 2 below. In this case, the storage and multiplication computations can be reduced to approximately 1 / 4 by a reduction factor.

[0145] In this disclosure, matrix operations can be understood as operations on column vectors obtained by multiplying a column vector by a matrix placed to the left of the column vector.

[0146] [Equation 2]

[0147] In Equation 2, r1 to r 64 The residual samples of the target block can be represented, and more specifically, they can be the transformation coefficients generated by applying a primary transformation. As a result of the calculation in Equation 2, the transformation coefficients c for the current block can be derived. i And export C i The process can be the same as in Equation 3.

[0148] [Formula 3]

[0149] As a result of Equation 3, the transformation coefficients c1 to c2 for the target block can be derived. R In other words, when R=16, the transformation coefficients c1 to c2 for the target block can be derived. 16 If a regular transform is applied instead of an RST, and a 64×64 (N×N) transform matrix is ​​multiplied by a 64×1 (N×1) residual sample, then although 64 (N) transform coefficients are derived for the target block, only 16 (R) transform coefficients are derived because an RST is applied. Since the total number of transform coefficients for the target block is reduced from N to R, the amount of data sent from the encoding device 200 to the decoding device 300 is reduced, thus improving the transmission efficiency between the encoding device 200 and the decoding device 300.

[0150] When considering the size of the transformation matrix, the size of a regular transformation matrix is ​​64×64 (N×N), but the size of a reduced transformation matrix is ​​reduced to 16×64 (R×N). Therefore, compared to performing a regular transformation, the storage usage ratio of performing LFNST can be reduced by R / N. Furthermore, compared to the number of multiplications (N×N) when using a regular transformation matrix, using a reduced transformation matrix can reduce the number of multiplications (R×N) by R / N.

[0151] In the example, the transformer 232 of the encoding device 200 can derive transform coefficients for the target block by performing a primary transform and an RST-based secondary transform on the residual samples for the target block. These transform coefficients can be passed to the inverse transformer of the decoding device 300, and the inverse transformer 322 of the decoding device 300 can derive modified transform coefficients based on the inverse reduced secondary transform (RST) for the transform coefficients, and can derive residual samples for the target block based on the inverse primary transform for the modified transform coefficients.

[0152] Based on the example inverse RST matrix T N×R The size is smaller than the size of the conventional inverse transformation matrix, which is N×N × R, and is similar to the reduced transformation matrix T shown in Equation 1. R×N It has a transpose relationship.

[0153] Figure 6 The matrix T in the reduced inverse transform block shown in (b) t It can refer to the inverse RST matrix T R×N T (The superscript T indicates transpose). For example... Figure 6 As shown in (b), when the inverse RST matrix T R×N T Multiplying by the transform coefficients for the target block yields either the modified transform coefficients for the target block or a residual sample of the target block. The inverse RST matrix T R×N T It can be represented as (T) R×N ) T N×R .

[0154] More specifically, when the inverse RST is used as a secondary inverse transformation, when the inverse RST matrix T R×N T When multiplied by the transform coefficients of the target block, the modified transform coefficients of the target block can be derived.

[0155] Furthermore, as illustrated in the example, the inverse RST can be applied as the inverse primary transformation. In this case, this can be achieved by multiplying the transformation coefficients of the target block by the inverse RST matrix T. R×N T To export the residual samples of the target block.

[0156] In the example, if the size of the block to which the inverse transform is applied is 8×8 and R=16 (i.e., R / N=16 / 64=1 / 4), then according to Figure 6 The RST of (b) can be represented as a matrix operation, as shown in Equation 4 below.

[0157] [Formula 4]

[0158] In Equation 4, c1 to c 16 This can represent the transform coefficients of the target block, that is, the transform coefficients derived through residual coding. As a result of the operation in Equation 4, the transform coefficients representing the modifications to the target block or the r values ​​of the residual samples of the target block can be derived. i And r i The derivation process can be the same as Equation 5.

[0159] [Formula 5]

[0160] As a result of the operation in Equation 5, the modified transformation coefficients of the target block or the residual samples of the target block can be derived from r1 to r. N Since N is 64 in Equation 4, 64 modified transformation coefficients can be derived through Equation 5.

[0161] Considering the size of the inverse transformation matrix, the size of the regular inverse transformation matrix is ​​64×64 (N×N), but compared to performing the regular inverse transformation, the size of the simplified inverse transformation matrix is ​​reduced to 64×16 (N×R). When performing the inverse RST, the storage utilization can be reduced to an R / N ratio. Furthermore, compared to the number of multiplication operations N×N when using the regular inverse transformation matrix, using the simplified inverse transformation matrix can reduce the number of multiplication operations to (N×R) at an R / N ratio. If there is no confusion between the simplified inverse transformation matrix or the inverse transformation matrix being a transformation or an inverse transformation, then the simplified inverse transformation matrix or the inverse transformation matrix can also be named a simplified transformation matrix or a transformation matrix.

[0162] According to embodiments of this disclosure, for the transformation during the encoding process, only 48 data points can be selected, and a maximum 16×48 transform kernel matrix can be applied to them, instead of applying a 16×64 transform kernel matrix to the 64 data points forming an 8×8 region. Here, "maximum" means that m has a maximum value of 16 in the m×48 transform kernel matrix used to generate m coefficients.

[0163] In other words, when performing RST by applying an m×48 transformation kernel matrix (m≤16) to an 8×8 region, 48 data points are input and m coefficients are generated. When m is 16, 48 data points are input and 16 coefficients are generated. That is, assuming the 48 data points form a 48×1 vector, the 16×48 matrix and the 48×1 vector are multiplied sequentially to generate a 16×1 vector. In this embodiment, the column vectors in Equation 2 are r1 to r... 48 The transformation matrix is ​​16×48 in size, and 16 modified transformation coefficients (c1 to c) are derived through matrix operations. 16 ).

[0164] Here, the 48 data points forming the 8×8 region can be appropriately arranged to form a 48×1 vector. For example, a 48×1 vector can be constructed based on the 48 data points constituting the region excluding the lower right 4×4 region of the 8×8 region. Here, when matrix operations are performed by applying a maximum 16×48 transformation kernel matrix, 16 modified transformation coefficients are generated, and these 16 modified transformation coefficients can be arranged in the upper left 4×4 region according to the scan order, and the upper right 4×4 region and the lower left 4×4 region can be filled with zeros.

[0165] In the inverse transform of the decoding process, the transpose of the aforementioned transform kernel matrix can be used. That is, when performing inverse RST or inverse LFNST as an inverse transform process performed by the decoding device, the input coefficient data for applying inverse RST consists of a one-dimensional vector arranged in a predetermined order (diagonal scan order). The modified coefficient vector obtained by multiplying the one-dimensional vector from the left by the corresponding inverse RST matrix can be arranged in a two-dimensional block in a predetermined order.

[0166] If an RST or LFNST is performed on an 8×8 region and an inverse RST or inverse LFNST is performed for this purpose, then the size of the transformation matrix in Equation 4 is 48×16, and the ten vectors are c1 to c2. 16 Through matrix operations, 48 ​​modified transformation coefficients (r1 to r) were derived. 48 ).

[0167] In summary, during the transformation process performed by the encoding device, when RST or LFNST is applied to an 8×8 region, matrix operations are performed between the 48 transform coefficients in the upper left, upper right, and lower left regions (excluding the lower right region of the 8×8 region) of the 8×8 region and the 16×48 transform kernel matrix. For the matrix operations, the 48 transform coefficients are input as a one-dimensional array. When this matrix operation is performed, 16 modified transform coefficients are derived, and these modified transform coefficients can be arranged in the upper left region of the 8×8 region.

[0168] Conversely, when the inverse RST or LFNST is applied to an 8×8 region during the inverse transform performed by the encoding or decoding device, the 16 transform coefficients corresponding to the upper left of the 8×8 region can be input as a one-dimensional array according to the scan order, and matrix operations are performed using a 48×16 transform kernel matrix. That is, the matrix operation in this case can be represented as a (48×16 matrix). (16×1 transformation coefficient vector) = (48×1 modified transformation coefficient vector). Here, since an n×1 vector can be interpreted in the same way as an n×1 matrix, it can also be expressed as an n×1 column vector. Furthermore, This represents a matrix multiplication operation. When this matrix operation is performed, 48 modified transformation coefficients are derived, which can be arranged in the upper left, upper right, and lower left regions of an 8×8 area (excluding the lower right region).

[0169] Furthermore, in existing LFNSTs, due to complexity constraints, a trade-off between coding performance and complexity must be considered. More specifically, the technique must be designed in a way that minimizes computational cost and implementation complexity by keeping the region of input primary transform coefficients used for LFNST, the number of output transform coefficients, the number of LFNST sets, the number of LFNST kernels in each set, and the number of operations required to process a sample for LFNST below a certain level.

[0170] However, in the next standard, the allowed complexity will increase due to advancements in implementation technology. Therefore, it is advisable to consider structurally extending existing LFNST techniques to improve coding performance. Based on various techniques for structurally extending LFNST, the coding performance that LFNST can offer can be maximized within the implementation complexity allowed by the next standard.

[0171] In this regard, various implementations that structurally extend the above-described LFNST are proposed below.

[0172] According to the implementation method, the region of interest (ROI) representing the input data region based on the forward LFNST can be configured as follows.

[0173] Specifically, the forward LFNST receives the transform coefficients of the primary transform as input. In this case, instead of receiving all transform coefficients as input, transform coefficients belonging to a predetermined specific region within the transform block can be received as input.

[0174] Figure 7 (a) shows an example of the input region for the forward LFNST in the VVC standard, and the corresponding input region is set up up to the third position in the scan sequence of the 4×4 sub-block. In the following text, this input region (that is, the region into which the input transform coefficients are input for the forward LFNST) is referred to as the region of interest or ROI.

[0175] Figure 7 (b) shows that Figure 7 The scan order of the 4×4 sub-blocks shown in (a) adds the fourth to sixth sub-blocks to the ROI. That is, since the ROI consists of six 4×4 sub-blocks, Figure 7 The ROI in (b) consists of 96 samples. (This is in contrast to the ROI used for VVC.) Figure 7 Compared to the ROI of (a), in Figure 7In case (b), more primary transform coefficients can be considered in the LFNST. The extended LFNST (LFNST) based on such an extended ROI can provide higher coding performance than the existing VVC LFNST for larger blocks (i.e., greater than or equal to 16×16).

[0176] When deriving an inseparable transformation matrix based on N samples belonging to the ROI, an N×N transformation matrix is ​​derived when the corresponding transformation matrix is ​​a square matrix. Here, when the reference... Figure 7 When R is applied to the reduced transform, the value of R can be equal to or less than N. The value of R can be interpreted as the number of output transform coefficients derived by applying LFNST from the perspective of the forward LFNST. Therefore, with Figure 7 The dimension of the corresponding LFNST transform kernel in (b) is not only 96×96, but also 16×96, 32×96, 48×96, 64×96, 80×96, etc. (In this example, it can be represented as a 16m×96 matrix (m≥1)). In this case, only the number of output transform coefficients that are multiples of 16 are allowed, and the value of R can be any positive integer equal to or less than N (i.e., 96). When generating an R×96 matrix from a 96×96 matrix, it can be generated by sampling R rows from the 96×96 matrix based on the forward LFNST. If the rows constituting the 96×96 matrix are arranged in order of importance from top to bottom, the R×96 matrix can be constructed by sequentially sampling R rows from top to bottom.

[0177] Figure 7 The ROIs in (a) and (b) consist of 48 and 96 input samples (input transform coefficients or input data), respectively. The order in which the input samples in the ROI are read can be preset, but the basic order can be set arbitrarily. More specifically, when the positive LFNST matrix applied to any ROI is an R×N matrix (i.e., the ROI consists of N input samples), even if the order in which the input samples are read is changed, if the N column vectors are rearranged according to the changed order, the output value (consisting of R transform coefficients) will not change compared to before the change, regardless of the order of the input samples.

[0178] Furthermore, for a specific positive LFNST transformation matrix (that is, assuming the positions of the column vectors are fixed), it can be done according to... Figure 8 or Figure 9 The sequence shown reads the input samples that constitute the ROI and configures them as input vectors. Figure 8 and Figure 9 The order in which input data is read from the ROI according to the intra-frame prediction mode is shown.

[0179] Figure 8 (a) and Figure 9 (a) corresponds to Figure 7 (a), and Figure 8 (b) and Figure 9 (b) corresponds to Figure 7 (b). Similar to VVC LFNST, if mode 34-centered symmetry is applied for intra-prediction modes, then for Figure 5 Patterns -14 to -1 and patterns 2 to 33, such as Figure 8 As shown in (a) and (b), the line priority order can be applied for patterns 35 to 80 as follows: Figure 9 The application column priority order is shown in (a) and (b). For modes 0, 1, and 34, which indicate planar mode and DC mode, the same priority order can be applied. Figure 8 (a) and Figure 8 (b) is in the same order, and Figure 8 (a) and Figure 8 (b) or Figure 9 (a) and Figure 9 (b) can be applied to each pattern.

[0180] As another example of a ROI, the upper left quadrant of the transform block can be set as the ROI. That is, in an M×N transform block, the upper left m×n (m≤M, n≤N) region can be set as the ROI (from the forward LFNST viewpoint), with the number of input samples (transform coefficients after the first transform) being m×n. In a more specific implementation, both m and n can be 8, and the dimension of the forward LFNST matrix can be R×64 (R is equal to or less than 64; examples of R values ​​are 16, 32, 48, 64, etc.). The method for selecting R rows from an mn×mn matrix (e.g., a 64×64 matrix) can be the same as the method described above for generating an R×96 matrix from a 96×96 matrix.

[0181] Furthermore, ROI may not be as... Figure 7 As shown, it consists only of 4×4 sub-blocks. In the VVC standard, a 4×4 sub-block can correspond to a transform group (coefficient group, CG) of transform blocks to which LFNST can be applied; these CGs are not necessarily 4×4 sub-blocks. According to the example, a CG can be any predetermined p×q sub-block other than a 4×4 sub-block. Even if sub-blocks (CGs) of a size other than 4×4 are applied, regardless of the sub-block configuration, the order in which the ROI input samples for forward LFNST are read can still follow the order shown. Figure 8 (a) and Figure 8 (b) or Figure 9 (a) and Figure 9 The specific order determined in (b) can be arranged according to the scan order of the corresponding CG and transform coefficients, which are output by the forward LFNST.

[0182] Furthermore, according to the VVC standard, different LFNST kernels can be applied based on the transform block size. That is, for 4×4 transform blocks or 4×N / N×4 (N≥8) transform blocks (transform blocks with both horizontal and vertical lengths greater than or equal to 4, and transform blocks with either a horizontal or vertical length of 4), an LFNST kernel with a 16×16 matrix form suitable for the top-left 4×4 region can be applied (which can be named LFNST_4×4). Additionally, for transform blocks with horizontal and vertical lengths equal to or greater than 8, an LFNST kernel with a 16×48 matrix form can be applied based on the forward LFNST to the following ROI (which can be named LFNST_8×8), which consists of a top-left 4×4 sub-block, a 4×4 sub-block to the right of the top-left 4×4 sub-block, and a 4×4 sub-block adjacent to the bottom of the top-left 4×4 sub-block.

[0183] For example, LFNST_4×4 and LFNST_8×8 each consist of four sets, each set comprising two transform kernels. Which kernel set to apply can be determined by the intra-frame prediction mode. For the determined set, the signaling of the LFNST index can specify which of the two kernels to apply and whether LFNST should be applied. If the LFNST index value is 0, LFNST is not applied; if it is 1, the first kernel can be applied; and if it is 2, the second kernel can be applied.

[0184] As mentioned above, the LFNST structure in the VVC standard has been simplified and described, but there are some exceptions. For example, for 4×4 transform blocks and 8×8 transform blocks, the 8×16 and 8×48 matrices sampled from the corresponding matrices are applied as forward LFNSTs, respectively, instead of the 16×16 and 16×48 matrices. When applying the MIP prediction mode, the intra-prediction mode can be regarded as a planar mode, and the LFNST set can be determined.

[0185] Since LFNST_4×4 and LFNST_8×8 are each composed of 4 LFNST sets, for the convenience of the following description, a list of LFNST sets named LFNST_4×4 or LFNST_8×8 can be used to represent a set of LFNST sets.

[0186] In addition, in this document, LFNST_8×8 can denote a list of LFNST sets applied to transform blocks with a horizontal or vertical length of 8, where both the horizontal and vertical lengths are greater than or equal to 8. Furthermore, a list of LFNST sets applied to transform blocks with both the horizontal and vertical lengths greater than or equal to 16 can be named LFNST_16×16.

[0187] Additional embodiments of the matrix dimensions and ROIs that LFNST_4×4, LFNST_8×8, and LFNST_16×16 can have are as follows. In the following embodiments, the transformation matrix is ​​based on when a forward transformation is applied.

[0188] 1. LFNST_4×4 can have a 16×16 matrix, and the ROI can be a 4×4 region in the upper left corner.

[0189] 2. LFNST_8×8 can have an R×48 matrix or an S×64 matrix, and 16, 32, and 48 can be used as R values, and 16, 32, 48, and 64 can be used as S values. The ROI for an R×48 matrix can be... Figure 7 (a), and the ROI for the S×64 matrix can be the 8×8 region in the top left corner.

[0190] 3. LFNST_16×16 can be an R×96 matrix, an S×64 matrix, or a T×48 matrix. 16, 32, 48, 64, 80, and 96 can be used as R values, 16, 32, 48, and 64 can be used as S values, and T values ​​can be 16, 32, and 48. The ROI for an R×96 matrix can be... Figure 7 In (b), the ROI for an S×64 matrix can be the 8×8 region in the top left corner, and the ROI for a T×48 matrix can be... Figure 7 (a) in the middle.

[0191] As for the architectures used for LFNST_4×4, LFNST_8×8, and LFNST_16×16, any combination of matrix dimensions and ROIs suggested in No. 1, No. 2, and No. 3 above is possible. For example, in the case of LFNST_4×4, the ROI of the top-left 4×4 region is applied to a 16×16 matrix; in the case of LFNST_8×8, the ROI of the top-left 8×8 region is applied to a 32×64 matrix; and in the case of LFNST_16×16, ... Figure 7 The ROI shown in (b) can be applied to a 32×96 matrix.

[0192] Additionally, if any pair of LFNST_4×4, LFNST_8×8, and LFNST_16×16 has the same matrix dimension, they can share the LFNST set and LFNST kernel for that pair. For example, if both LFNST_8×8 and LFNST_16×16 have a matrix dimension of 32×64, the same list of LFNST sets can be assigned to both LFNST_8×8 and LFNST_16×16, and the same ROI can be set (e.g., the ROI can be set to the top-left 8×8 region).

[0193] As another example, when the matrix dimension of LFNST_8×8 is 32×48 and the matrix dimension of LFNST_16×16 is 32×48, the same list of LFNST sets can be assigned to both LFNST_8×8 and LFNST_16×16, and the same ROI can be set (e.g., as shown below). Figure 7 (as shown in (a)) ROI settings.

[0194] On the other hand, when applying the inverse LFNST, when the input vector is constructed using R transform coefficients and the left side of the input vector is multiplied by an N×R matrix, N output samples (output transform coefficients) are generated. Here, the N×R matrix becomes the transpose of the R×N matrix in the forward LFNST, and the N output samples can be arranged in... Figures 7 to 12 Within the ROI. When arranged within the ROI, based on the intra-frame prediction mode value, it can follow... Figure 8 or Figure 9 The order shown. For example, when utilizing the symmetry between intra-prediction modes, Figure 8 The row priority order can be applied to intra-prediction modes -14 to -1 and 2 to 33. Figure 9 The column priority can be applied to modes 35 through 80. Regarding modes 0, 1, and 34, which indicate planar and DC modes, Figure 8 The order of (a) and (b) can be applied to all items, or Figure 8 The order of (a) and (b) or Figure 9 The order of (a) and (b) can be applied to each pattern.

[0195] According to the implementation method, the output data area can be configured based on the forward LFNST as follows.

[0196] Specifically, in the VVC standard, the scan order for transform coefficients is configured hierarchically. There is a scan order for the CGs, and an internal scan order for each CG. Figure 10 This is a diagram showing the order in which these transformation coefficients are scanned. (See diagram for example.) Figure 10 As shown, the scan sequence progresses diagonally from the bottom left to the top right. Figure 10 In this diagram, a small square can represent a transformation coefficient, and the numbers inside the small square indicate the scanning order.

[0197] If a scan line is formed by scanning from the bottom left to the top right in a CG unit, then the first scan line consists of 1 CG, and the second and third scan lines consist of two and three CGs respectively. Following the same method, the Nth scan line also consists of multiple CGs.

[0198] Figure 7 (a) and Figure 7The ROIs shown in (b) are all composed of these CG unit scan lines. Figure 7 (a) and Figure 7 (b) shows the ROI consisting of the first two scan lines and the first three scan lines, respectively. Naturally, the ROI can be composed of more scan lines.

[0199] As mentioned above, when the number of output transform coefficients in the forward LFNST criterion is R and the number of input samples is N, R can be set to be less than or equal to N. Specifically, as... Figure 7 As shown, R can be set to a multiple of 16 (i.e., R = 16k, k ≥ 1). Considering that the CG of the transform block to which LFNST is applied is a 4×4 sub-block, this is set such that the number of output transform coefficients of the forward LFNST is a multiple of the number of samples present in one CG. Therefore, if R is set to a multiple of 16, the transform coefficients obtained by applying the forward LFNST can be designed to be arranged only in a specific 4×4 sub-block. This design further simplifies the design of the residual coding section and the LFNST index signaling.

[0200] For example, if transform coefficients are resolved in regions other than where LFNST transform coefficients might exist, the signaling for the LFNST index can be omitted, and it can be inferred that LFNST is not applied. Here, if the regions where LFNST transform coefficients might exist are configured in 4×4 sub-blocks and residual coding is performed in corresponding 4×4 sub-blocks, it is easier to check whether transform coefficients exist in regions other than where LFNST transform coefficients might exist.

[0201] According to another implementation, the CG can have a shape other than a 4×4 sub-block, and in this case (e.g., an m×n block, m≠n), the R value can be set to a multiple of m×n. Additionally, the CG with positive LFNST output transform coefficients may consist of the first k CGs arranged according to the scan order of the CGs.

[0202] Basically, the output coefficients of a forward LFNST can be arranged according to the scan order of the transform coefficients. Since the row vectors of the forward LFNST kernel are typically arranged from top to bottom in order of importance, assuming that the transform coefficients constituting the output vector are arranged in top-to-bottom order (here, assuming the output vector is a column vector), the coefficients can be arranged sequentially, starting with the more significant coefficients. It is generally assumed that the scan order of the transform coefficients starts with the most important coefficients, because the distance from the DC position increases as the scan proceeds from the DC position marked by the top-left position, so less important transform coefficients are arranged and mainly have values ​​of 0 or close to 0. Therefore, in terms of coding performance, it may be advantageous to arrange the output transform coefficients of the forward LFNST sequentially according to the scan order starting from the DC position. Furthermore, in many cases, the residual coding section is designed to improve coding efficiency as transform coefficients with values ​​of 0 or close to 0 frequently appear with increasing distance from the DC position.

[0203] Furthermore, the output transform coefficients of the forward LFNST do not necessarily have to be arranged according to a fixed scan order. That is, according to another embodiment, the output transform coefficients of the LFNST can be ordered according to an order other than the scan order.

[0204] If it is statistically determined that a scan order other than the scan order in the VVC standard is suitable for the corresponding LFNST output coefficients, then an LFNST-specific scan order can be applied instead of a previously determined scan order, provided it is known in advance whether LFNST will be applied before residual coding is performed. Furthermore, when the optimal scan order varies depending on the coding context, such as intra-prediction modes, different scan orders can be applied to the forward LFNST output transform coefficients for each intra-prediction mode (or group of intra-prediction modes), as illustrated by the example.

[0205] According to the implementation method, the LFNST set list, LFNST set, and LFNST kernel can be applied based on the size of the transform block.

[0206] As illustrated in the example, unlike the configuration of the LFNST set list, LFNST set, and LFNST core for each set in the existing VVC standard, the LFNST set list, LFNST set, and LFNST core can be applied in a more granular manner for each transform block size. Here, the configuration of the LFNST core for each set can indicate which LFNST set consists of how many candidate cores.

[0207] For example, a different list of LFNST sets can be applied for each possible transform block shape (i.e., each possible M×N block), and the corresponding list of sets can be expressed as, for example, LFNST_M×N. Alternatively, the corresponding list of LFNST sets can be applied to each group by grouping the transform block shapes. In the case of the VVC standard, it can be seen that two types of LFNST set lists are applied by dividing the transform block shapes into two groups, namely LFNST_4×4 and LFNST_8×8. Examples of other groupings are as follows.

[0208] 1. For transform blocks whose horizontal and vertical lengths are both equal to or greater than 16, a separate group can be set up, and an LFNST set list can be assigned to this group. Here, the LFNST set list can be named LFNST_16×16. When combined with the grouping of the VVC standard, it can be divided into three groups: (Group 1) 4×4, 4×N / N×4 (N≥8) transform blocks; (Group 2) 8×8, 8×N / N×8 (N≥16) transform blocks; and (Group 3) transform blocks whose width and height are both greater than or equal to 16. Each group and / or LFNST set list applied to this group can be named LFNST_4×4, LFNST_8×8, or LFNST_16×16.

[0209] 2. Within the grouping of No. 1 above, group 1 can be further divided into 4×4 transform blocks and 4×N / N×4 (N≥8) transform blocks, which can be classified as group 1A and group 1B. Group 2 can also be divided into 8×8 transform blocks and 8×N / N×8 (N≥16) transform blocks, which can be classified as group 2A and group 2B. Furthermore, group 3 can be divided into group 3A and group 3B based on specific criteria. For example, 16×16 and 16×N / N×16 (N≥16) transform blocks can be set as group 3A, and the remaining cases can be classified as group 3B.

[0210] Additionally, Groups 1, 2, and 3 may or may not be divided into detailed groups as described above. For example, if only Groups 1 and 3 are divided into detailed groups, then all groups can be configured as Group 1A, Group 1B, Group 2, Group 3A, and Group 3B. Naturally, if Groups 1, 2, and 3 are all divided, then the groups can be classified as Group 1A, Group 1B, Group 2A, Group 2B, Group 3A, and Group 3B.

[0211] In addition to the two embodiments described above, grouping can be applied based on the size of the transform block according to various criteria, and a corresponding LFNST set list can be assigned to each group. This LFNST set list can be configured differently for each group.

[0212] For example, the number of cores for each LFNST set that makes up the list of LFNST sets can be set differently (e.g., for group 1, the number of LFNST cores for each set is 3, and for group 2, the number of LFNST cores for each set is given, that is, for group 1, the number of LFNST cores for each set is set for every three sets, and for group 2, the number of LFNST cores for each set is set for every two sets). More specifically, the number of cores for each set can be set differently for each LFNST set that makes up a list of LFNST sets.

[0213] Alternatively, the number of LFNST sets included in each LFNST set list can be set differently; for example, group 1 can consist of 18 LFNST sets, and group 2 can consist of 10 LFNST sets. Naturally, the dimensions of the kernel matrix can be set differently depending on the LFNST set list. Taking the VVC standard as an example, LFNST_4×4 consists of a 16×16 matrix, and LFNST_8×8 consists of a 16×48 matrix.

[0214] Furthermore, the dimensions of the kernel matrix can be set differently for each LFNST set that makes up the list of LFNST sets. Specific examples of detailed configurations for the list of LFNST sets are shown below.

[0215] 1. Group 1 (LFNST_4×4) consists of 18 LFNST sets, each with 3 kernels, and the corresponding kernel matrix can have a dimension of 16×16. Group 2 (LFNST_8×8) consists of 18 LFNST sets, each with 3 kernels, and the corresponding kernel matrix can have a dimension of 16×48. Group 3 (LFNST_16×16) consists of 18 LFNST sets, each with 3 kernels, and the corresponding kernel matrix can have a dimension of 32×96.

[0216] 2. In configuration 1 above, all LFNST sets can be configured with 2 cores instead of 3 cores.

[0217] 3. All LFNST set lists in the above configuration can be configured with a number of sets other than 18. For example, the LFNST set list can consist of 16, 15, 10, 6, or 4 transform sets.

[0218] 4. In the first configuration, the dimensions of the kernel matrix constituting LFNST_8×8 can be set from 32×48 to 48×48.

[0219] 5. In the first configuration, the dimension of the kernel matrix constituting LFNST_16×16 can be set to one of 16×96, 48×96, 64×96, 80×96, and 96×96. Here, 96 can represent the number of input samples (input transform coefficients) constituting the ROI according to the forward LFNST, and the ROI can be as follows: Figure 7 The configuration shown in (b) is as follows. If the ROI corresponding to LFNST_16×16 is as shown in (b) Figure 7 (a) instead of Figure 7 If (b) is configured, the dimensions of the kernel matrix constituting LFNST_16×16 can be set to one of 16×48, 32×48, and 48×48.

[0220] 6. Based on No.1 above, No.2, No.3, No.4, and No.5 can be freely combined. For example, by applying No.3, the number of LFNST sets is set to 15, and by applying No.4, the dimension of the kernel matrix constituting LFNST_8×8 can be set to 32×48.

[0221] According to the implementation method, it can be determined whether to apply LFNST based on color components, and when it is determined that LFNST should be applied, an LFNST set list, an LFNST set, and an LFNST core can be applied based on color components.

[0222] In the VVC standard, when the tree type of the coding unit is a single tree, LFNST is applied only to the luma component. In the case of a split tree (that is, a two-tree tree), LFNST is applied to the luma component in the case of a split tree used for the luma component (two-tree luma), and to the chroma component in the case of a split tree used for the chroma component (two-tree chroma).

[0223] Unlike the VVC standard, according to this embodiment, LFNST can be applied only to the luma component. If LFNST is applied only to the luma component, then in a single tree, LFNST is applied only to the luma component and not to the chroma component, as in the VVC standard, where the LFNST index only identifies the LFNST core applied to the luma component. If LFNST is applied only to the luma component, then because LFNST is not applied when it is a segmented tree used for the chroma component, the LFNST index is not signaled (if the LFNST index is not signaled, it can be assumed that LFNST is not applied by default).

[0224] Alternatively, according to another embodiment, in the case of a single tree, LFNST can be applied to both the luma and chroma components, which differs from the VVC standard. In this case, it can be implemented in two ways. That is, 1) the image information is configured such that an LFNST index can be signaled to select a corresponding LFNST core for both the luma and chroma components; 2) the image information can be configured such that individual LFNST indices are signaled for both the luma and chroma components, allowing the selection of the most suitable LFNST core for each component.

[0225] When the image information is configured such that the LFNST index is signaled separately for the luma and chroma components when using a single tree, the LFNST set list, LFNST set, and LFNST kernel can be configured differently for the luma and chroma components.

[0226] In the case of a single tree, the LFNST set list, LFNST set, and LFNST core for the luma component and chroma component are set differently. When the LFNST core for the luma component and chroma component is selected by sending an LFNST index with a signal, the LFNST core for the luma component and chroma component specified by a signaled LFNST index can be different because they are selected from different LFNST set lists and LFNST sets.

[0227] Furthermore, in the VVC standard, the same LFNST set list is applied to both the luminance and chrominance components.

[0228] According to embodiments, unlike the VVC standard, different LFNST set lists, different LFNST sets, and different LFNST kernels can be applied to the luma and chroma components. Alternatively, according to embodiments, another LFNST set list, another LFNST set, or another LFNST kernel can be applied to all color components. For example, different LFNST set lists can be applied to Y, Cb, and Cr. Furthermore, when applying other color formats (e.g., YUV 4:2:2, YUV 4:4:4, RGB 4:4:4), different LFNST set lists, different LFNST sets, and different LFNST kernels can be applied to the corresponding color format. As a more specific example, in the case of RGB 4:4:4 format, different LFNST set lists, different LFNST sets, and different LFNST kernels can be applied to the R component, G component, and B component, respectively. Of course, the same LFNST set list, LFNST set, and LFNST kernel can be applied to all three components.

[0229] According to the implementation, different LFNST set lists can be applied for each transform block size. As a more specific example, instead of applying only LFNST_4×4 and LFNST_8×8 as in the VVC standard, different LFNST set lists can be applied to each of the following cases: 4×4, 4×N / N×4 (N≥8), 8×8, 8×N / N×8 (N≥16), 16×16, 16×N / N×16 (N≥32), 32×32 or more (both horizontal and vertical lengths are 32 or greater). Here, the LFNST set list for each block size set can be represented as LFNST_4×4, LFNST_4×N_N×4, LFNST_8×8, LFNST_8×N_N×8, LFNST_16×16, LFNST_16×N_N×16, LFNST_32×32.

[0230] For example, when applying the i-th LFNST set list among the M LFNST set lists for the block size set according to the index value specified by the encoding mode, condition, or LFNST set list index information, the i-th LFNST set list can be represented as a tuple (LFNST_4×4_i, LFNST_4×N_N×4_i, LFNST_8×8_i, LFNST_8×N_N×8_i, LFNST_16×16_i, LFNST_16×N_N×16_i, LFNST_32×32_i).

[0231] According to the VVC standard, LFNST_4×4 and LFNST_8×8, which each identify a list of LFNST sets, can each consist of four LFNST sets, and these four LFNST sets can be distinguished by index values ​​0, 1, 2, and 3. That is, the LFNST sets can be classified as the zeroth LFNST set, the first LFNST set, the second LFNST set, and the third LFNST set, and the LFNST set index of each LFNST set can have a value of 0, 1, 2, or 3.

[0232] Additionally, the VVC standard supports Wide Angle Intra Prediction (WAIP) mode, which corresponds to modes -14 to -1 and modes 67 to 80 within the intra prediction mode. In the VVC standard, the existing first LFNST set can be mapped and used for WAIP mode without needing to allocate a separate LFNST set for WAIP mode. That is, as shown in Table 3 below, the first LFNST set is mapped to intra prediction mode values ​​from -14 to 80.

[0233] [Table 3]

[0234] In Table 3, Intra pred. mode can represent the intra prediction mode of the current block, and the value of Intra pred. mode is either -14 to -1 or 67 to 80, which can indicate that the intra prediction mode of the current block is WAIP mode.

[0235] In this regard, according to the implementation proposed in this disclosure, unlike the VVC standard, a larger number of LFNST sets can be introduced, and a separate LFNST set can be allocated for the WAIP mode. If a separate LFNST set is not allocated to the WAIP mode, a first LFNST set can be allocated as shown in Table 3 above.

[0236] According to one implementation, when a mode other than WAIP mode is named Normal Angle Intra Prediction (NAIP) mode (i.e., modes from 0 to 66), if the number of LFNST sets allocated to NAIP mode is N, and the number of LFNST sets allocated solely to WAIP mode is M, then the composition of the LFNST sets can be represented by (N, M). An example of (N, M) is as follows.

[0237] 1. (35, 0)

[0238] 2. (35, 1)

[0239] 3. (35, 3)

[0240] 4. (15, 0)

[0241] 5. (15, 1)

[0242] 6. (15, 3)

[0243] Here, it can be assumed that the LFNST set index value is assigned from 0 to N-1 for the N LFNST sets in the NAIP mode, and it can be assumed that the LFNST set index value is assigned from N to N+M-1 for the M LFNST sets in the WAIP mode. In this case, examples of the mapping tables between the intra-prediction modes and LFNST sets in the first to sixth embodiments described above can be shown in Tables 4 to 9 respectively.

[0244] [Table 4]

[0245] [Table 5]

[0246] [Table 6]

[0247] [Table 7]

[0248] [Table 8]

[0249] [Table 9]

[0250] In Tables 4 to 9 above, Intra pred. mode can represent the intra prediction mode of the current block, and the value of Intra pred. mode is either -14 to -1 or 67 to 80, which can indicate that the intra prediction mode of the current block is WAIP mode.

[0251] The following describes the specific LFNST kernel, that is, the LFNST matrix, applied to LFNST_16×16. As an example, the following LFNST matrix can be applied when the ROI for LFNST_8×8 is a 4×4 sub-block in the target transform block, located at the fourth position from the top left in scan order.

[0252] Tables 10 through 21 below show examples of kernel coefficient data for LFNST_16×16 that can be applied to M×N blocks (M≥16, N≥16). In the g_lfnst16×16

[36] [3]

[32]

[96] array in Tables 10 through 21,

[36] indicates that the number of LFNST sets is 36, [3] indicates that the number of LNFST kernel candidates per LFNST set is 3, and

[32]

[96] indicates that it is a 32×96 matrix based on forward LFNST (the corresponding array definitions in Tables 10 through 21 are described according to C / C++ syntax). For example, 32 can represent the horizontal (x-axis) length of the matrix, and 96 can represent the vertical (y-axis) length of the matrix. If Tables 10 through 21 are used for LFNST, where the number of LFNST sets is 35, then the array can be represented as g_lfnst4×4

[35] [3]

[16]

[16] .

[0253] The LFNST kernels listed in Tables 10 to 21 can be applied to the following ROIs: Figure 7Region (b) in the diagram. An LFNST kernel consists of 32 transform basis vectors (row vectors), and each vector has a length of 96 (

[32]

[96] ). When performing a forward secondary transformation in the encoding device, the row basis vectors and transform coefficients of the LFNST kernels (32×96 dimensional matrices) in Tables 10 to 21 can be multiplied during matrix operations. For the inverse secondary transformation performed in the decoding device, the row basis vectors of the transpose kernel (96×32 dimensional matrix) of the following LFNST kernel can be multiplied by the transform coefficients.

[0254] Tables 10 to 21 can represent a portion of 36 LFNST sets. As mentioned above, LFNST sets can be selected based on the intra-frame prediction mode and mapped according to Table 4 or Table 5. According to Table 4, 35 LFNST sets are used, and according to Table 5, 36 LFNST sets are used. Tables 10 to 21 can be kernels corresponding to specific set numbers within the 35 or 36 sets.

[0255] As shown in the figure, Tables 10 to 21 can represent the kernels applied when the LFNST set index of Table 4 or Table 5 is 0 (when the intra-prediction mode is planar mode). One LFNST kernel can be represented by four tables. For example, Tables 10 to 13 can be the first LFNST kernel applied when the LFNST set index is 0, Tables 14 to 17 can be the second LFNST kernel applied when the LFNST set index is 0, and Tables 18 to 21 can be the third LFNST kernel applied when the LFNST set index is 0.

[0256] [Table 10]

[0257] [Table 11]

[0258] [Table 12]

[0259] [Table 13]

[0260] [Table 14]

[0261] [Table 15]

[0262] [Table 16]

[0263] [Table 17]

[0264] [Table 18]

[0265] [Table 19]

[0266] [Table 20]

[0267] [Table 21]

[0268] According to another example, when the ROI for LFNST_8×8 is a 4×4 sub-block in the target transform block, from the top left to the third position in scan order ( Figure 7 (b) can be applied using the following LFNST matrix.

[0269] Tables 22 to 33 below show examples of kernel coefficient data for LFNST_16×16 that can be applied to M×N blocks (M≥16, N≥16). In the g_lfnst16×16

[36] [3]

[32]

[96] array in Tables 22 to 33,

[36] indicates that the number of LFNST sets is 36, [3] indicates that the number of LNFST kernel candidates per LFNST set is 3, and

[32]

[96] indicates that it is a 32×96 matrix based on forward LFNST (the corresponding array definitions in Tables 22 to 33 are described according to C / C++ syntax). For example, 32 can represent the horizontal (x-axis) length of the matrix, and 96 can represent the vertical (y-axis) length of the matrix. If Tables 22 to 33 are used for LFNST, where the number of LFNST sets is 35, then the array can be represented as g_lfnst4×4

[35] [3]

[16]

[16] .

[0270] The LFNST kernels in Tables 22 and 33 can be applied to the following ROIs: Figure 7 Region (b) in the table. An LFNST kernel consists of 32 transform basis vectors (row vectors), and each vector has a length of 96 (

[32]

[96] ). When performing a forward secondary transformation in the encoding device, the row basis vectors and transform coefficients of the LFNST kernels (32×96 dimensional matrices) in Tables 22 to 33 can be multiplied during matrix operations. For the inverse secondary transformation performed in the decoding device, the row basis vectors of the transpose kernel (96×32 dimensional matrix) of the LFNST kernel below can be multiplied by the transform coefficients.

[0271] Tables 22 to 33 can represent a portion of 36 LFNST sets. As mentioned above, LFNST sets can be selected based on the intra-frame prediction mode and mapped according to Table 4 or Table 5. According to Table 4, 35 LFNST sets are used, and according to Table 5, 36 LFNST sets are used. Tables 10 to 21 can be kernels corresponding to specific set numbers within the 35 or 36 sets.

[0272] As shown in the figure, Tables 22 to 33 can represent the kernels applied when the LFNST set index of Table 4 or Table 5 is 0 (when the intra-prediction mode is planar mode). One LFNST kernel can be represented by four tables. For example, Tables 22 to 25 can be the first LFNST kernel applied when the LFNST set index is 0, Tables 26 to 29 can be the second LFNST kernel applied when the LFNST set index is 0, and Tables 30 to 33 can be the third LFNST kernel applied when the LFNST set index is 0.

[0273] [Table 22]

[0274] [Table 23]

[0275] [Table 24]

[0276] [Table 25]

[0277] [Table 26]

[0278] [Table 27]

[0279] [Table 28]

[0280] [Table 29]

[0281] [Table 30]

[0282] [Table 31]

[0283] [Table 32]

[0284] [Table 33]

[0285] The following figures are used to illustrate specific examples of this disclosure. Since the names of specific devices or signals / messages / fields described in the figures are provided as examples, the technical features of this disclosure are not limited to the specific names used in the following figures.

[0286] Figure 11 An example of a video / image coding method according to an embodiment of the present disclosure is illustrated schematically.

[0287] Figure 11 The method disclosed in the article can be derived from Figure 2 The encoding device disclosed in the document executes the commands. Specifically, for example, Figure 11 The S1100 can be made by Figure 2 At least one of the predictor 220 and / or residual processor 230 of the encoding device 200 is executed, and Figure 11 The S1110 can be made by Figure 2 The predictor 220 of the encoding device 200 performs the operation. Figure 11 S1120 to S1160 can be executed by the residual processor 230 of the encoding device 200, and Figure 11 S1170 can be executed by the entropy encoder 240 of the encoding device 200. Figure 11 The methods disclosed herein may include the embodiments described above.

[0288] refer to Figure 11 The encoding device 200 determines the intra-prediction mode of the current block and determines the LFNST set index (S1100). For example, the encoding device 200 may determine the intra-prediction mode of the current block and determine the LFNST set index based on the intra-prediction mode.

[0289] For example, the LFNST set index for the current block can be derived based on Tables 3 to 9. In these tables, Intrapred. mode can represent the intra prediction mode of the current block, and a value of Intrapred. mode between -14 and -1 and between 67 and 80 can indicate that the intra prediction mode of the current block is WAIP mode.

[0290] According to one implementation, an LFNST set index derived based on a transform set of 35 and the intra-prediction mode of the current block being WAIP mode can be different from 35 LFNST set indices derived based on the intra-prediction mode of the current block being NAIP mode.

[0291] Encoding device 200 generates information related to intra-prediction modes. For example, encoding device 200 can generate information related to intra-prediction modes. For example, the information related to intra-prediction modes can be associated with any of intra-prediction modes 0 to 66. In addition, when the WAIP mode is applied to the current block, after signaling / parsing the information related to the intra-prediction mode, the information related to the intra-prediction mode can be remapped to the index associated with the WAIP mode.

[0292] The encoding device 200 generates a prediction sample for the current block based on the intra-frame prediction mode (S1110). For example, the encoding device 200 may generate a prediction sample for the current block based on the intra-frame prediction mode.

[0293] The encoding device 200 generates a residual sample for the current block based on the prediction sample (S1120). For example, the encoding device 200 may generate a residual sample for the current block based on the prediction sample.

[0294] Encoding device 200 derives transform coefficients based on residual samples (S1130). For example, encoding device 200 can derive transform coefficients based on residual samples. For example, encoding device 200 can derive transform coefficients for the current block by performing a primary transform on the residual samples. For example, the reduction transform can be applied as a primary transform or a normal separable transform. In addition, the primary transform can be based on multiple core transforms, which are based on multiple transform kernels selected from DCT-2, DCT-7, DCT-8, and DST-1.

[0295] Encoding device 200 derives modified transform coefficients based on the transform coefficients and the LFNST matrix (S1140). For example, encoding device 200 can derive modified transform coefficients based on the transform coefficients and the LFNST matrix. For example, encoding device 200 can derive modified transform coefficients by performing LFNST on the transform coefficients using the LFNST matrix. For example, the LFNST can be a secondary transform and can include an inseparable secondary transform reflecting the RST. In this case, the LFNST matrix applied to the secondary transform can be a non-square matrix where the number of rows is less than the number of columns. According to one example, if the width or height of the current block is equal to or greater than 16, the LFNST matrix can be derived as a 32×96 dimensional matrix.

[0296] According to the present disclosure, different from the existing VVC standard, LFNST may be applied to the top-left 16×16 region of a transform block that is equal to or larger than 16×16. In this document, that a P×Q block is greater than or equal to an M×N block means that each of P and Q is respectively greater than or equal to each of M and N. For example, LFNST may be applied to a region of 96 samples composed of 6 4×4 sub-blocks in the top-left 16×16. That is, modified transform coefficients that are fewer than the transform coefficients may be derived based on some transform coefficients belonging to the top-left 16×16 region of the target block. According to an example, the encoding device 200 processes L (48<L≤96) transform coefficients in the top-left region of the target block with a size of M×N (M≥16, N≥16). The derived R modified transform coefficients may be derived as an output array according to a predetermined scanning order. R is less than L.

[0297] The input region representing the region of the input transform coefficients subjected to the secondary transform in the encoding device may correspond to the ROI described with reference to the output region described in the above decoding method and the accompanying drawings. Therefore, redundant description of the ROI is omitted.

[0298] The number L of input transform coefficients arranged in the input region and the number R of modified transform coefficients derived through matrix operation may vary according to the dimension of the transform kernel. According to an example, R may be 16, 32, 48, 80, etc., and L may be 64 or 96.

[0299] The encoding device 200 generates residual information based on the modified transform coefficients (S1150). For example, the encoding device 200 may generate the residual information based on the modified transform coefficients. For example, the encoding device 200 may derive quantized transform coefficients by performing quantization on the modified transform coefficients, and generate residual information about the quantized transform coefficients. For example, the residual information may include the transform-related information / syntax elements described above.

[0300] The encoding device 200 generates LFNST index information indicating an LFNST matrix (S1160). For example, the encoding device 200 may generate LFNST index information indicating an LFNST matrix in an LFNST set indicated by an LFNST set index. That is, as shown in the table, there may be multiple LFNST sets based on the mapping relationship with the intra prediction mode, and one LFNST set may include multiple LFNST matrices. Which one of the multiple LFNST matrices constituting one LFNST set is applied to LFNST may be related to the LFNST index.

[0301] The encoding device 200 encodes video / image information including intra-frame prediction mode related information, residual information, and LFNST index information (S1170). For example, the encoding device 200 can encode video / image information including information related to intra-frame prediction mode, residual information, and LFNST index information.

[0302] Furthermore, according to embodiments of this disclosure, the video / image information may include various types of information. For example, the video / image information may include information for image reconstruction. Information for image reconstruction may include prediction-related information, transformation-related information, etc. Additionally, the video / image information may include information related to the LFNST applied to the current block. The LFNST-related information may be included in SPS, PPS, PH, or SH. This information includes at least one of the following: information on whether LFNST is applied, information on the minimum transform size for applying LFNST, information on the maximum transform size for applying LFNST, and a transform index indicating which transform kernel is included in the transform set.

[0303] Encoded video / image information can be output as a bitstream. The bitstream can be sent to a decoding device via a network or storage medium.

[0304] Specifically, according to embodiments of this disclosure, video / image information may include various types of information.

[0305] According to the embodiments proposed in this disclosure, the image information may further include LFNST set list index information. The LFNST set list index information may be information used to specify one of a plurality of LFNST set lists. For example, when there are M LFNST set lists, the LFNST set list index information may have values ​​from 0 to M-1.

[0306] At this point, for example, encoding device 200 can determine a list of LFNST sets for the current block associated with the LFNST matrix applied to the transformed coefficients of the derived modification, and can generate and encode index information of the LFNST set list associated with the list of LFNST sets.

[0307] Figure 12 An example of a video / image decoding method according to an embodiment of the present disclosure is illustrated schematically.

[0308] Figure 12 The method disclosed in the article can be derived from Figure 3 The decoding device disclosed in the document performs the operation. Specifically, for example, Figure 12S1200 can be executed by the entropy decoder 310 of the decoding device 300, and S1210 can be executed by the residual processor 320 of the decoding device 300. S1220 can be executed by at least one of the predictor 330 and / or the residual processor 320 of the decoding device 300. In addition, S1230 to S1250 can be executed by the residual processor 320 of the decoding device 300, and S1260 can be executed by at least one of the residual processor 320 and / or the adder 340. Figure 12 The methods disclosed herein may include the embodiments described above.

[0309] refer to Figure 12 Decoding device 300 receives / obtains video / image information from the bitstream (S1200). For example, the video / image information may include residual information, information related to the intra-frame prediction mode, and LFNST index information. For example, decoding device 300 can receive / obtain video / image information from the bitstream including information related to the intra-frame prediction mode, residual information, and LFNST index information.

[0310] Furthermore, according to embodiments of this disclosure, the video / image information may include various types of information. For example, the video / image information may include information for image reconstruction. Information for image reconstruction may include prediction-related information, transformation-related information, etc. Additionally, the video / image information may include information related to the LFNST applied to the current block. The LFNST-related information may be included in SPS, PPS, PH, or SH. This information includes information about whether LFNST is applied, information about the minimum transform size for applying LFNST, information about the maximum transform size for applying LFNST, and a transform index identifying one of the transform kernels included in the transform set. It may include at least one of the related information.

[0311] Decoding device 300 derives transform coefficients for the current block from residual information (S1210). For example, decoding device 300 can derive quantized transform coefficients for the current block from residual information. Furthermore, decoding device 300 can derive transform coefficients for the current block by performing inverse quantization on the quantized transform coefficients.

[0312] The derived transform coefficients can be arranged in a reverse diagonal scan order in 4×4 sub-blocks, and the transform coefficients within the 4×4 sub-blocks can also be arranged in a reverse diagonal scan order. In other words, the inverse-quantized transform coefficients can be arranged according to the reverse scan order used in video codecs such as VVC or HEVC.

[0313] Decoding device 300 derives an LFNST set index for the current block based on the intra-prediction mode (S1220). For example, decoding device 300 may derive an LFNST set index for the current block based on the intra-prediction mode of the current block derived from information related to the intra-prediction mode.

[0314] Here, as an example, information related to the intra-prediction mode can be associated with any of the intra-prediction modes from 0 to 66. Additionally, when the WAIP mode is applied to the current block, after signaling / resolving the information related to the intra-prediction mode, this information can be remapped to the index associated with the WAIP mode.

[0315] The decoding device 300 derives the LFNST matrix for the current block based on the LFNST index and the LFNST set index (S1230). For example, the decoding device 300 can derive the LFNST matrix for the current block based on the LFNST set index and the LFNST index derived from the LFNST index information.

[0316] For example, decoding device 300 can derive an LFNST matrix identified by an LFNST index derived from LFNST index information into an LFNST matrix for the current block from an LFNST set identified by an LFNST set index. That is, as shown in the table above, multiple LFNST sets can exist based on the mapping to intra-frame prediction modes, and an LFNST set can include multiple LFNST matrices. It can be determined based on the LFNST index which of the multiple LFNST matrices constituting an LFNST set should be applied to the LFNST. According to one example, if the width or height of the current block is equal to or greater than 16, the LFNST matrix can be derived as a 96×32 dimensional matrix.

[0317] Decoding device 300 derives modified transform coefficients based on the transform coefficients and the LFNST matrix (S1240). For example, decoding device 300 can derive modified transform coefficients based on the transform coefficients and the LFNST matrix. For example, decoding device 300 can derive modified transform coefficients by performing LFNST on the transform coefficients using the LFNST matrix. For example, the LFNST can be an inverse secondary transform and can include an inseparable secondary transform reflecting the RST. In this case, the LFNST matrix applied to the inverse secondary transform can be a non-square matrix where the number of columns is less than the number of rows.

[0318] Decoding device 300 generates residual samples for the current block based on the modified transform coefficients (S1250). For example, decoding device 300 can generate residual samples for the current block based on the modified transform coefficients. For example, decoding device 300 can generate residual samples for the current block by performing a primary transform on the modified transform coefficients. In this case, the primary transform can be an inverse primary transform. For example, a reduced inverse transform can be applied to the primary transform or a normal separation transform. In addition, the primary transform can be based on multiple core transforms, which are based on multiple transform kernels selected from DCT-2, DCT-7, DCT-8, and DST-1.

[0319] Decoding device 300 generates a reconstructed image based on residual samples (S1260). For example, decoding device 300 can generate a reconstructed image based on residual samples. Furthermore, decoding device 300 can generate a modified reconstructed image by applying various filtering methods to the reconstructed image. The modified reconstructed image can be stored as output and / or stored as a decoded image in a decoded image buffer or memory 360, and can be used as a reference image in inter-frame prediction during subsequent image decoding.

[0320] Specifically, according to embodiments of this disclosure, video / image information may include various types of information.

[0321] According to the embodiments proposed in this disclosure, the image information may further include LFNST set list index information. The LFNST set list index information may be information used to specify one of a plurality of LFNST set lists. For example, when there are M LFNST set lists, the LFNST set list index information may have values ​​from 0 to M-1.

[0322] At this time, for example, the decoding device 300 can derive the LFNST set list for the current block based on the LFNST set list index derived from the LFNST set list index information for the current block.

[0323] Additionally, the LFNST matrix for the current block can be derived from the list of LFNST sets for the current block based on the LFNST index and the LFNST set index.

[0324] In the above embodiments, a flowchart description method with a series of steps or blocks is used. However, this embodiment is not limited to the order of the steps or blocks, and some steps may occur simultaneously or in a different order than the other steps described above. Furthermore, those skilled in the art will understand that the steps shown in the flowcharts are not exclusive and may include other steps, or one or more steps in the flowcharts may be deleted without affecting the scope of the embodiments of this disclosure.

[0325] The methods described above according to the embodiments of this disclosure can be implemented in software. The encoding and / or decoding devices according to this disclosure can be included in apparatuses performing image processing, such as TVs, computers, smartphones, set-top boxes, or display devices.

[0326] When the embodiments of this disclosure are implemented in software, the above methods can be implemented by modules (processing, functions, etc.) that perform the above functions. Such modules can be stored in memory and executed by a processor. The memory can be internal or external to the processor, and the memory can be connected to the processor using various well-known means. The processor may include application-specific integrated circuits (ASICs), other chipsets, logic circuits, and / or data processing devices. The memory may include ROM (read-only memory), RAM (random access memory), flash memory, memory cards, storage media, and / or other storage devices. That is, the embodiments described in this disclosure can be implemented and executed on a processor, microprocessor, controller, or chip. For example, the functional units shown in each figure can be implemented and executed on a computer, processor, microprocessor, controller, or chip. In this case, the information used for implementation (e.g., information about instructions) or algorithms can be stored in a digital storage medium.

[0327] Furthermore, the decoding and encoding devices to which the embodiments of this disclosure apply can be used in multimedia communication devices, such as multimedia broadcasting transmitters and receivers, mobile communication terminals, home theater video devices, digital theater video devices, surveillance cameras, video chat devices, (3D) video devices, video telephony devices, and medical video devices. These devices can be included, for example, in storage media, cameras, video-on-demand (VoD) service providers, OTT video (over-the-top video), internet streaming service providers, 3D video devices, virtual reality (VR) devices, augmented reality (AR) devices, video call devices, and vehicle terminals (e.g., vehicle (including autonomous vehicle) terminals, aircraft terminals, ship terminals, etc.), and can be used to process video signals or data signals. For example, OTT video (over-the-top video) devices can include game consoles, Blu-ray players, internet access TVs, home theater systems, smartphones, tablet PCs, and digital video recorders (DVRs).

[0328] Furthermore, the processing methods applying the embodiments of this disclosure can be generated in the form of a computer-executable program and can be stored in a computer-readable recording medium. Multimedia data having a data structure according to the embodiments of this disclosure can also be stored in a computer-readable recording medium. Computer-readable recording media include various storage devices and distributed storage devices storing computer-readable data. Computer-readable recording media can be, for example, Blu-ray discs (BD), Universal Serial Bus (USB), ROM, PROM, EPROM, EEPROM, RAM, CD-ROM, magnetic tape, floppy disks, and optical data storage devices. Additionally, computer-readable recording media include media implemented in the form of carrier waves (e.g., transmitted via the Internet). Furthermore, bitstreams generated by encoding methods can be stored in computer-readable recording media or transmitted via wired or wireless communication networks.

[0329] Furthermore, the embodiments of this disclosure can be implemented as a computer program product by program code, and the program code can be executed in a computer according to the embodiments of this disclosure. The program code can be stored on a computer-readable medium.

[0330] Figure 13 This is a diagram illustrating the structure of a content streaming system to which embodiments of the present disclosure are applied.

[0331] refer to Figure 13 The embodiments of this disclosure are applied to a content streaming system that may include an encoding server, a streaming server, a network server, a media storage device, a user device, and a multimedia input device.

[0332] An encoding server compresses content input from multimedia input devices such as smartphones, cameras, and camcorders into digital data to generate a bitstream, and then transmits this bitstream to a streaming server. As another example, when multimedia input devices such as smartphones, cameras, and camcorders directly generate bitstreams, the encoding server can be omitted.

[0333] Bitstreams can be generated using the encoding methods or bitstream generation methods described in the embodiments of this document. Furthermore, the streaming server can temporarily store the bitstreams during the sending or receiving process.

[0334] A streaming server sends multimedia data to a user's device via a web server based on a user's request, and the web server acts as a medium for informing the user of services. When a user requests a desired service from the web server, the web server forwards it to the streaming server, which then sends the multimedia data to the user. In this scenario, the content streaming system may include a separate control server. In this case, the control server is used to manage the commands / responses between devices within the content streaming system.

[0335] A streaming server can receive content from media storage devices and / or encoding servers. For example, when receiving content from an encoding server, the content can be received in real time. In this case, to provide a smooth streaming service, the streaming server can store the bit stream for a predetermined period of time.

[0336] For example, user equipment may include mobile phones, smartphones, laptops, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, board PCs, tablet PCs, ultrabooks, wearable devices (e.g., watch-type terminals (smartwatches), glasses-type terminals (smart glasses), head-mounted displays (HMDs)), digital TVs, desktop computers, digital signage, etc.

[0337] Each server in a content streaming system can be operated as a distributed server, and in this case, the data received by each server can be processed in a distributed manner.

[0338] The claims disclosed herein can be combined in various ways. For example, the technical features of the method claims can be combined to be implemented or performed in a device, and the technical features of the device claims can be combined to be implemented or performed in a method. Furthermore, the technical features of the method claims and the device claims can be combined to be implemented or performed in a device, and the technical features of the method claims and the device claims can be combined to be implemented or performed in a method.

Claims

1. An image decoding method performed by a decoding device, the image decoding method comprising the following steps: Image information is obtained through a bitstream, wherein the image information includes residual information, information related to the intra-frame prediction mode, and low-frequency non-separable transform (LFNST) index information; Derive the transformation coefficients for the current block from the residual information; Based on the intra-prediction mode for the current block derived from the information associated with the intra-prediction mode, LFNST set index for the current block is derived. Based on the LFNST set index and the LFNST index derived from the LFNST index information, derive the LFNST matrix for the current block; Based on the transformation coefficients and the LFNST matrix, the modified transformation coefficients are derived; and Based on the modified transformation coefficients, a residual sample for the current block is generated. Specifically, based on the fact that the width or height of the current block is equal to 16, and that both the width and the height are equal to or greater than 16, the LFNST matrix is ​​derived as a 96×32 dimensional matrix. Based on the fact that the LFNST set index is equal to 0, the 96×32 dimensional matrix includes the following first to eighth column vectors. ， Based on the fact that the LFNST set index is equal to 0, the 96×32 dimensional matrix includes the following ninth to sixteenth column vectors. ， Based on the fact that the LFNST set index is equal to 0, the 96×32 dimensional matrix includes the following seventeenth to twenty-fourth column vectors. ， Wherein, based on the LFNST set index being equal to 0, the 96×32 dimensional matrix includes the following twenty-fifth to thirty-second column vectors, and 。 2. An image encoding method performed by an encoding device, the image encoding method comprising the following steps: Determine the intra-prediction mode for the current block, and based on the intra-prediction mode for the current block, determine the low-frequency inseparable transform (LFNST) set index. Based on the intra-frame prediction mode, a prediction sample is generated for the current block; Based on the predicted samples, a residual sample is generated for the current block; Based on the residual samples, the transformation coefficients are derived; Based on the transformation coefficients and the LFNST matrix, the modified transformation coefficients are derived; Based on the modified transformation coefficients, residual information is generated; Generate LFNST index information that identifies the LFNST matrix in the LFNST set identified by the LFNST set index; as well as Image information, including information related to the intra-frame prediction mode, the residual information, and the LFNST index information, is encoded. Specifically, based on the fact that the width or height of the current block is equal to 16, and that both the width and the height are equal to or greater than 16, the LFNST matrix is ​​derived as a 32×96 dimensional matrix. Based on the fact that the LFNST set index is equal to 0, the 32×96 dimensional matrix includes the following first row vector to eighth row vector. ， Based on the fact that the LFNST set index is equal to 0, the 32×96 dimensional matrix includes the following ninth to sixteenth row vectors. ， Based on the fact that the LFNST set index is equal to 0, the 32×96 dimensional matrix includes the following vectors from the seventeenth row to the twenty-fourth row. ， Wherein, based on the LFNST set index being equal to 0, the 32×96 dimensional matrix includes the following twenty-fifth row vector to thirty-second row vector, and 。 3. A method for transmitting data for an image, the method comprising the steps of: Perform the following steps to obtain a bitstream for the image: determine the intra-prediction mode for the current block; determine the Low Frequency Inseparable Transform (LFNST) set index based on the intra-prediction mode for the current block; generate prediction samples for the current block based on the intra-prediction mode; and generate residual samples for the current block based on the prediction samples. Based on the residual samples, the transformation coefficients are derived; Based on the transformation coefficients and the LFNST matrix, the modified transformation coefficients are derived; based on the modified transformation coefficients, residual information is generated. Generate LFNST index information that identifies the LFNST matrix in the LFNST set indexed by the LFNST set index; and encode image information including information related to the intra-frame prediction mode, the residual information, and the LFNST index information; as well as Send the data including the bit stream. Specifically, based on the fact that the width or height of the current block is equal to 16, and that both the width and the height are equal to or greater than 16, the LFNST matrix is ​​derived as a 32×96 dimensional matrix. Based on the fact that the LFNST set index is equal to 0, the 32×96 dimensional matrix includes the following first row vector to eighth row vector. ， Based on the fact that the LFNST set index is equal to 0, the 32×96 dimensional matrix includes the following ninth to sixteenth row vectors. ， Based on the fact that the LFNST set index is equal to 0, the 32×96 dimensional matrix includes the following vectors from the seventeenth row to the twenty-fourth row. ， Wherein, based on the LFNST set index being equal to 0, the 32×96 dimensional matrix includes the following twenty-fifth row vector to thirty-second row vector, and 。

Images