Video encoder, video encoding method, video decoder, video decoding method

Abstract

A video encoding method, a video decoding method, a video encoder, and a video decoder are provided. The video decoder includes an entropy decoding module, an inverse quantization module, and an inverse transform module. The entropy decoding module receives an input bitstream. The entropy decoding module decodes the input bitstream to obtain an NSPT index. The inverse quantization module is coupled to the entropy decoding module and performs inverse quantization to generate a plurality of reconstructed coefficients. The inverse transform module is coupled to the inverse quantization module and, if a current coding block (CB) has a specific block size and the value of the NSPT index is 1, 2, or 3, performs an inverse non-separable linear transform (NSPT) on the plurality of reconstructed coefficients based on a kernel having a specific dimension corresponding to the specific block size to generate a reconstructed residual block. Here, the specific block size includes an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 63 / 439,052, filed January 13, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present disclosure relates to the field of image data processing, and in particular to video encoding methods, video decoding methods, video encoders, and video decoders. [Background technology]

[0003] In conventional image coding techniques, how to efficiently predict and compress data has always been an important challenge in the field. In particular, new transform methods (i.e., non-separable primary transforms (NSPTs)) can achieve efficient data compression for small coding blocks, but their computational cost is high when applied to large coding blocks. Summary of the Invention [Problem to be solved by the invention]

[0004] What is needed is a new image processing method that efficiently encodes video data and efficiently decodes the corresponding bitstream. [Means for solving the problem]

[0005] The video decoder of the present invention includes an entropy decoding module, an inverse quantization module, and an inverse transform module. The entropy decoding module is configured to receive an input bitstream. The entropy decoding module is configured to decode the input bitstream to obtain an NSPT index. The inverse quantization module is coupled to the entropy decoding module and configured to perform inverse quantization to generate a plurality of reconstructed coefficients. The inverse transform module is coupled to the inverse quantization module. When a current coding block (CB) has a specific block size and the value of the NSPT index is 1, 2, or 3, the inverse transform module performs an inverse non-separable linear transform (NSPT) on the plurality of reconstructed coefficients based on a kernel having a specific dimension corresponding to the specific block size to generate a reconstructed residual block. Here, the specific block size includes an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size.

[0006] In one embodiment of the present invention, the video decoder includes a prediction module configured to determine a predicted block of the current CB by intra prediction if the flag or set of flags indicates application of intra prediction.

[0007] In one embodiment of the present invention, if the current CB has a block size of 8x16, the inverse transform module performs inverse NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 128xB1xCxD, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0008] In one embodiment of the present invention, if the current CB has a block size of 16x8, the inverse transform module performs inverse NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 128xB1xCxD, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0009] In one embodiment of the present invention, the parameter B1 is 32 or 36.

[0010] In one embodiment of the present invention, parameter B1 is close to, but not equal to, 32 or 36.

[0011] In one embodiment of the present invention, the inverse transform module is configured to perform inverse NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 64 × B2 × C × D when the current CB has a block size of 16 × 4. Parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0012] In one embodiment of the present invention, if the current CB has a block size of 4x16, the inverse transform module performs inverse NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 64xB2xCxD, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0013] In one embodiment of the present invention, parameter B2 is 20 or 16.

[0014] In one embodiment of the present invention, parameter B2 is close to, but not equal to, 20 or 16.

[0015] In one embodiment of the present invention, the inverse transform module is configured to perform inverse NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 32×B3×C×D when the current CB has a block size of 8×4, where parameter B3 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0016] In one embodiment of the present invention, the inverse transform module is configured to perform inverse NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 32×B3×C×D when the current CB has a block size of 4×8. Parameter B3 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0017] In one embodiment of the present invention, parameter B3 is 16.

[0018] In one embodiment of the present invention, parameter B3 is close to, but not equal to, sixteen.

[0019] The video decoding method of the present invention includes the steps of receiving an input bitstream, entropy decoding the input bitstream to obtain an NSPT index, performing inverse quantization to generate a plurality of reconstructed coefficients, and, if a current coding block (CB) has a specific block size and the value of the NSPT index is 1, 2, or 3, performing an inverse non-separable linear transform (NSPT) on the plurality of reconstructed coefficients based on a kernel having a specific dimension corresponding to the specific block size to generate a reconstructed residual block. The specific block size includes an 8×16 block size, a 16×8 block size, a 16×4 block size, a 4×16 block size, an 8×4 block size, or a 4×8 block size.

[0020] In one embodiment of the present invention, if the flag or set of flags indicates the application of intra prediction, the predicted block of the current CB is determined by intra prediction.

[0021] In one embodiment of the present invention, if the current CB has a block size of 8x16, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of specific dimensions 128xB1xCxD. Parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0022] In one embodiment of the present invention, if the current CB has a block size of 16x8, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of specific dimensions 128xB1xCxD. Parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0023] In one embodiment of the present invention, the parameter B1 is 32 or 36.

[0024] In one embodiment of the present invention, parameter B1 is close to, but not equal to, 32 or 36.

[0025] In one embodiment of the present invention, if the current CB has a block size of 16x4, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of specific dimensions 64xB2xCxD. Parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0026] In one embodiment of the present invention, if the current CB has a block size of 4x16, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of specific dimensions 64xB2xCxD. Parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0027] In one embodiment of the present invention, parameter B2 is 20 or 16.

[0028] In one embodiment of the present invention, parameter B2 is close to, but not equal to, 20 or 16.

[0029] In one embodiment of the present invention, if the current CB has a block size of 8x4, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of specific dimensions 32xB3xCxD. Parameter B3 is the number of transform coefficients generated by the NSPT on the encoder side, parameter C is 3, and parameter D is 35.

[0030] In one embodiment of the present invention, if the current CB has a block size of 4x8, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of specific dimensions 32xB3xCxD. Parameter B3 is the number of transform coefficients generated by the NSPT on the encoder side, parameter C is 3, and parameter D is 35.

[0031] In one embodiment of the present invention, parameter B3 is 16.

[0032] In one embodiment of the present invention, parameter B3 is close to, but not equal to, sixteen.

[0033] The video encoder of the present invention includes a computation module, a transform module, a quantization module, and an entropy coding module. The computation module is configured to calculate a residual block based on a current CB and a prediction block. The transform module is coupled to the computation module and configured to perform a non-separable linear transform (NSPT) on the residual block based on a kernel having a specific dimension corresponding to a specific block size when the residual block has a specific block size and the determined NSPT index value is 1, 2, or 3, to generate multiple coefficients. The specific block sizes include an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size. The quantization module is coupled to the transform module and configured to quantize the multiple coefficients to generate multiple quantized coefficients. The entropy coding module is coupled to the quantization module and configured to encode the multiple quantized coefficients into an output bitstream. The entropy coding module is configured to encode the NSPT index into the output bitstream.

[0034] In one embodiment of the present invention, the video encoder includes a prediction module configured to generate a prediction block of the current CB by intra prediction.

[0035] In one embodiment of the present invention, when the residual block has a block size of 8x16, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 128xB1xCxD, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0036] In one embodiment of the present invention, when the residual block has a block size of 16x8, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 128xB1xCxD. Parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0037] In one embodiment of the present invention, the parameter B1 is 32 or 36.

[0038] In one embodiment of the present invention, parameter B1 is close to, but not equal to, 32 or 36.

[0039] In one embodiment of the present invention, when the residual block has a block size of 4×16, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 64×B2×C×D, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0040] In one embodiment of the present invention, when the residual block has a block size of 16 × 4, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 64 × B2 × C × D. Parameter B2 is the number of transform coefficients generated by the NSPT on the encoder side, parameter C is 3, and parameter D is 35.

[0041] In one embodiment of the present invention, parameter B2 is 20 or 16.

[0042] In one embodiment of the present invention, parameter B2 is close to, but not equal to, 20 or 16.

[0043] In one embodiment of the present invention, when the residual block has a block size of 8 × 4, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 32 × B3 × C × D. Parameter B3 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0044] In one embodiment of the present invention, when the residual block has a block size of 4×8, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of specific dimensions 32×B3×C×D, where parameter B3 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

[0045] In one embodiment of the present invention, parameter B3 is 16.

[0046] In one embodiment of the present invention, parameter B3 is close to, but not equal to, sixteen.

[0047] In one embodiment of the present invention, a zero-out is performed on the residual block, and the residual block after the zeroing has zeroed regions.

[0048] In one embodiment of the present invention, the transform module is further configured to split the residual block into multiple smaller transform blocks if the size of the residual block is larger than the block size supported by the largest NSPT kernel.

[0049] In one embodiment of the present invention, an NSPT index is signaled for each of multiple small transform blocks.

[0050] In one embodiment of the present invention, an NSPT index is signaled for each unique transform block size, and multiple small transform blocks with the same block size share a common NSPT index.

[0051] The video encoding method of the present invention includes the steps of: calculating a residual block based on a current CB and a predicted block; if the residual block has a specific block size and the determined NSPT index value is 1, 2, or 3, performing a non-separable linear transform (NSPT) on the residual block based on a kernel having a specific dimension corresponding to the specific block size to generate a plurality of coefficients, where the specific block size includes an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size; quantizing the plurality of coefficients to generate a plurality of quantized coefficients; entropy encoding the plurality of quantized coefficients to encode them into an output bitstream; and encoding the NSPT index into the output bitstream. [Effects of the Invention]

[0052] Based on the above, the video encoding method, video decoding method, video encoder, and video decoder of the present invention can effectively improve the encoding performance of video encoding.

[0053] To make the above content easier to understand, several embodiments are described in detail below in conjunction with drawings. [Brief explanation of the drawings]

[0054] [Figure 1] 1 is a schematic diagram of a video encoder according to an embodiment of the present invention; [Figure 2] FIG. 2 is a schematic diagram of a video encoding process according to an embodiment of the present invention; [Figure 3] 1 is a flowchart of a video encoding method according to an embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram of an image divided into a plurality of blocks according to an embodiment of the present invention. [Figure 5]1 is a schematic diagram showing a CTU divided into multiple CUs according to an embodiment of the present invention; [Figure 6A] FIG. 10 is a schematic diagram of LFNST for block sizes of 4×N and N×4 according to an embodiment of the present invention. [Figure 6B] FIG. 1 is a schematic diagram of LFNST for large block sizes according to an embodiment of the present invention. [Figure 7A] FIG. 10 is a schematic diagram of LFNST for block sizes of 4×N and N×4 according to an embodiment of the present invention. [Figure 7B] FIG. 10 is a schematic diagram of LFNST for block sizes of 8×N and N×8 according to an embodiment of the present invention. [Figure 7C] FIG. 10 is a schematic diagram of LFNST for block sizes of 16×N and N×16 according to an embodiment of the present invention. [Figure 8] FIG. 2 is a schematic diagram of a zeroing region according to an embodiment of the present invention. [Figure 9A] FIG. 2 is a schematic diagram of residual block division according to an embodiment of the present invention; [Figure 9B] FIG. 10 is a schematic diagram of residual block division according to another embodiment of the present invention; [Figure 10] FIG. 4 is a schematic diagram of local prediction according to another embodiment of the present invention; [Figure 11] 1 is a schematic diagram of a video decoder according to an embodiment of the present invention; [Figure 12] FIG. 2 is a schematic diagram of a video decoding process according to an embodiment of the present invention; [Figure 13] 3 is a flowchart of a video decoding method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0055] In order to provide a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings, which are provided for reference and description purposes only and are not intended to limit the embodiments of the present application.

[0056] Modern international video coding standards usually describe a block-based hybrid method for decoding a bitstream. For example, the High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) standards are block-based hybrid spatial and temporal predictive coding schemes. That is, to generate such a bitstream when encoding each image, the image is divided into multiple blocks and then each of these blocks is coded.

[0057] FIG. 1 is a schematic diagram of a video encoder according to an embodiment of the present invention. Referring to FIG. 1, the video encoder 100 includes a processor 110, a storage device 120, a communication interface 130, and a data bus 140. The processor 110 is electrically connected to the storage device 120 and the communication interface 130 via the data bus 140. In an embodiment of the present invention, the storage device 120 may store associated instructions and may further store associated video encoder algorithms. The processor 110 may output a bitstream to the communication interface 130. The processor 110 may execute the associated instructions to implement the video encoding method of the present invention.

[0058] In one embodiment of the present invention, video encoder 100 may be implemented by one or more personal computers (PCs), one or more server computers, and one or more workstation computers, or may be configured with multiple computing devices, but the present invention is not limited thereto. In another embodiment of the present invention, video encoder 100 may include more processors for executing associated video encoders and / or associated instructions to implement the video encoding method of the present invention. Also, in one embodiment of the present invention, video encoder 100 may include more processors for executing associated video encoders, associated video decoders, and / or associated instructions to implement the video encoding method of the present invention. Video encoder 100 may be used to implement a video codec and may perform the video encoding and video decoding functions of the present invention.

[0059] In one embodiment of the present invention, processor 110 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), other similar processing circuitry, or a combination of these devices. In one embodiment of the present invention, storage device 120 may be a non-transitory computer-readable recording medium, such as, but not limited to, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a non-volatile memory (NVM).

[0060] In one embodiment of the present invention, the associated video encoder and / or associated instructions may be stored on a non-transitory computer-readable storage medium of one device and executed by a processor of another device. Communication interface 130 may be, for example, a network card supporting a wired network connection such as Ethernet, a wireless network card supporting a wireless communication standard such as Institute of Electrical and Electronics Engineers (IEEE) 802.11n / b / g / ac / ax / be, or any other network connection device, but this embodiment is not limited thereto. Communication interface 130 is configured to acquire input video.

[0061] Figure 2 is a schematic diagram of a video encoding process according to an embodiment of the present invention. Referring to Figures 1 and 2, the video encoder 100 can encode an input video into an output bitstream by performing the video encoding process of Figure 2. The storage device 120 can store algorithms for a division module 201, an arithmetic module 202 (e.g., an adder module or a subtractor module), a prediction module 203, a transform module 204, a quantization module 205, an entropy encoding module 206, an inverse quantization module 207, an inverse transform module 208, a reconstruction module 209 (e.g., an adder module or a subtractor module), a filtering module 210, and a decoded image buffer module 211. The processor 110 can execute the above modules to perform the video encoding process.

[0062] In one embodiment of the present invention, the processor 110 can receive input video from an external video source. The partitioning module 201 receives the input video, divides each image of the input video into multiple coding tree units (CTUs), and divides each CTU into one or more CUs to generate multiple coding units (CUs). Each CU is composed of one or more spatially co-located coding blocks (CBs), each of which corresponds to a color component of the video. Thus, the partitioning module 201 equivalently generates multiple coding blocks (CBs). The prediction module 203 can receive the current CB and perform intra prediction to generate a prediction block for the current CB, although the present invention is not limited thereto. The prediction module 203 can alternatively perform inter prediction, motion prediction, and / or other prediction to generate a prediction block for the current CB. The calculation module 202 receives the current CB and a prediction block and performs a subtraction operation between the current CB and the prediction block to generate a residual block.

[0063] The transform module 204 may perform a non-separable linear transform (NSPT) on the residual block based on an (NSPT) kernel to transform the data of the residual block into a plurality of coefficients (also referred to as transform coefficients). The transform module 204 may alternatively perform other transforms, such as a Karhunen-Loeve transform (KLT), a two-dimensional discrete cosine transform (DCT), and / or a low-frequency non-separable quadratic transform (LFNST). The quantization module 205 may further quantize the plurality of coefficients to generate a plurality of quantized coefficients. The entropy coding module 206 may then encode the plurality of quantized coefficients to generate an output bitstream. The entropy coding module 206 may first binarize the plurality of quantized coefficients into a series of binary bins and then apply an entropy coding algorithm to compress the binary bins into coded bits. Examples of binarization methods include, but are not limited to, truncated unary code, combined truncated Rice (TR) and restricted k-th order exponential-Golomb code (EGk) binarization, and k-th order exponential-Golomb code binarization. Examples of entropy coding algorithms include, but are not limited to, variable length coding (VLC), context-adaptive VLC (CAVLC), arithmetic coding, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), and probability interval partitioning entropy (PIPE). The entropy coding module 206 can further encode other parameters (e.g., partition mode flag, prediction mode flag, coded block flag, sub-block coded flag, etc.) required for decoding images from the video encoder 100 into the output bitstream. The video encoder 100 can then output the output bitstream.

[0064] The inverse quantization module 207 may perform a scaling operation on the quantized coefficients to output a plurality of reconstructed coefficients. The inverse transform module 208 may perform one or more inverse transforms corresponding to the transform in the transform module 204 to output a reconstructed residual block. The reconstruction module 209 may calculate a reconstructed CB by adding the reconstructed residual block and a prediction block of the current CB generated by the prediction module 203. The reconstruction module 209 may send the reconstructed CB to the prediction module 203 to be used as an intra-prediction reference. After all CBs in the current image or current sub-image are reconstructed, the reconstruction module 209 may generate a reconstructed image or a reconstructed sub-image by merging the reconstructed CBs. The filtering module 210 may perform loop filtering on the reconstructed image or the reconstructed sub-image. The filtering module 210 may include one or more in-loop filtering operations, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, a luma mapping with chroma scaling (LMCS) filter, and a neural network-based loop filter (NNLF). The output of the filtering module 210 is a decoded image or a decoded sub-image, which may be buffered in a decoded image buffer module 211. The decoded image buffer module 211 may output the decoded image or the decoded sub-image based on timing and control information. The images stored in the decoded image buffer module 211 may also be used as references by the prediction module 203 to perform inter-prediction or intra-prediction.

[0065] 3 is a flowchart of a video encoding method according to an embodiment of the present invention. Referring to FIGS. 1 to 3, the video encoder 100 can implement the video encoding method by performing the following steps S310 to S350. In step S310, the calculation module 202 can calculate a residual block based on a current CB and a predicted block. In detail, the division module 201 can first receive a current image of the input video and divide the current image into multiple CBs.

[0066] FIG. 4 is a schematic diagram of an image divided into multiple blocks according to an embodiment of the present invention. As shown in FIG. 4, a current image 400 may first be divided into square blocks called CTUs 401. For example, the CTUs 401 may be blocks of 256×256 pixels. Referring further to FIG. 5, FIG. 5 is a schematic diagram of a current CTU divided into multiple CUs according to an embodiment of the present invention. In an embodiment of the present invention, each CTU 401 in the image may be further divided into one or more CUs 402. Each CU 402 may be rectangular or square, and each CU 402 may be the same size as the root CTU 401 or may be divided into subdivisions of up to 4×4 blocks from the root CTU 401, as shown in FIG. 5. Each CU 402 is composed of one or more spatially co-located coding blocks (CBs), each corresponding to one color component of the video.

[0067] Next, the prediction module 203 can generate a predicted block of the current CB. In one embodiment of the present invention, the prediction module 203 generates the predicted block of the current CB by intra prediction. After prediction, the residual may still have high spatial correlation. Although conditional entropy coding can capture some of the spatial dependence between neighboring samples, it is computationally impractical to create an entropy coding statistical model that can fully exploit the spatial correlation within the residual. In contrast, transform coding is a practical and effective method for spatially decorrelating the residual.

[0068] JPEG2026503449000002.jpg121161

[0069] According to this second interpretation, transform coding gain, and thus the overall coding gain of a video coding system, can be achieved if the resulting transform coefficients exhibit energy compaction properties, i.e., the variance distribution is concentrated in a small number of transform coefficients compared to the original residual samples, which are likely to be evenly distributed.

[0070] The use of one-dimensional transforms applied separately in the horizontal and vertical directions has good computational scalability as block size increases. In the above example, the transform coefficients are obtained by a matrix realization of the DCT, which results in (M + N) multiplications per sample. By using a "butterfly" decomposition, even fewer multiplications per sample can be achieved, at the expense of a slightly higher computational delay. Separable transforms can achieve optimal energy compaction for spatial features along Cartesian directions (i.e., vertical or horizontal). For example, vertical edges are fully compressed by a vertical DCT. However, separable transforms cannot optimally exploit spatial features along directions other than Cartesian directions. In such cases, a properly designed non-separable transform can achieve higher coding performance.

[0071] While a separable transform applies a one-dimensional transformation in the horizontal and vertical directions, a two-dimensional non-separable transform is applied directly to a block of input samples. One desirable property of a transform is that the transform vector spans the space of input samples. This means that any input vector (i.e., any combination of input sample values) can be represented by a weighted sum of transform vectors. One necessary condition for a transform to span is that there must be at least as many transform vectors as the number of dimensions of the input space; in other words, the number of output transform coefficients must be equal to or greater than the number of input samples. For example, the one-dimensional DCT in VVC is a spanning transform. Thus, for a spanning non-separable transform, if a block of input samples is an M × N residual, the transform also outputs a block of M × N transform coefficients, which can be realized by a matrix implementation of (M × N) × (M × N) multiplications.

[0072] To derive a non-separable transform that produces coding gain for a particular directional feature, the transform can be trained. For example, a representative set of residual blocks corresponding to the directional feature of interest can be grouped together, and then the KLT can be calculated from the covariance matrix of the set of residual blocks. This process can be repeated across k different sets of residual blocks. Thus, in this example, an overall transform kernel of dimension (M × N) × (M × N) × K is derived.

[0073] The spanning non-separable transforms described in this section have two problems. First, they have high computational complexity. Non-separable transforms are usually learned and therefore generally not decomposable. The matrix implementation of the spanning non-separable transform in the above example requires (M × N) multiplications per sample. The second problem is that the transform kernels occupy a large amount of storage space in the encoder and decoder. In the above example, a single kernel that adapts to K different directional features requires (M × N) × (M × N) × K weights. This kernel can only be applied to residual blocks of size M × N. To apply a non-separable transform to multiple block sizes, a transform kernel must be learned for each discrete block size.

[0074] Therefore, VVC introduced a low-frequency non-separable quadratic transform (LFNST) tool with several improvements to solve the above problems of spanning non-separable transforms. First, although the LFNST tool is applicable to a wide range of block sizes, only two types of LFNST kernels are defined. A small LFNST kernel is applied to block sizes of 4×N or N×4 (N≧4). A large LFNST kernel is applied to large block sizes (e.g., 8×8 or larger). Referring to Figures 6A and 6B, the sample positions affected by LFNST are additionally shown. For example, for a block size of 4×N or N×4 from the encoder side, the top-left 4×4 sample position (shown as the shaded area in Figure 6A) is transformed by the small LFNST. The remaining sample positions (shown as the white area in Figure 6A) are ignored, or "zeroed." From the decoder side, an inverse LFNST is applied to generate the top-left 4×4 sample, and the remaining samples are filled with zeros. A similar strategy applies for larger block sizes: LFNST operates on the top-left three 4x4 block sample positions (shown as the shaded area in Figure 6B). The remaining sample positions are zeroed.

[0075] As a result of the "zeroing" strategy, LFNST achieves significant size reduction compared to a full-size transform applied to all sample positions. However, it is inherently lossy and cannot recover values ​​at sample positions ignored by LFNST. If the LFNST tool were applied directly to the residual samples, these losses would be too great for the LFNST tool to use. However, LFNST is called a second-order transform because it is applied after a separable DCT is first performed in the encoder, operating on the first-order transform coefficients to produce the second-order transform coefficients. In other words, the DCT can be considered a first-order transform. In this disclosure, the leftmost sample position in a block of first-order transform coefficients corresponds to the horizontal low frequencies of the DCT, and the topmost sample position corresponds to the vertical low frequencies of the DCT. By preferentially transforming the top-left sample position and reconstructing it at the decoder, LFNST is able to reconstruct low-frequency information from the original residual. As mentioned above, transforms provide coding gain through their energy compaction properties, and it has been experimentally well established that the variance (energy) of images and video signals captured by a camera is primarily concentrated in the low-frequency DCT coefficients. Thus, although "zeroing" does not allow LFNST to reconstruct arbitrary residual blocks losslessly, in practice the loss is minimal for most types of image and video signals.

[0076] The second modification is that the applied transform is not a spanning transform for both the small and large LFNST kernels. From the encoder's perspective, the number of output (secondary transform) coefficients is smaller than the number of input (primary transform) coefficients. For example, the small LFNST kernel takes 4 × 4 = 16 primary transform coefficients as input but outputs only 8 secondary transform coefficients. The large LFNST kernel takes 3 × 4 × 4 = 48 primary transform coefficients as input and outputs 8 secondary transform coefficients. While the use of a non-spanning transform incurs additional reconstruction loss, this loss can be controllably traded off against the achieved complexity reduction. A spanning non-separable transform can be first designed using the KLT method described above. According to this method, the basis vectors of the transform correspond to the eigenvectors of the covariance matrix computed from a representative set of residual blocks. These eigenvectors are ranked by importance according to their corresponding eigenvalues, and the most important eigenvectors are selected to construct the non-spanning non-separable transform. For example, to form a non-spanning transform of a small LFNST kernel, the eight eigenvectors with the largest eigenvalues ​​may be selected.

[0077] Therefore, the above two modifications significantly reduce the complexity of the LFNST kernel compared to spanning non-separable transforms. For small blocks, using a small LFNST kernel reduces the potential complexity per transform block from (4 × N) × (4 × N) multiplications (for N ≥ 4) to 16 × 8 multiplications. For large blocks, using a large LFNST kernel reduces the complexity per transform block from (8 × N) × (8 × N) multiplications (for N ≥ 8) to 48 × 8 multiplications.

[0078] The LFNST kernel does not consist of only a single transformation matrix. Multiple transformation matrices are learned to achieve higher coding gains for various image and video signals. The number of different transformation matrices is the product of the third and fourth dimensions of the LFNST kernel. A small LFNST kernel has 16x8x2x4 dimensions, while a large LFNST kernel has 48x8x2x4 dimensions. The LFNST kernel is expressed with two additional dimensions because the specific transformation matrix to apply to a transform block is selected through a combination of explicit signaling and implicit selection.

[0079] Explicit signaling is achieved by an LFNST index signaled in the bitstream, which can take values ​​of 0, 1, or 2, where 0 indicates that LFNST is not used for the transform block, and values ​​of 1 or 2 indicate the selection of the third dimension of the LFNST kernel. The drawbacks of potential reconstruction loss due to zeroing and non-spanning simplifications are mitigated by the explicit signaling mechanism. If using LFNST for a transform block results in excessive reconstruction loss, the LFNST tool can be disabled by signaling the LFNST index as 0.

[0080] Implicit selection is enabled by restricting LFNST to only coding blocks that use intra prediction. Intra prediction generates a prediction block for a coding block from neighboring reference samples above and to the left of the current block. The specific method for constructing the prediction block is signaled in the bitstream by the intra prediction mode. Simple methods for intra prediction include the "DC" mode, which averages reference samples, and the "planar" mode, which performs affine interpolation between several reference samples. However, most intra prediction modes are reserved for signaling intra angle directions, in which the prediction block is constructed assuming that the values ​​of the reference samples are replicated in a specific direction. If an intra angle direction is used, this may strongly indicate the directional characteristics of the residual block. Implicit selection of the LFNST transform is performed by mapping the intra prediction mode to one of four possible values ​​of the "transform set index," which is used to index the fourth dimension of the LFNST kernel. The mapping used in VVC is shown in Table 1 below (i.e., mapping from intra prediction mode to LFNST transform set index).

[0081] [Table 1]

[0082] Intra prediction modes 0 and 1 correspond to intra prediction plane mode and intra DC prediction mode, respectively. These modes are treated as special cases by mapping them to transform set index 0. Other intra prediction modes correspond to intra angular directions. Intra prediction mode 2 corresponds to diagonal intra angular prediction from the bottom left. As the intra prediction mode number increases, the intra prediction direction rotates clockwise, with intra prediction mode 34 corresponding to diagonal intra angular prediction from the top left and intra prediction mode 66 corresponding to diagonal intra angular prediction from the top right.

[0083] For intra-prediction modes greater than 34, corresponding to intra-angular prediction directions clockwise from the top-left diagonal, the selected LFNST transform matrix is ​​applied to the primary transform coefficients in transposed form. In one embodiment, this is achieved by scanning the primary transform coefficients in the transposed direction before applying the LFNST transform. For example, if the current block, from the encoder's perspective, is predicted using intra-prediction mode 2, the primary transform coefficients are rearranged from their two-dimensional pattern within the block into a one-dimensional vector using a row-majoRscan before applying the selected LFNST transform matrix. Thus, in this example, if the current block were instead predicted using intra-prediction mode 66 and the same signaled LFNST indexes were used, the primary transform coefficients would be rearranged into a one-dimensional vector using a column-majoRscan before applying the same LFNST transform matrix. In another embodiment, the same current block with intra-prediction mode 66 could be equivalently transformed by still performing a row-majoRscan on the primary transform coefficients, but instead rearranging the rows of the transform matrix.

[0084] For more than 34 intra prediction modes, the same LFNST transform matrix can be shared between symmetric intra angular prediction directions by transposing the primary transform coefficients.

[0085] In addition, in research activities following VVC, an extension to LFNST has been proposed and incorporated into the Enhanced Compression Test Model (ECM). The LFNST tool in the ECM achieves higher coding gain by relaxing some of the complexity reduction imposed by the original LFNST tool employed in VVC. Three types of LFNST kernels exist in the ECM. Similar to the LFNST tool in VVC, most transform blocks are zeroed. Referring to Figures 7A-7C, the shaded areas indicate the locations of primary transform coefficients affected by LFNST in the ECM, while the white areas indicate the locations of zeroed transform coefficients. For block sizes of 4xN or Nx4 (N ≥ 4), a small LFNST kernel is applied to the primary transform coefficients of the top-left 4x4 blocks. For block sizes of 8xN or Nx8 (N ≥ 8), a medium-sized LFNST kernel is applied to the primary transform coefficients of the top-left four 4x4 blocks. For blocks of 16x16 or larger, a large LFNST kernel is applied to the primary transform coefficients of the top-left six 4x4 blocks.

[0086] The sizes of the LFNST kernels in ECM are 16x16x3x35 for small LFNST kernels, 64x32x3x35 for medium LFNST kernels, and 96x32x3x35 for large LFNST kernels. Compared to the LFNST tool in VVC, the range of LFNST indices signaled is expanded from 2 to 3, and the number of LFNST transform sets is increased from 4 to 35. The mapping from intra prediction modes to LFNST transform set indices is shown in Table 2 below (i.e., the mapping from intra prediction modes to LFNST transform set indices in ECM).

[0087] [Table 2]

[0088] The complexity of an LFNST tool can be evaluated from three perspectives: first, the additional memory burden imposed on the decoder that must store the LFNST kernel; second, the maximum number of multiplications per sample that the decoder must perform when the LFNST tool is used; and third, the additional number of multiplications per sample that the encoder must perform when performing the LFNST tool's full search. In all three of these criteria, the enhanced LFNST proposed in ECM is more complex than the LFNST in VVC. However, in terms of total multiplications per sample, its worst-case decoder complexity may be lower than that of other transform options.

[0089] In a matrix multiplication realization of the DCT applied individually to transforms of size MxN, the number of multiplications per sample is (M+N). Therefore, the worst-case complexity occurs at a maximum of (M+N). In practice, alternative realizations of the DCT, such as butterfly decomposition, can reduce the complexity, but it is still useful to estimate the complexity of a matrix multiplication realization. In ECM, the separable DCT is extended to include the 128-point DCT as the largest transform. Therefore, the worst-case complexity of the separable DCT is 128+128=256 multiplications per sample.

[0090] The worst-case decoder complex of LFNST in ECM can be evaluated by considering several different block sizes. For a fair comparison, this evaluation also includes the cost of performing the linear transform. For a 4x4 block, the linear transform consists of 4 + 4 = 8 multiplications per sample. LFNST is a 16x16 matrix multiplication, or 16 multiplications per sample. Therefore, the total cost of LFNST for a 4x4 block is 24 multiplications per sample.

[0091] For a 4x8 block, a naive implementation of the linear transform typically requires eight 4x4 transforms along the short dimension and four 8x8 transforms along the long dimension, for a total of 4 + 8 = 12 multiplications per sample. However, because LFNST reconstructs nonzero coefficient values ​​only in the top-left 4x4 block of linear transform coefficients, an optimized decoder can take advantage by performing four 4x4 transforms along the short dimension and then four 4x8 transforms along the long dimension, for a total of 2 + 4 = 6 multiplications per sample. Because the order of separable transforms is generally fixed, in the worst case, the decoder must first perform four 4x8 transforms along the long dimension, then eight 4x4 transforms along the short dimension, for a total of 4 + 4 = 8 multiplications per sample. LFNST is still a 16x16 matrix multiplication, with the cost distributed across the larger block, resulting in 8 multiplications per sample. Therefore, the worst-case cost of LFNST for a 4x8 block is 16 multiplications per sample. This principle generally applies to block sizes of 4xN or Nx4. Secondly, the number of multiplications per sample in a 4xN or Nx4 block is always less than or equal to the number of multiplications per sample in a 4x4 block.

[0092] For an 8x8 block, the linear transform consists of 8 + 8 = 16 multiplications per sample. The LFNST is a 64x32 matrix multiplication, or 32 multiplications per sample. Then the total cost of the LFNST for an 8x8 block is 48 multiplications per sample.

[0093] For 8x16 blocks, assume the optimized decoder again exploits the zeroing property of the LFNST reconstruction. Only the top-left 8x8 block of primary transform coefficients is nonzero. The optimized decoder then exploits this property by performing eight 8x8 transforms along the short dimension, followed by eight 8x16 transforms along the long dimension, for a total of 4 + 8 = 12 multiplications per sample. Alternatively, the optimized decoder could first perform eight 8x16 transforms along the long dimension, followed by 16 8x8 transforms along the short dimension, for a total of 8 + 8 = 16 multiplications per sample. The LFNST adds an additional (64 x 32) / (8 x 16) = 16 multiplications per sample, for a total worst-case complex of 32 multiplications per sample. As before, the number of multiplications per sample for an 8xN or Nx8 block is always less than or equal to the number of multiplications per sample for an 8x8 block.

[0094] For 16x16 blocks, due to the zeroing property of the LFNST reconstruction, only six 4x4 blocks of primary transform coefficients in the pattern shown in Figure 7C have nonzero values. For simplicity, we assume a more relaxed pattern in which only the top-left 12x12 block of the primary transform location has nonzero values. The optimized decoder then exploits this property by first performing 12 12x16 transforms in one dimension, then 16 12x16 transforms in the second dimension, resulting in 9 + 12 = 21 multiplications per sample. The LFNST involves (96x32) / (16x16) = 12 multiplications per sample, for a total of 33 complex multiplications per sample.

[0095] For M × N blocks, where M, N ≥ 16, the optimized decoder performs (12 × 12) / N + 12 multiplications per sample to implement a separable DCT by first performing 12 12 × M transforms in one dimension and then M 12 × N transforms in the second dimension. The worst-case complexity then occurs for the smallest value of N = 16, resulting in 21 multiplications per sample, equal to the complexity of a 16 × 16 block. The LFNST adds an additional (96 × 32) / (M × N) multiplications per sample, which is always less than or equal to the multiplications per sample of a 16 × 16 block. Therefore, the overall complexity of the LFNST within the ECM for larger M × N block sizes is always less than or equal to the multiplications per sample of a 16 × 16 block.

[0096] A comprehensive evaluation of the decoder complexity of the LFNST within ECM across a range of block sizes shows that the worst-case complexity is 48 multiplications per sample (occurs for 8x8 blocks). Somewhat surprisingly, this worst-case complexity includes the cost of performing a separable DCT, but thanks to the zeroing optimizations in the LFNST, it is still significantly lower than the worst-case complexity of performing only the separable DCT (estimated at 256 multiplications per sample).

[0097] As can be seen, the use of non-separable quadratic transforms can significantly reduce complexity by utilizing zeroing of selected regions of linear transform coefficients. However, further coding is possible using NSPT. Initial work on non-separable linear transforms showed that significant gains could be achieved (average rate reduction of 3.43% according to the Bjontegaard index), but the implemented transforms were complex and the kernel weights were obtained by overfitting to a test dataset.

[0098] A practical implementation of NSPT has been proposed. This proposal applies NSPT to small block sizes: 4x4, 4x8, 8x4, and 8x8. For these block sizes, NSPT replaces LFNST. Like LFNST, an NSPT kernel is trained to guide the appropriate matrix for a particular block, both via signaled indices and implicit selection by the intra-prediction mode. Three types of NSPT kernels are proposed. For 4x4 blocks, a small NSPT kernel with dimensions 16x16x3x35 is used. For 4x8 and 8x4 blocks, a medium-sized NSPT kernel with dimensions 32x20x3x35 is used. For 8x8 blocks, a large NSPT kernel with dimensions 64x32x3x35 is used.

[0099] In general, the proposed NSPT does not use zeroing transforms, so the first dimension of each kernel is always equal to the number of samples in the block. For medium- and large-sized NSPT kernels, the second dimension is smaller than the first dimension, and NSPT in these cases is a lossy transform. Similar to LFNST, an NSPT index is signaled in the bitstream and can take values ​​of 0, 1, 2, or 3. Here, 0 indicates that NSPT is not used for the transform block, and values ​​1 through 3 indicate a selection along the third dimension within the corresponding NSPT kernel. The selection along the fourth dimension of the NSPT kernel is determined by a mapping from the intra-prediction mode, in a manner similar to that of the enhanced LFNST in ECM.

[0100] The kernel and block sizes for which NSPT is effective are carefully designed to make NSPT practical. This can be confirmed by comparing the complexity of NSPT for each block size with that of the corresponding LFNST it replaces. For 4x4 blocks, NSPT has a complexity of 16 multiplications per sample compared to the LFNST's complexity of 24 multiplications per sample. For 4x8 and 8x4 blocks, NSPT has a complexity of 20 multiplications per sample compared to the LFNST's complexity of 16 multiplications per sample. For 8x8 blocks, NSPT has a complexity of 32 multiplications per sample compared to the LFNST's complexity of 48 multiplications per sample. Therefore, there is no increase in worst-case decoder complexity.

[0101] In the proposed implementation of NSPT described above, the NSPT kernel is designed to replace the LFNST for a specific block size so that the worst-case decoder complex number of multiplications per sample does not increase. The following embodiments provide multiple solutions to improve the gain of NSPT while keeping the worst-case decoder complex number within a practical range.

[0102] 1-3 again, in step S320, if the residual block has a specific block size and the determined NSPT index is 1, 2, or 3, the transform module 204 performs a non-separable linear transform on the residual block based on a kernel of a specific dimensionality corresponding to the specific block size to generate a plurality of coefficients. The specific block size may be an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size.

[0103] In one embodiment of the present invention, if the residual block has a particular block size of 8x16, 16x8, 16x4, 4x16, 8x4, or 4x8, the NSPT transform matrix is ​​selected from a kernel of dimension AxBxCxD, where the value of parameter C is 3 and the value of parameter D is 35. In this disclosure, dimension "A" represents the number of input samples from the residual block, dimension "B" represents the number of transform coefficients generated by NSPT, dimension "C" represents the number of indexed sets, and dimension "D" represents the number of transform sets. The kernel dimension order described in this disclosure is merely an example for purposes of illustrating how an NSPT transform matrix is ​​selected and may vary in NSPT implementations.

[0104] The NSPT transform matrix is ​​selected by indexing the "C" and "D" dimensions of the kernel, resulting in an NSPT transform matrix of size A x B. The NSPT index can take on a value of 1, 2, or 3, and the value of the NSPT index is used to index the "C" dimension of the kernel. The value of the NSPT index may be determined by video encoder 100 performing rate-distortion optimization (RDO) on all possible values ​​of the NSPT index and selecting the value that minimizes the RD cost. As described above, a predictive block is generated by intra prediction, which is associated with an intra prediction mode. The intra prediction mode is mapped to a transform set index by the mapping process described above with reference to Table 2. The transform set index is used to index the "D" dimension of the kernel.

[0105] JPEG2026503449000005.jpg101161

[0106] JPEG2026503449000006.jpg73161

[0107] JPEG2026503449000007.jpg41161

[0108] Furthermore, for intra prediction modes greater than 34, the LFNST transform matrix T is selected from the NSPT kernel for NxM shaped blocks. For square block shapes, this is the same kernel, and therefore T is the same transform matrix. However, for non-square block sizes, the transform matrix is ​​selected from a different NSPT kernel.

[0109] JPEG2026503449000008.jpg60161

[0110] JPEG2026503449000009.jpg16161

[0111] The kernel and block sizes for which NSPT is effective are designed to make NSPT practically feasible. This can be confirmed by comparing the complexity of NSPT for each block size with that of the corresponding LFNST it replaces. For 4x4 blocks, NSPT has a complexity of 16 multiplications per sample compared to the LFNST's complexity of 24 multiplications per sample. For 4x8 and 8x4 blocks, NSPT has a complexity of 20 multiplications per sample compared to the LFNST's complexity of 16 multiplications per sample. For 8x8 blocks, NSPT has a complexity of 32 multiplications per sample compared to the LFNST's complexity of 48 multiplications per sample. In some cases, NSPT has a lower complexity than the LFNST it replaces, while in other cases, NSPT has a higher complexity. Overall, the NSPT complexity is designed to avoid increasing the burden on the encoder. From the decoder's perspective, the worst-case complexity of NSPT, 32 multiplications per sample, is lower than the worst-case complexity of all transform options the decoder needs to support, so there is no increase in worst-case decoder complexity.

[0112] To improve the coding performance of the NSPT tool, extensions to larger block sizes have been proposed. In this approach, NSPT is extended outside the original set of block sizes (4x4, 4x8, 8x4, and 8x8). NSPT is further applied to block sizes 4x16, 16x4, 8x16, and 16x8, replacing LFNST. In some cases, an NPST kernel with dimensions 64x24x3x35 is used for 4x16 and 16x4 block sizes. In other cases, an NPST kernel with dimensions 128x40x3x35 is used for 8x16 and 16x8 block sizes.

[0113] While the above-mentioned extended NSPT achieves improved coding performance, the increased complexity of the extended NSPT kernel is a problem. For 4x16 and 16x4 block sizes, the extended NSPT has a complexity of 24 multiplications per sample compared to the LFNST matrix multiplication complexity of four multiplications per sample, and also consider the potential complexity of eight multiplications per sample of the matrix-multiplication DCT implementation (in the worst case, the decoder must first perform four 4x16 transforms along the long dimension, then sixteen 4x4 transforms along the short dimension, resulting in 4 + 4 = 8 multiplications per sample). For 8x16 and 16x8 block sizes, the extended NSPT has a complexity of 40 multiplications per sample compared to the LFNST complexity of 32 multiplications per sample.

[0114] For all four block shapes, the extended NSPT kernel has significantly higher complexity than the LFNST it replaces, imposing additional burden on the encoder. The extended NSPT kernel can also increase the worst-case complexity of the transform. Assuming a separable DCT implemented with efficient butterfly decomposition and negligible cost, the previous worst-case complexity is 32 multiplications per sample for LFNST and 32 multiplications per sample for NSPT for an 8x8 block size. However, the 8x16 and 16x8 NSPT kernels have a complexity of 40 multiplications per sample. This increase in worst-case complexity is a serious problem for decoders, since the hardware performing the transform must be able to handle the worst-case throughput for all blocks, regardless of which blocks a particular bitstream processes.

[0115] JPEG2026503449000010.jpg84161

[0116] JPEG2026503449000011.jpg84161

[0117] Specifically, NSPT can replace LFNST for block sizes of 8x16 and 16x8. For these block sizes, an additional NSPT kernel with dimensionality 128xB1xCxD is trained, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder's perspective. Parameter C is set to 3 and parameter D is set to 35 to match the current dimensionality of the extended LFNST kernel in the ECM and the kernel used in the proposed NSPT, but these dimensionality values ​​may change if the overall design of LFNST and NSPT is further modified. The number of transform coefficients (i.e., B1) is set so that the worst-case decoder complex with the extended NSPT and the overall complex of the encoder with the extended NSPT are no worse than the corresponding complex of the LFNST tool it replaces. The worst-case LFNST complex for 8x16 and 16x8 blocks is 32 multiplications per sample. In one embodiment of the present invention, parameter B1=32. The number of transform coefficients is set to B1=32 to ensure that the worst-case decoder complexity with the extended NSPT is no worse than the worst-case decoder complexity of the transform stage before adding the extended NSPT kernel. However, in other embodiments of the invention, parameter B1 is set to a value close to, but not equal to, 32, with the exact value being chosen after empirical verification that the reference encoder complexity does not increase for this value of parameter B1. In another configuration, B1 may be set to 36, which largely takes into account the cost of the butterfly DCT implementation, which the LFNST tool also needs to run. However, in other embodiments of the invention, parameter B1 is set to a value close to, but not equal to, 36, with the exact value being chosen after empirical verification that the reference encoder complexity does not increase for this value of parameter B1.

[0118] JPEG2026503449000012.jpg84161

[0119] JPEG2026503449000013.jpg84161

[0120] Specifically, NSPT can replace LFNST for block sizes of 4x16 and 16x4. For these block sizes, an additional NSPT kernel with dimensionality 64xB2xCxD is trained, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder's perspective. Parameter C is set to 3 and parameter D is set to 35 to match the current dimensionality of the extended LFNST kernel in the ECM and the kernel used in the proposed NSPT, but these dimensionality values ​​may change if the overall design of LFNST and NSPT is further modified. The number of transform coefficients (i.e., B2) is set so that the worst-case decoder complex with the extended NSPT and the overall complex of the encoder with the extended NSPT do not exceed the corresponding complex of the replacing LFNST tool. The worst-case LFNST complex for 4x16 and 16x4 blocks is 12 multiplications per sample. In one embodiment of the present invention, parameter B2=20. However, in another embodiment of the present invention, parameter B2 may be set to a value close to, but not equal to, 20, with the exact value being chosen after empirical verification that the reference encoder complexity does not increase for this value of parameter B2. In another configuration, B2 may be set to 16. However, in another embodiment of the present invention, parameter B2 may be set to a value close to, but not equal to, 16, with the exact value being chosen after empirical verification that the reference encoder complexity does not increase for this value of parameter B2.

[0121] JPEG2026503449000014.jpg83161

[0122] JPEG2026503449000015.jpg84161

[0123] Specifically, NSPT can replace LFNST for block sizes of 4x8 and 8x4. For these block sizes, an additional NSPT kernel with dimensions 32xB3xCxD is trained, where parameter B3 is the number of transform coefficients generated by NSPT from the encoder's perspective. Parameter C is set to 3 and parameter D is set to 35 to match the current dimensions of the extended LFNST kernel in the ECM and the kernel used in the proposed NSPT, but these dimensions may change if the overall design of LFNST and NSPT is further modified. The number of transform coefficients (i.e., B3) is set so that the worst-case decoder complexity with the extended NSPT and the overall complexity of the encoder with the extended NSPT do not exceed the corresponding complexity of the replacing LFNST tool. In one embodiment of the present invention, parameter B3=16. However, in another embodiment of the present invention, parameter B3 is set to a value close to but not equal to 16, and the exact value may be selected after empirically verifying that the reference encoder's complexity does not increase for this value of parameter B3.

[0124] In step S330, the quantization module 207 may quantize the plurality of coefficients to generate a plurality of quantized coefficients. In step S340, the entropy coding module 208 may encode the plurality of quantized coefficients into an output bitstream. In step S350, the entropy coding module 208 encodes the NSPT index into the output bitstream. Therefore, the video encoder 100 and its video coding method can realize a low-complexity NSPT for compressing residual data through the kernel matrix designed above.

[0125] FIG. 8 is a schematic diagram of a zeroing region according to an embodiment of the present invention. Referring to FIGS. 1-3 and 8, the prediction module 203 further performs zeroing on the residual block, resulting in the zeroing region in the zeroed residual block. From the encoder's perspective, zeroing is applied to the input side (residual sample position) of the NSPT kernel, enabling further complexity reduction. In the LFNST tool, the retained DCT transform coefficient positions occupy the upper left (low frequency) part of the transform block, and the zeroing region is located in the lower right (high frequency) part of the transform block. In contrast, in NSPT, the retained residual sample positions are located in the lower right part of the residual block, and the zeroing region is located at the sample position at the upper left edge of the residual block. This is because intra prediction is based on a set of reference samples occupying the upper left neighborhood of the coding block, and the correlation between the reference samples and the coding block samples is highly dependent on spatial proximity. Residual samples (corresponding to prediction errors) tend to have the smallest magnitude at sample positions close to the reference neighborhood. These locations are located at the top left edge of the residual block and are therefore least likely to cause large reconstruction errors if completely ignored by zeroing.

[0126] For example, as shown in Figure 8, Figure 8 shows an example of an 8x8 block. In this example, the top-left edge sample position of thickness 1 is zeroed. From the encoder side, the bottom-right 7x7 sample position (indicated by the shaded area in Figure 8) is transformed by an NSPT kernel of size 49x32x3x35, and the top-left edge sample position (indicated by the white area in Figure 8) is ignored or "zeroed." From the decoder side, an inverse NSPT is applied to generate the bottom-right 7x7 residual sample, and the remaining residual samples are filled with zeros. Compared to the NSPT kernel for an 8x8 block, which has a complexity of 32 multiplications per sample, this kernel achieves a 24% complexity reduction with 24.5 multiplications per sample. Similar zeroing regions can be defined along the top left edge of the NSPT kernel for 4x4, 4x8, and 8x4 block sizes, with the kernel for a 4x4 block being 9x9x3x35 (5 multiplications per sample, 69% complexity reduction) and the kernel for a 4x8 or 8x4 block being 21x20x3x35 (13.1 multiplications per sample, 35% complexity reduction).

[0127] In another embodiment of the present invention, a reduced zeroing region can be applied to some NSPT kernels compared to the example of Figure 8. For example, the zeroing region may be reduced to only a single upper-left sample position of the residual block. The size of the zeroing region can be designed for each NSPT kernel by considering a trade-off between complexity and reconstruction error.

[0128] In another embodiment of the present invention, an expanded zeroed region of thickness N can be applied to some NSPT kernels compared to the example of Figure 8. For example, the thickness of the zeroed region may be expanded to 2 samples wide at the top left edge sample location.

[0129] In another embodiment of the present invention, a reduced zeroing region compared to Figure 8 can be selected depending on the intra prediction mode. For example, if the intra prediction mode is mainly horizontal and the CU is mainly predicted by reference samples on the left side, the reduced zeroing region may be the left edge of the residual block.

[0130] 1 to 3, if the size of the residual block exceeds the block size supported by the maximum NSPT kernel, the transform module 204 can divide the residual block into multiple smaller transform blocks. Specifically, the maximum NSPT kernel size is defined as being applied to an M×N block. For example, M is 16 and N is 16. However, the NSPT tool is effective for block sizes equal to or smaller than P×Q (where P>M and Q>N) and replaces LFNST. The coding block size is defined as R×S. In this embodiment, the coding block exceeds the block size supported by the maximum NSPT kernel, i.e., R≧M and S≧N. An intra prediction mode is signaled for the coding block, and a predictor is determined according to the intra prediction mode from the encoder side and subtracted from the samples of the coding block to generate a residual. In this regard, a non-zero NSPT index is signaled to indicate that NSPT is applied to the residual, and a zero NSPT index indicates that NSPT is not applied to the residual. In this embodiment of the present invention, instead of applying a transform directly to the residual block, the residual block is implicitly partitioned or divided into smaller transform blocks before applying NSPT, and therefore the NSPT of the above embodiment is scalable to a wider range of block sizes.

[0131] FIG. 9A is a schematic diagram of residual block partitioning according to one embodiment of the present invention. FIG. 9B is a schematic diagram of residual block partitioning according to another embodiment of the present invention. Referring to FIGS. 1-3, 9A, and 9B, in one embodiment of the present invention, the R×S residual block of the above embodiment is implicitly partitioned into M×N transform blocks whenever possible. If R and S are not integer multiples of M and N, respectively, small blocks of size (R%M) and (S%N) are generated, where % denotes the "modulus" operator. In one embodiment of the present invention, the implicit partitioning is aligned with the upper left corner of the residual block, so that small blocks, if present, appear below and to the right of the residual block. In another embodiment of the present invention, the implicit partitioning is aligned with the lower right corner of the residual block, so that small blocks, if present, appear above and to the left of the residual block. Both implicit partitioning configurations are shown in FIGS. 9A and 9B. Also, each transform block is processed by the NSPT kernel for the corresponding block size, and the selection of a specific transform matrix is ​​determined by the signaled NSPT index and the mapping of intra prediction modes to transform set indexes.

[0132] Its advantages can be seen by considering the implicit partitioning of an M × N transform block into integer multiples. In this case, the number of multiplications per sample is equal to the number of output transform coefficients, B, generated by the selected NSPT. However, the total number of transform coefficients generated for the coding block is B per transform block. If NSPT were applied directly to the coding block without the implicit partitioning, the NSPT kernel would also need to include B output transform coefficients to achieve the same number of multiplications per sample. However, in this case, the total number of transform coefficients is B for the entire coding block. This amount of transform coefficients is insufficient to reconstruct the coding block without incurring very large reconstruction errors, making the direct application of NSPT to large blocks ineffective for video coding.

[0133] Furthermore, for the embodiment of Figure 8, zeroing can be applied exclusively to transform blocks located on the top or left edge of the residual block, meaning that either a zeroed or full-size version of the same NSPT kernel can be applied depending on the position of the transform block within the residual.

[0134] In one embodiment of the present invention, the NSPT index can be signaled for each transform block, which increases the signaling cost of the NSPT tool but allows for improved reconstruction error and an overall improved rate-distortion tradeoff.

[0135] In one embodiment of the present invention, an NSPT index can be signaled for each unique transform block size. Transform blocks of the same block size share a common NSPT index. Thus, the total number of NSPT indexes signaled can be 1, 2, or 4, depending on whether the residual block is an integer multiple of the maximum block size supported by the NSPT kernel.

[0136] In one embodiment of the present invention, an intra-prediction mode is signaled for a coding block, and then a non-zero NSPT index is signaled to indicate that NSPT is to be applied. Unlike the previous embodiment, if a coding block exceeds the block size supported by the maximum NSPT kernel, from the encoder's perspective, the coding block is implicitly divided into smaller prediction blocks before applying intra-prediction. This implicit division follows the same principle as described above with reference to FIG. 9, except that prediction blocks are generated instead of transform blocks. Each prediction block is predicted according to the signaled intra-prediction mode, and a residual block is generated from each prediction block. However, each predictor is determined from a reference sample immediately adjacent to the upper left boundary of the corresponding prediction block. An example of this prediction process for a prediction block is shown in FIG. 10. FIG. 10 is a schematic diagram of local prediction according to another embodiment of the present invention. From the decoder's perspective, this means that a predictor cannot be determined until the upper and left sides of the prediction block are fully reconstructed. This makes the prediction more localized, thereby reducing the size of the residual, but at the cost of increased order dependency. However, this order dependency is no worse than the worst case that the decoder would already encounter if the CTUs were divided into the smallest possible size for CUs.

[0137] FIG. 11 is a schematic diagram of a video decoder according to an embodiment of the present invention. Referring to FIG. 11, the video decoder 1100 includes a processor 1110, a storage device 1120, a communication interface 1130, and a data bus 1140. The processor 1110 is electrically connected to the storage device 1120 and the communication interface 1130 via the data bus 1140. The storage device 1120 may store associated instructions and may further store associated video decoder algorithms. The processor 1110 may receive a bitstream from the communication interface 1130. The processor 1110 may execute the associated video decoder and / or associated instructions to implement the video decoding method of the present invention.

[0138] In one embodiment of the present invention, the video decoder 1100 may be implemented by one or more personal computers, one or more server computers, and one or more workstation computers, or may be configured with multiple computing devices, but the present invention is not limited thereto. In another embodiment of the present invention, the video decoder 1100 may include more processors for executing associated video encoders and / or associated instructions to implement the video decoding method of the present invention. Furthermore, in one embodiment of the present invention, the video decoder 1100 may include more processors for executing associated video decoders and / or associated instructions to implement the video decoding method of the present invention. The video decoder 1100 may be used to implement a video codec and perform the video decoding function of the present invention.

[0139] In one embodiment of the invention, processor 1110 may include, for example, a CPU, GPU, or other programmable general-purpose or special-purpose microprocessor, DSP, ASIC, PLD, other similar processing circuitry, or a combination of these devices. In one embodiment of the invention, storage device 1120 may be a non-transitory computer-readable storage medium such as ROM, EPROM, EEPROM, or NVM, although the invention is not limited thereto.

[0140] In one embodiment of the present invention, the associated video decoder and / or associated instructions may also be stored in a non-transitory computer-readable recording medium of one device and executed by a processor of another device. The communication interface 1130 may be, for example, a network card supporting a wired network connection such as Ethernet, a wireless network card supporting a wireless communication standard such as IEEE 802.11n / b / g / ac / ax / be, or any other network connection device, but this embodiment is not limited thereto. The communication interface 1130 is configured to obtain an input bitstream.

[0141] Figure 12 is a schematic diagram of a video decoding process according to an embodiment of the present invention. Referring to Figures 11 and 12, the video decoder 1100 can decode an input bitstream into an output video by performing the video decoding process of Figure 12. In one embodiment of the present invention, the storage device 120 can store algorithms for an entropy decoding module 1201, a prediction module 1202, an inverse quantization module 1203, an inverse transform module 1204, a reconstruction module 1205 (e.g., an addition module or a subtraction module), a filtering module 1206, and a decoded image buffer module 1207. The processor 1110 can execute the above modules to perform the video decoding process.

[0142] In one embodiment of the present invention, the processor 1110 may receive an input bitstream from an external video source. The entropy decoding module 1201 may parse the input bitstream and obtain values ​​of syntax elements from the input bitstream. The entropy decoding module 1201 may decode the input bitstream to obtain quantized coefficients and may decode entropy-coded syntax elements from the input bitstream. The entropy decoding module 1201 may convert binary representations of the entropy-coded syntax elements into numeric values. The entropy decoding module 1201 may transmit the values ​​of the syntax elements and one or more variables set or determined according to the values ​​of the syntax elements to a module within the video decoder 1200 to obtain one or more decoded images. The prediction module 1202 may determine a prediction block for a current coding block (CB). Note that although this process is decoding, such a block is still referred to as a coding block. The prediction module 1202 may include an intra-prediction module. If the prediction module 1202 indicates that an intra prediction mode is used for decoding the current CB, it can send related parameters from the entropy decoding module 1201 to the intra prediction module to obtain an intra prediction block.

[0143] The inverse quantization module 1203 may inverse quantize the quantized coefficients to generate a plurality of reconstructed coefficients. The inverse transform module 1204 may perform inverse NSPT on the reconstructed coefficients to generate a reconstructed residual block based on an (NSPT) kernel. The reconstruction module 1205 may generate a reconstructed CB according to the reconstructed residual block and the prediction block. Furthermore, the reconstructed CB may be sent to the prediction module 1202 as a reference for other blocks coded in intra prediction mode.

[0144] After all CBs in the current image or current sub-image are reconstructed, the reconstructed CBs are merged to generate a reconstructed image or sub-image. The filtering module 1206 can perform in-loop filtering on the reconstructed image or sub-image. The filtering module 1206 can include one or more in-loop filtering operations, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, a luma mapping and chromaticity scaling (LMCS) filter, and a neural network-based loop filter (NNLF). The filtering module 1206 outputs a decoded image or sub-image, which is buffered in a decoded image buffer module 1207. The decoded image buffer module 1207 can output the decoded image or sub-image according to timing and control information. The image stored in the decoded image buffer module 1207 can be used as a reference for the prediction module 1202 to perform inter prediction or intra prediction.

[0145] FIG. 13 is a flowchart of a video decoding method according to an embodiment of the present invention. Referring to FIGS. 11 to 13, the video decoder 1100 may implement the video decoding method by performing the following steps S1310 to S1340. In step S1310, the video decoder 1100 may receive an input bitstream. Furthermore, the prediction module 1202 may determine a prediction block for a current coding block (CB) according to an intra-prediction mode. The intra-prediction mode may be determined by a prediction mode flag decoded from the input bitstream. In step 1320, the entropy decoding module 1201 may decode the input bitstream to obtain an NSPT index having a value of 1, 2, or 3. Furthermore, the entropy decoding module 1201 may decode the input bitstream to obtain a plurality of quantized coefficients. In step S1330, the inverse quantization module 1203 may perform inverse quantization on the quantized coefficients to generate a plurality of reconstructed coefficients. In step S1340, if the current coding block has a specific block size and the NSPT index is 1, 2, or 3, the inverse transform module 1204 may perform an inverse non-separable linear transform on the multiple reconstructed coefficients based on a kernel of a specific dimensionality corresponding to the specific block size to generate a reconstructed residual block.

[0146] In one embodiment of the present invention, if the current CB has a specific block size of 8x16, 16x8, 16x4, 4x16, 8x4, or 4x8, an NSPT transform matrix is ​​selected from a kernel of dimensions AxBxCxD, where the value of parameter C is 3 and the value of parameter D is 35. The size of the current CB may be determined by a partitioning mode flag pre-decoded from the input bitstream. Note that in this disclosure, the dimension "A" represents the number of input samples from the residual block, the dimension "B" represents the number of transform coefficients generated by NSPT, the dimension "C" represents the number of indexed sets, and the dimension "D" represents the number of transform sets. The order of kernel dimensions described in this disclosure is merely provided for illustrative purposes to explain how the NSPT transform matrix is ​​selected and may differ in NSPT implementations.

[0147] An NSPT transform matrix is ​​selected by indexing the "C" and "D" dimensions of the kernel, resulting in an NSPT transform matrix of size A x B. The value of the NSPT index is used to index the "C" dimension of the kernel. As described above, a prediction block is generated by intra prediction, which is associated with an intra prediction mode. The intra prediction mode is mapped to a transform set index by the mapping process described above with reference to Table 2. The transform set index is used to index the "D" dimension of the kernel.

[0148] In one example of this embodiment of the present invention, inverse NSPT is performed on multiple reconstructed coefficients by matrix multiplication. The reconstructed coefficients are typically decoded in a hierarchical reverse diagonal scan order, meaning that coefficients corresponding to higher spatial frequencies are decoded before coefficients corresponding to lower spatial frequencies. Next, the reconstructed coefficients are arranged in reverse order (i.e., a hierarchical forward diagonal scan order) to form a one-dimensional column vector P of B × 1 samples. Let T be the selected NSPT transform matrix, with size A × B. Then, inverse NSPT is performed by performing matrix multiplication R = T on the reconstructed coefficients with the NSPT transform matrix, and the result R is a one-dimensional column vector of A × 1 reconstructed residual coefficients. Next, the column vector R can be arranged in a raster scan order into a two-dimensional reconstructed residual block of the same size as the current CB. Note that in the matrix multiplication, each column of T corresponds to a sample position in the column vector R, which corresponds to a spatial position within the reconstructed residual block according to the above-mentioned raster scan order. The above raster scan order for constructing the reconstructed residual block is only an example, and in different embodiments, different scan orders can be used, provided that the NSPT kernel is modified so that the columns of T correspond to the same spatial locations within the reconstructed residual block.

[0149] In one embodiment of the present invention, if the current CB has a block size of 16x8, then the NSPT transform matrix T is selected from a kernel of dimensions AxB1xCxD, where the value of parameter A is 128, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, the inverse NSPT is performed on a plurality of reconstructed coefficients by matrix multiplication R=TP, where the samples of R are rearranged to generate a two-dimensional reconstructed residual block.

[0150] In one embodiment of the present invention, if the current CB has a block size of 8x16, then the NSPT transform matrix T is selected from a kernel of dimensions AxB1xCxD, where the value of parameter A is 128, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, the inverse NSPT is performed on a plurality of reconstruction coefficients by matrix multiplication R=TP, where the samples of R are rearranged to generate a two-dimensional reconstructed residual block.

[0151] Specifically, NSPT replaces LFNST for block sizes of 8x16 and 16x8. For these block sizes, an additional NSPT kernel of dimensionality 128xB1xCxD is trained, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder's perspective. The additional NSPT kernel is the same as the additional NSPT kernel available to video encoder 100. In one embodiment of the present invention, parameter B1=32. However, in another embodiment of the present invention, parameter B1 is set to a value close to but not equal to 32, and the exact value may be selected after empirically verifying that the reference encoder's complexity does not increase for this value of parameter B1. In another configuration, B1 may be set to 36. However, in another embodiment of the present invention, parameter B1 is set to a value close to but not equal to 36, and the exact value may be selected after empirically verifying that the reference encoder's complexity does not increase for this value of parameter B1.

[0152] In one embodiment of the present invention, if the current CB has a block size of 16x4, then the NSPT transform matrix T is selected from a kernel of dimensionality AxB2xCxD, where the value of parameter A is 64, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, the inverse NSPT is performed on a plurality of reconstructed coefficients by matrix multiplication R=TP, where the samples of R are rearranged to generate a 2D reconstructed residual block.

[0153] In one embodiment of the present invention, if the current CB has a block size of 4x16, then the NSPT transform matrix T is selected from a kernel of dimensionality AxB2xCxD, where the value of parameter A is 64, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, the inverse NSPT is performed on a plurality of reconstructed coefficients by matrix multiplication R=TP, where the samples of R are rearranged to generate a 2D reconstructed residual block.

[0154] Specifically, NSPT may replace LFNST for block sizes of 4x16 and 16x4. For these block sizes, an additional NSPT kernel of dimensionality 64xB2xCxD is trained, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder's perspective. The additional NSPT kernel is the same as the additional NSPT kernel available to video encoder 100. In one embodiment of the present invention, parameter B2=20. However, in other embodiments of the present invention, parameter B2 is set to a value close to but not equal to 20, and the exact value may be selected after empirically verifying that the reference encoder's complexity does not increase for this value of parameter B2. In another configuration, B2 may be set to 16. However, in other embodiments of the present invention, parameter B2 is set to a value close to but not equal to 16, and the exact value may be selected after empirically verifying that the reference encoder's complexity does not increase for this value of parameter B2.

[0155] In one embodiment of the present invention, if the current CB has a block size of 8x4, then the NSPT transform matrix T is selected from a kernel of dimensions AxB3xCxD, where the value of parameter A is 32, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, the inverse NSPT is performed on a plurality of reconstructed coefficients by matrix multiplication R=TP, where the samples of R are rearranged to generate a 2D reconstructed residual block.

[0156] In one embodiment of the present invention, if the current CB has a block size of 4x8, then the NSPT transform matrix T is selected from a kernel of dimensionality AxB3xCxD, where the value of parameter A is 32, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, the inverse NSPT is performed on a plurality of reconstructed coefficients by matrix multiplication R=TP, where the samples of R are rearranged to generate a 2D reconstructed residual block.

[0157] Specifically, NSPT may replace LFNST for block sizes of 4x8 and 8x4. For these block sizes, an additional NSPT kernel of dimensionality 32xB3xCxD is trained, where parameter B3 is the number of transform coefficients generated by NSPT from the encoder's perspective. The additional NSPT kernel is the same as the additional NSPT kernel available to video encoder 100. In one embodiment of the present invention, parameter B3=16. However, in another embodiment of the present invention, parameter B3 is set to a value close to but not equal to 16, and the exact value may be selected after empirically verifying that the reference encoder does not increase its complexity for this value of parameter B3.

[0158] The reconstruction module 1205 may then generate a reconstructed CB by adding the reconstructed residual block and the prediction block sample-by-sample. The video decoder 1100 may combine the reconstructed CB with other reconstructed CBs to generate a reconstructed image or a reconstructed sub-image. After the filtering module 1206 performs in-loop filtering to generate a decoded image or a decoded sub-image, the video decoder 1100 may output the decoded image or the decoded sub-image to the output video.

[0159] As described above, the video encoding method, video decoding method, video encoder, and video decoder of the present invention propose several methods for low-complexity implementation of NSPT in video encoding and video decoding. The video encoding method, video decoding method, video encoder, and video decoder of the present invention can effectively improve coding performance for video coding by introducing a larger NSPT kernel with the same complexity as the corresponding LFNST kernel, a novel zeroing policy for the NSPT kernel, and / or extending the NSPT kernel to a larger block size through implicit partitioning. The video encoding method, video decoding method, video encoder, and video decoder of the present invention can be used for future video coding standards.

[0160] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of the disclosed embodiments provided they come within the scope of the following claims and their equivalents. [Explanation of symbols]

[0161] 100: Video Encoder 110,1110:Processor 120,1120: Storage device 130, 1130: Communication interface 140: Data bus 201: Split module 202: Calculation module 203: Prediction module 204: Conversion module 205: Quantization module 206: Entropy coding module 207: Inverse quantization module 208: Inverse quantization module 209: Reconstruction module 210: Filtering module 211: Decoded image buffer module 400: Input image 401:CTU 402:CU 1100: Video decoder 1201: Entropy decoding module 1202: Prediction module 1203: Inverse quantization module 1204: Inverse conversion module 1205: Reconstruction module 1206: Filtering module 1207: Decoded image buffer module S310~S350, S1310~S1340: Step

Claims

1. 1. A video decoder comprising: an entropy decoding module configured to receive an input bitstream and decode the input bitstream to obtain an NSPT index; an inverse quantization module coupled to the entropy decoding module and configured to perform inverse quantization to generate a plurality of reconstructed coefficients; an inverse transform module coupled to the inverse quantization module, the inverse transform module being configured to, when a current coding block (CB) has a specific block size and the value of the NSPT index is 1, 2, or 3, perform an inverse non-separable linear transform (NSPT) on the reconstructed coefficients based on a kernel having a specific dimension corresponding to the specific block size to generate a reconstructed residual block, wherein the specific block size includes an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size; Video decoder.

2. The video decoder further includes a prediction module, wherein the prediction module is configured to determine a predicted block of the current CB by intra prediction if the flag or the set of flags indicates application of intra prediction.

2. The video decoder of claim 1.

3. The inverse transform module performs the inverse NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 128×B1×C×D when the current CB has a block size of 8×16, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

2. The video decoder of claim 1.

4. The parameter B1 is 32 or 36; 4. A video decoder according to claim 3.

5. the parameter B1 is close to but not equal to 32 or 36; 4. A video decoder according to claim 3.

6. The inverse transform module is configured to perform the inverse NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 64×B2×C×D when the current CB has a block size of 16×4, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

2. The video decoder of claim 1.

7. The parameter B1 is 32 or 36; 7. A video decoder according to claim 6.

8. the parameter B1 is close to but not equal to 32 or 36; 7. A video decoder according to claim 6.

9. The inverse transform module is configured to perform the inverse NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 64×B2×C×D when the current CB has a block size of 16×4, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

2. The video decoder of claim 1.

10. The parameter B2 is 20 or 16; 10. A video decoder according to claim 9.

11. said parameter B2 is close to but not equal to 20 or 16; 10. A video decoder according to claim 9.

12. The inverse transform module is configured to perform the inverse NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 64×B2×C×D when the current CB has a block size of 4×16, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

2. The video decoder of claim 1.

13. The parameter B2 is 20 or 16; 13. A video decoder according to claim 12.

14. said parameter B2 is close to but not equal to 20 or 16; 13. A video decoder according to claim 12.

15. The inverse transform module is configured to perform the inverse NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 32×B3×C×D when the current CB has a block size of 8×4, where parameter B3 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

2. The video decoder of claim 1.

16. The parameter B3 is 16.

16. A video decoder according to claim 15.

17. The parameter B3 is close to but not equal to 16, 16. A video decoder according to claim 15.

18. The inverse transform module is configured to perform the inverse NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 32×B3×C×D when the current CB has a block size of 4×8, where parameter B3 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

2. The video decoder of claim 1.

19. The parameter B3 is 16.

19. A video decoder according to claim 18.

20. The parameter B3 is close to but not equal to 16, 19. A video decoder according to claim 18.

21. 1. A video decoding method comprising: receiving an input bitstream; entropy decoding the input bitstream to obtain an NSPT index; performing inverse quantization to generate a plurality of reconstructed coefficients; and if the current coding block (CB) has a specific block size and the value of the NSPT index is 1, 2, or 3, performing an inverse non-separable linear transform (NSPT) on the reconstructed coefficients based on a kernel having a specific dimension corresponding to the specific block size to generate a reconstructed residual block, wherein the specific block size includes an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size. Video decoding methods.

22. If the flag or set of flags indicates application of intra prediction, the predicted block of the current CB is determined by intra prediction.

22. The video decoding method of claim 21.

23. If the current CB has a block size of 8x16, the inverse NSPT is performed using an NSPT transform matrix selected from the kernel of the specific dimensions 128xB1xCxD, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

22. The video decoding method of claim 21.

24. The parameter B1 is 32 or 36; 24. The video decoding method of claim 23.

25. the parameter B1 is close to but not equal to 32 or 36; 24. The video decoding method of claim 23.

26. If the current CB has a block size of 16x8, the inverse NSPT is performed using an NSPT transform matrix selected from the kernel of the specific dimensions 128xB1xCxD, where parameter B1 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

22. The video decoding method of claim 21.

27. The parameter B1 is 32 or 36; 27. The video decoding method of claim 26.

28. the parameter B1 is close to but not equal to 32 or 36; 27. The video decoding method of claim 26.

29. If the current CB has a block size of 16x4, the inverse NSPT is performed using an NSPT transform matrix selected from the kernel of the specific dimensions 64xB2xCxD, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

22. The video decoding method of claim 21.

30. The parameter B2 is 20 or 16; 30. The video decoding method of claim 29.

31. said parameter B2 is close to but not equal to 20 or 16; 30. The video decoding method of claim 29.

32. If the current CB has a block size of 4x16, the inverse NSPT is performed using an NSPT transform matrix selected from the kernel of the specific dimensions 64xB2xCxD, where parameter B2 is the number of transform coefficients generated by NSPT from the encoder side, parameter C is 3, and parameter D is 35.

22. The video decoding method of claim 21.

33. The parameter B2 is 20 or 16; 33. The video decoding method of claim 32.

34. said parameter B2 is close to but not equal to 20 or 16; 33. The video decoding method of claim 32.

35. If the current CB has a block size of 8x4, the inverse NSPT is performed using an NSPT transform matrix selected from the kernel of the specific dimensions 32xB3xCxD, where parameter B3 is the number of transform coefficients generated by the NSPT on the encoder side, parameter C is 3, and parameter D is 35.

22. The video decoding method of claim 21.

36. The parameter B3 is 16.

36. The video decoding method of claim 35.

37. The parameter B3 is close to but not equal to 16, 36. The video decoding method of claim 35.

38. If the current CB has a block size of 4x8, the inverse NSPT is performed using an NSPT transform matrix selected from the kernel of the specific dimensions 32xB3xCxD, where parameter B3 is the number of transform coefficients generated by the NSPT on the encoder side, parameter C is 3, and parameter D is 35.

22. The video decoding method of claim 21.

39. The parameter B3 is 16.

39. The video decoding method of claim 38.

40. The parameter B3 is close to but not equal to 16, 39. The video decoding method of claim 38.

41. 1. A video encoder comprising: a calculation module configured to calculate a residual block based on the current CB and the predicted block; a transform module coupled to the arithmetic module, the transform module being configured to, when the residual block has a specific block size and the determined NSPT index value is 1, 2, or 3, perform a non-separable linear transform (NSPT) on the residual block based on a kernel having specific dimensions corresponding to the specific block size to generate a plurality of coefficients, the specific block size including an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size; a quantization module coupled to the transform module and configured to quantize the plurality of coefficients to generate a plurality of quantized coefficients; an entropy encoding module coupled to the quantization module, the entropy encoding module configured to encode the plurality of quantized coefficients and the NSPT index into an output bitstream. Video encoder.

42. The video encoder further includes a prediction module configured to generate the prediction block of the current CB by intra prediction.

42. The video encoder of claim 41.

43. The transform module is configured to perform the NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 128×B1×C×D when the residual block has a block size of 8×16, where parameter B1 is the number of transform coefficients generated by the NSPT from the encoder side, parameter C is 3, and parameter D is 35.

42. The video encoder of claim 41.

44. The parameter B1 is 32 or 36; 44. The video encoder of claim 43.

45. the parameter B1 is close to but not equal to 32 or 36; 44. The video encoder of claim 43.

46. The transform module performs the NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 128×B1×C×D when the residual block has a block size of 16×8, where parameter B1 is the number of transform coefficients generated by the NSPT from the encoder side, parameter C is 3, and parameter D is 35.

42. The video encoder of claim 41.

47. The parameter B1 is 32 or 36; 47. A video encoder according to claim 46.

48. the parameter B1 is close to but not equal to 32 or 36; 47. A video encoder according to claim 46.

49. The transform module is configured to perform the NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 64×B2×C×D when the residual block has a block size of 16×4, where parameter B2 is the number of transform coefficients generated by the NSPT from the encoder side, parameter C is 3, and parameter D is 35.

42. The video encoder of claim 41.

50. The parameter B2 is 20 or 16; 50. The video encoder of claim 49.

51. said parameter B2 is close to but not equal to 20 or 16; 50. The video encoder of claim 49.

52. The inverse transform module is configured to perform the NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 64×B2×C×D when the residual block has a block size of 4×16, where parameter B2 is the number of transform coefficients generated by the NSPT from the encoder side, parameter C is 3, and parameter D is 35.

42. The video encoder of claim 41.

53. The parameter B2 is 20 or 16; 53. The video encoder of claim 52.

54. said parameter B2 is close to but not equal to 20 or 16; 53. The video encoder of claim 52.

55. The transform module is configured to perform the NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 32×B3×C×D when the residual block has a block size of 8×4, where parameter B3 is the number of transform coefficients generated by the NSPT from the encoder side, parameter C is 3, and parameter D is 35.

42. The video encoder of claim 41.

56. The parameter B3 is 16.

56. A video encoder as recited in claim 55.

57. The parameter B3 is close to but not equal to 16, 56. A video encoder as recited in claim 55.

58. The transform module is configured to perform the NSPT using an NSPT transform matrix selected from the kernel of the specific dimensions 32×B3×C×D when the residual block has a block size of 8×4, where parameter B3 is the number of transform coefficients generated by the NSPT from the encoder side, parameter C is 3, and parameter D is 35.

42. The video encoder of claim 41.

59. The parameter B3 is 16.

59. The video encoder of claim 58.

60. The parameter B3 is close to but not equal to 16, 59. The video encoder of claim 58.

61. performing zeroing on the residual block, the residual block after zeroing having a zeroed region; 42. The video encoder of claim 41.

62. the transform module is further configured to divide the residual block into multiple smaller transform blocks if the size of the residual block is larger than a block size supported by a largest NSPT kernel.

42. The video encoder of claim 41.

63. an NSPT index is signaled for each of the plurality of small transform blocks; 63. The video encoder of claim 62.

64. an NSPT index is signaled for each unique transform block size, and the multiple small transform blocks having the same block size share a common NSPT index; 63. The video encoder of claim 62.

65. 1. A video encoding method comprising: calculating a residual block based on the current CB and the predicted block; if the residual block has a specific block size and the determined NSPT index value is 1, 2, or 3, performing a non-separable linear transform (NSPT) on the residual block based on a kernel having specific dimensions corresponding to the specific block size to generate a plurality of coefficients, wherein the specific block size includes an 8x16 block size, a 16x8 block size, a 16x4 block size, a 4x16 block size, an 8x4 block size, or a 4x8 block size; quantizing the plurality of coefficients to generate a plurality of quantized coefficients; entropy encoding the plurality of quantized coefficients into an output bitstream; encoding the NSPT index into the output bitstream. Video coding methods.

Images