Video encoder, video encoding method, video decoder, and video decoding method

Abstract

The invention provides a video encoding method, a video decoding method, a video encoder and a video decoder. The video decoder comprises an entropy decoding module, a prediction module, an inverse quantization module, an inverse transformation module and a reconstruction module. The entropy decoding module receives an input code stream. The prediction module determines a prediction block. The entropy decoding module decodes an input code stream to obtain a plurality of quantization coefficients and an NSPT index. The inverse quantization module inversely quantizes the quantized coefficients to generate a plurality of reconstructed coefficients. In response to the current coded block (CB) having an 8 * 16, 16 * 8, or 16 * 16 block size, the inverse transform module performs an inverse non-separable main transform (NSPT) on the reconstruction coefficients to generate a reconstructed residual block based on the kernel. A reconstruction module generates a reconstructed CB from the reconstructed residual block and the prediction block.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 387,497, filed on December 14, 2022. The entire contents of the above patent application are hereby incorporated by reference herein and constitute a part of this specification. Technical Field

[0003] The present invention relates to the field of image data processing, and in particular to a video encoding method, a video decoding method, a video encoder and a video decoder. Background Art

[0004] In traditional image coding techniques, efficient data prediction and compression have always been a key issue. In particular, new transform methods (i.e., the non-separable primary transform) can efficiently compress data for small coding blocks, but are computationally expensive when applied to large coding blocks. Summary of the Invention

[0005] Technical issues

[0006] New image processing methods are needed for efficiently encoding video data and efficiently decoding the corresponding bitstreams.

[0007] Problem Solution

[0008] The video encoder of the present invention includes a partition module, a prediction module, an arithmetic module, a transform module, a quantization module, and an entropy coding module. The partition module is configured to receive an input video and generate a plurality of coding blocks (CBs) of the input video. The prediction module is coupled to the partition module and configured to generate a prediction block for a current CB. The arithmetic module is coupled to the partition module and the prediction module and configured to calculate a residual block based on the current CB and the prediction block. The transform module is coupled to the arithmetic module and, in response to the residual block having an 8×16 block size, a 16×8 block size, or a 16×16 block size, and a determined NSPT index having a value of 1, 2, or 3, the transform module is configured to perform a non-separable primary transform (NSPT) on the residual block to generate a plurality of coefficients based on a kernel. The quantization module is coupled to the transform module and configured to quantize the plurality of coefficients to generate a plurality of quantized coefficients. The entropy coding module is coupled to the quantization module and configured to encode the plurality of quantized coefficients into an output bitstream. The entropy coding module is configured to encode the NSPT index into the output bitstream.

[0009] In one embodiment of the present invention, the partitioning module is used to divide the input picture of the input video into multiple coding tree units (CTUs), and divide each CTU into one or more coding units (CUs) to generate multiple CUs and equivalently generate the multiple coding blocks (CBs).

[0010] In one embodiment of the present invention, the prediction module is configured to generate a prediction block of the current CB through intra-frame prediction.

[0011] In one embodiment of the present invention, in response to the residual block having an 8×16 block size, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0012] In one embodiment of the present invention, in response to the residual block having a 16×8 block size, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0013] In one embodiment of the present invention, in response to the residual block having a 16×16 block size, the transform module performs NSPT using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D. Parameter B2 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0014] In one embodiment of the present invention, parameter B1 and parameter B2 are 32.

[0015] In one embodiment of the present invention, parameter B1 and parameter B2 are close to but not equal to 32.

[0016] In one embodiment of the present invention, the prediction module further performs zeroing on the residual block, so that the residual block after zeroing has a zeroing area.

[0017] In one embodiment of the present invention, in response to the size of the residual block being larger than the block size supported by the maximum NSPT kernel, the transform module is further configured to divide the residual block into a plurality of smaller transform blocks.

[0018] In one embodiment of the present invention, for each transform block in the plurality of smaller transform blocks, an NSPT index is indicated by signaling.

[0019] In one embodiment of the present invention, for each unique transform block size, an NSPT index is indicated by signaling, and transform blocks with the same block size in the plurality of smaller transform blocks share a common NSPT index.

[0020] In one embodiment of the present invention, in response to the size of the residual block being larger than the block size supported by the maximum NSPT kernel, the current CB is implicitly split into multiple smaller prediction blocks before prediction.

[0021] The video encoding method of the present invention includes the following steps: receiving an input video and generating multiple coding blocks (CBs) of the input video; generating a prediction block of a current CB; calculating a residual block based on the current CB and the prediction block; in response to the residual block having an 8×16 block size, a 16×8 block size or a 16×16 block size, and a determined NSPT index having a value of 1, 2 or 3, performing a non-separable primary transform (NSPT) on the residual block to generate multiple coefficients based on a kernel; quantizing the multiple coefficients to generate multiple quantized coefficients; encoding the multiple quantized coefficients to an output code stream by entropy coding; and encoding the NSPT index to the output code stream.

[0022] In one embodiment of the present invention, the video encoding method further includes the following steps: dividing the input picture of the input video into multiple coding tree units (CTUs), and dividing each CTU into one or more CUs to generate multiple CUs and equivalently generate multiple coding blocks (CBs).

[0023] In one embodiment of the present invention, the prediction block of the current CB is generated by intra-frame prediction.

[0024] In one embodiment of the present invention, in response to the residual block having an 8×16 block size, the video encoding method performs NSPT using an NSPT transform matrix selected from a kernel having dimensions of 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0025] In one embodiment of the present invention, in response to the residual block having a 16×8 block size, the video encoding method performs NSPT using an NSPT transform matrix selected from a kernel having dimensions of 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0026] In one embodiment of the present invention, in response to the residual block having a 16×16 block size, a video encoding method performs NSPT using an NSPT transform matrix selected from a kernel having dimensions of 256×B2×C×D. Parameter B2 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0027] In one embodiment of the present invention, parameter B1 and parameter B2 are 32.

[0028] In one embodiment of the present invention, parameter B1 and parameter B2 are close to but not equal to 32.

[0029] In one embodiment of the present invention, the video encoding method further includes: performing zeroing on the residual block, so that the residual block after zeroing has a zeroing area.

[0030] In one embodiment of the present invention, the video encoding method further comprises the following step: in response to the size of the residual block being larger than the block size supported by the maximum NSPT kernel, dividing the residual block into a plurality of smaller transform blocks.

[0031] In one embodiment of the present invention, for each transform block in the plurality of smaller transform blocks, an NSPT index is indicated by signaling.

[0032] In one embodiment of the present invention, for each unique transform block size, an NSPT index is indicated by signaling, and transform blocks with the same block size in the plurality of smaller transform blocks share a common NSPT index.

[0033] In one embodiment of the present invention, the video encoding method further comprises the following steps: in response to the size of the residual block being larger than the block size supported by the maximum NSPT kernel, implicitly splitting the current CB into multiple smaller prediction blocks before prediction.

[0034] The video decoder of the present invention includes an entropy decoding module, a prediction module, an inverse quantization module, an inverse transform module, and a reconstruction module. The entropy decoding module is configured to receive an input bitstream. The prediction module is coupled to the entropy decoding module and configured to determine a prediction block. The entropy decoding module is configured to decode the input bitstream to obtain a plurality of quantized coefficients and an NSPT index. The inverse quantization module is coupled to the entropy decoding module and configured to inverse quantize the plurality of quantized coefficients to generate a plurality of reconstructed coefficients. The inverse transform module is coupled to the inverse quantization module and, in response to a current coding block (CB) having an 8×16 block size, a 16×8 block size, or a 16×16 block size, and the NSPT index having a value of 1, 2, or 3, the inverse transform module is configured to perform an inverse non-separable primary transform (NSPT) on the plurality of reconstructed coefficients to generate a reconstructed residual block based on a kernel. The reconstruction module is coupled to the prediction module and the inverse transform module and configured to generate a reconstructed CB based on the residual block and the prediction block.

[0035] In one embodiment of the present invention, the reconstruction module is configured to combine the reconstructed CB with a plurality of reconstructed CBs to generate the output video.

[0036] In one embodiment of the present invention, in response to a flag or a flag set indicating application of intra-frame prediction, the prediction module is configured to determine a prediction block of the current CB by intra-frame prediction.

[0037] In one embodiment of the present invention, in response to the current CB having an 8×16 block size, the inverse transform module performs inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0038] In one embodiment of the present invention, in response to the current CB having a 16×8 block size, the inverse transform module performs inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0039] In one embodiment of the present invention, in response to the current CB having a 16×16 block size, the inverse transform module performs inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D. Parameter B2 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0040] In one embodiment of the present invention, parameter B1 and parameter B2 are 32.

[0041] In one embodiment of the present invention, parameter B1 and parameter B2 are close to but not equal to 32.

[0042] The video decoding method of the present invention includes: receiving an input code stream; determining a prediction block based on the input code stream; performing entropy decoding on the input code stream to obtain multiple quantization coefficients and NSPT indexes; inverse quantizing the multiple quantization coefficients to generate multiple reconstruction coefficients; in response to a current coding block (CB) having an 8×16 block size, a 16×8 block size or a 16×16 block size, and the NSPT index having a value of 1, 2 or 3, performing an inverse non-separable primary transform (NSPT) on the multiple reconstruction coefficients to generate a reconstructed residual block based on a kernel; and generating a reconstructed CB based on the residual block and the prediction block.

[0043] In one embodiment of the present invention, the video decoding method further comprises the following step: combining the reconstructed CB with a plurality of reconstructed CBs to generate the output video.

[0044] In one embodiment of the present invention, in response to a flag or a flag set indicating application of intra prediction, the prediction block of the current CB is determined by intra prediction.

[0045] In one embodiment of the present invention, in response to the current CB having an 8×16 block size, the video decoding method performs inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0046] In one embodiment of the present invention, in response to the current CB having a 16×8 block size, the video decoding method performs inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D. Parameter B1 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0047] In one embodiment of the present invention, in response to the current CB having a 16×16 block size, the video decoding method performs inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D. Parameter B2 is the number of transform coefficients generated by the NSPT from the encoder's perspective, parameter C is 3, and parameter D is 35.

[0048] In one embodiment of the present invention, parameter B1 and parameter B2 are 32.

[0049] In one embodiment of the present invention, parameter B1 and parameter B2 are close to but not equal to 32.

[0050] Effects of the Invention

[0051] Based on the above, the video encoding method, video decoding method, video encoder and video decoder of the present invention can effectively improve the encoding performance of video encoding.

[0052] In order to make the above content easier to understand, several embodiments will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a video encoder provided by an embodiment of the present invention.

[0054] Figure 2 A schematic diagram of a video encoding process provided by an embodiment of the present invention.

[0055] Figure 3 This is a flowchart of a video encoding method provided by an embodiment of the present invention.

[0056] Figure 4 FIG. 4 is a schematic diagram of a picture divided into multiple blocks according to an embodiment of the present invention.

[0057] Figure 5 FIG. 4 is a schematic diagram of a CTU divided into multiple CUs according to an embodiment of the present invention.

[0058] Figure 6A Schematic diagram of LFNST for 4×N and N×4 block sizes according to an embodiment of the present invention.

[0059] Figure 6B FIG. 1 is a schematic diagram of LFNST for large block size according to an embodiment of the present invention.

[0060] Figure 7A Schematic diagram of LFNST for 4×N and N×4 block sizes according to an embodiment of the present invention.

[0061] Figure 7B Schematic diagram of LFNST for 8×N and N×8 block sizes according to an embodiment of the present invention.

[0062] Figure 7C Schematic diagram of LFNST for 16×N and N×16 block sizes according to an embodiment of the present invention.

[0063] Figure 8 FIG. 4 is a schematic diagram of a zero-out region according to an embodiment of the present invention.

[0064] Figure 9A FIG. 4 is a schematic diagram of segmenting a residual block according to an embodiment of the present invention.

[0065] Figure 9B FIG. 4 is a schematic diagram of segmenting a residual block according to another embodiment of the present invention.

[0066] Figure 10 FIG. 4 is a schematic diagram of local prediction according to another embodiment of the present invention.

[0067] Figure 11 A schematic diagram of a video decoder provided by an embodiment of the present invention.

[0068] Figure 12 A schematic diagram of a video decoding process provided by an embodiment of the present invention.

[0069] Figure 13 This is a flowchart of a video decoding method provided by an embodiment of the present invention. Specific embodiments

[0070] In order to understand the features and technical content of the embodiments of the present application in more detail, the implementation of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and explanation purposes only and are not intended to limit the embodiments of the present application.

[0071] Modern international video coding standards typically describe a hybrid block-based approach for decoding bitstreams. For example, the High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) standards are block-based hybrid spatial and temporal prediction coding schemes. That is, to generate such a bitstream when encoding each picture, the picture is divided into multiple blocks, and then each of these blocks is encoded.

[0072] Figure 1 Schematic diagram of a video encoder provided by an embodiment of the present invention. Figure 1 The video encoder 100 includes a processor 110, a storage device 120, a communication interface 130, and a data bus 140. The processor 110 is electrically connected to the storage device 120 and the communication interface 130 via the data bus 140. In an embodiment of the present invention, the storage device 120 may store relevant instructions and may also store relevant video encoder algorithms. The processor 110 may output a bitstream to the communication interface 130. The processor 110 may execute relevant instructions to implement the video encoding method of the present invention.

[0073] In one embodiment of the present invention, the video encoder 100 can be implemented by one or more personal computers (PCs), one or more server computers, and one or more workstation computers, or can be composed of multiple computing devices, but the present invention is not limited thereto. In another embodiment of the present invention, the video encoder 100 can include more processors to execute relevant video encoders and / or relevant instructions to implement the video encoding method of the present invention. In addition, in one embodiment of the present invention, the video encoder 100 can include more processors to execute relevant video encoders, relevant video decoders, and / or relevant instructions to implement the video encoding method of the present invention. The video encoder 100 can be used to implement a video codec and can perform the video encoding function and the video decoding function of the present invention.

[0074] In one embodiment of the present invention, the processor 110 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), or other programmable general-purpose or special-purpose microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), other similar processing circuits, or a combination of these devices. In one embodiment of the present invention, the storage device 120 may be a non-transitory computer-readable recording medium, such as a read-only memory (ROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a non-volatile memory (NVM), but the present invention is not limited thereto.

[0075] In one embodiment of the present invention, the relevant video encoder and / or related instructions may also be stored in a non-transitory computer-readable recording medium of one device and executed by a processor of another device. For example, the communication interface 130 may be a network card supporting a wired network connection such as Ethernet, a wireless network card supporting wireless communication standards such as Institute of Electrical and Electronics Engineers (IEEE) 802.11n / b / g / ac / ax / be, or any other network connection device, but the embodiment is not limited thereto. The communication interface 130 is configured to obtain input video.

[0076] Figure 2 FIG is a schematic diagram of a video encoding process according to an embodiment of the present invention. Figure 1 and Figure 2 , the video encoder 100 may perform Figure 2 The video encoding process encodes the input video into an output bitstream. The storage device 120 may store algorithms for the partitioning module 201, the arithmetic module 202 (e.g., an addition module or a subtraction module), the prediction module 203, the transform module 204, the quantization module 205, the entropy coding module 206, the inverse quantization module 207, the inverse transform module 208, the reconstruction module 209 (e.g., an addition module or a subtraction module), the filtering module 210, and the decoded picture buffer module 211. The processor 110 may execute the above modules to perform the video encoding process.

[0077] In one embodiment of the present invention, the processor 110 may receive an input video from an external video source. The partitioning module 201 may receive the input video and partition each picture of the input video into a plurality of coding tree units (CTUs), and partition each CTU into one or more CUs to generate a plurality of coding units (CUs). Each CU includes one or more spatially co-located coding blocks (CBs), each of which corresponds to a color component of the video. Therefore, the partitioning module 201 is equivalent to generating a plurality of coding blocks (CBs). The prediction module 203 may receive the current CB and perform intra-frame prediction to generate a prediction block of the current CB, but the present invention is not limited thereto. The prediction module 203 may also perform inter-frame prediction, motion prediction and / or other predictions to generate a prediction block of the current CB. The arithmetic module 202 may receive the current CB and the prediction block and perform a subtraction operation on the current CB and the prediction block to generate a residual block.

[0078] The transform module 204 may perform a non-separable primary transform (NSPT) on the residual block based on a (NSPT) kernel, thereby transforming the data of the residual block to generate a plurality of coefficients. Alternatively, the transform module 204 may perform other transforms, such as a Karhunen-Loeve transform (KLT), a two-dimensional discrete cosine transform (DCT), and / or a low frequency non-separable secondary transform (LFNST). The quantization module 205 may also quantize the plurality of coefficients to generate a plurality of quantized coefficients. The entropy coding module 206 may then encode the plurality of quantized coefficients to generate an output bitstream. The entropy coding module 206 may first binarize the plurality of quantized coefficients into a series of binary symbols (binary bins), and then apply an entropy coding algorithm to compress the binary symbols into coded bits. Examples of binarization methods include, but are not limited to, truncated unary codes, combined truncated Rice (TR) codes with limited k-order exponential-Golomb (EGK) binarization, and k-order exponential-Golomb binarization. Examples of entropy coding algorithms include, but are not limited to, variable length coding (VLC), context adaptive VLC (CAVLC), arithmetic coding, context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), and probability interval partitioning entropy (PIPE). The entropy coding module 206 may also encode other parameters required for decoding the picture from the video encoder 100 (e.g., partition mode flag, prediction mode flag, coding block flag, sub-block coding flag, etc.) into the output bitstream. The video encoder 100 may then output the output bitstream.

[0079] The inverse quantization module 207 can perform a scaling operation on multiple quantized coefficients to output multiple reconstruction coefficients. The inverse transform module 208 can perform one or more inverse transforms corresponding to the transform in the transform module 204 and output a reconstructed residual block. The reconstruction module 209 can calculate a reconstructed CB by adding the reconstructed residual block to the prediction block of the current CB generated by the prediction module 203. The reconstruction module 209 can send the reconstructed CB to the prediction module 203 for use as an intra-frame prediction reference. After all CBs in the current picture or current sub-picture are reconstructed, the reconstruction module 209 can generate a reconstructed picture or reconstructed sub-picture by merging the reconstructed CBs. The filtering module 210 can perform loop filtering on the reconstructed picture or reconstructed sub-picture. The filtering module 210 may include one or more in-loop filtering operations, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, a luma mapping with chroma scaling (LMCS) filter, and a neural network-based loop filter (NNLF). The output of the filtering module 210 is a decoded picture or a decoded sub-picture, and these decoded pictures or decoded sub-pictures can be cached in the decoded picture cache module 211. The decoded picture cache module 211 can output the decoded picture or decoded sub-picture according to the timing and control information. In addition, the pictures stored in the decoded picture cache module 211 can be used as a reference for the prediction module 203 to perform inter-frame prediction or intra-frame prediction.

[0080] Figure 3 Flowchart of the video encoding method provided by the embodiment of the present invention. Figures 1 to 3 The video encoder 100 may perform the following steps S310 to S370 to implement the video encoding method. In step S310, the division module 201 may receive a current picture of an input video and divide the current picture into a plurality of CBs.

[0081] Figure 4 FIG is a schematic diagram of a picture divided into multiple blocks according to an embodiment of the present invention. Figure 4 As shown, the current picture 400 may first be divided into square blocks, referred to as CTUs 401. For example, CTUs 401 may be blocks of 256×256 pixels. Figure 5 , Figure 54 is a schematic diagram of a current CTU divided into multiple CUs according to an embodiment of the present invention. In an embodiment of the present invention, each CTU 401 in a picture can be further divided into one or more CUs 402. Each CU 402 can be rectangular or square, and the size of each CU 402 can be the same as its root CTU 401, or can be divided from the root CTU 401 into subdivisions as small as 4×4 blocks, such as Figure 5 Each CU 402 includes one or more spatially co-located coding blocks (CBs), where each coding block corresponds to a color component of the video.

[0082] In step S320, the prediction module 203 may generate a prediction block for the current CB. In an embodiment of the present invention, the prediction module 203 generates the prediction block for the current CB through intra-frame prediction. In step S330, the arithmetic module 202 may calculate a residual block based on the current CB and the prediction block. After prediction, the residuals may still be highly spatially correlated. Although conditional entropy coding can capture some spatial dependencies between adjacent samples, it is computationally unrealistic to form an entropy coding statistical model that can fully utilize the spatial correlation in the residuals. In contrast, transform coding is a practical and effective method for spatially decorrelating the residuals.

[0083] It should be noted that in general, the residual can be transformed by an integer version of the DCT, which can be applied separately in the horizontal and vertical directions. For an M×N block of residual samples (M is the width of the block and N is the height of the block), the transform coefficients can be obtained by applying an M×M DCT to each row to obtain intermediate transform coefficients, and then applying an N×N DCT to each column of the intermediate transform coefficients. The benefit of applying the transform can be measured by the transform coding gain (G TC ) is used to estimate the transform coding gain, which is defined as the distortion (D SQ ) and the transform coefficient x are scalar quantities quantized at the same bit rate (D TC Assuming that the residual is a wide-sense stationary Gaussian white noise source and the transform is orthogonal, the transform coding gain can be further interpreted as the transform coefficient variance The arithmetic mean of The ratio of the geometric means of .

[0084]

[0085] Through this second interpretation, transform coding gain and corresponding overall coding gain of the video coding system can be achieved when the resulting transform coefficients exhibit energy concentration characteristics, that is, the variance distribution is concentrated in a few transform coefficients compared to the original residual samples which are likely to be uniformly distributed.

[0086] A one-dimensional transform applied separately in the horizontal and vertical directions is used, which has good computational scalability as the block size increases. In the above example, the transform coefficients are obtained by a matrix implementation of the DCT, which results in (M+N) multiplications per sample. An even lower number of multiplications per sample can be achieved by a "butterfly" decomposition, but at the expense of a slightly higher computational latency. Separable transforms can best achieve energy concentration of spatial features along the Cartesian direction (i.e., vertical or horizontal). For example, vertical edges are perfectly compressed by the vertical DCT. However, separable transforms cannot optimally exploit spatial features oriented along non-Cartesian directions. In such cases, a well-designed non-separable transform can achieve higher coding performance.

[0087] A separable transform applies a one-dimensional transform in the horizontal and vertical directions separately, while a two-dimensional non-separable transform is applied directly to the input sample block. A desirable property of a transform is that the transform vector spans the space of input samples. This means that any input vector (i.e., any combination of the values ​​of the input samples) can be represented by a weighted sum of the transform vectors. A necessary condition for a transform to be a spanning transform is that there must be at least as many transform vectors as the number of dimensions of the input space, or in other words, the number of transform coefficients output is at least equal to the number of input samples. For example, the one-dimensional DCT in VVC is a spanning transform. Then, for a spanning non-separable transform, if the input sample block is an M×N residual, the transform will also output an M×N block of transform coefficients, which can be achieved through a matrix implementation of (M×N)×(M×N) multiplications.

[0088] In order to obtain a non-separable transform that produces coding gain for a specific directional feature, the transform can be learned. For example, a representative set of residual blocks corresponding to the directional feature of interest can be grouped, and then the KLT can be calculated based on the covariance matrix of this set of residual blocks. This process can be repeated on K different sets of residual blocks. Then, in this example, an overall transform kernel of dimension (M×N)×(M×N)×K is obtained.

[0089] As described in this section, there are two problems with spanning inseparable transforms. First, the computational complexity is high. Since inseparable transforms are usually learned, they are usually not factorizable. In the above example, the matrix implementation of spanning inseparable transforms results in a complexity of (M×N) multiplications per sample. The second problem is that the transform kernel takes up a lot of storage space in the encoder and decoder. In the above example, a single kernel suitable for K different directional features has (M×N)×(M×N)×K weights. This kernel can only be applied to residual blocks of size M×N. In order to allow the inseparable transform to be applied to multiple block sizes, the transform kernel must be learned for each discrete block size.

[0090] Therefore, in VVC, a low frequency non-separable secondary transform (LFNST) tool was introduced with several modifications to address the issues described above for spanning inseparable transforms. First, although the LFNST tool is applicable to various block sizes, only two LFNST kernels are limited. For blocks of size 4×N or N×4 (where N≥4), a smaller LFNST kernel is applied. For all larger block sizes (i.e., 8×8 or larger), a larger LFNST kernel is applied. Figure 6A and Figure 6B , and the sample positions where LFNST is applied are also shown. For example, from the encoder’s perspective, for a block size of 4×N or N×4, the top left 4×4 sample position (denoted by Figure 6A The remaining sample positions (indicated by the shaded area in Figure 6A ) are ignored, or “zeroed out”. From the decoder’s perspective, the inverse LFNST is applied to produce the top-left 4×4 samples, while the remaining samples are padded with zeros. A similar strategy is used for larger block sizes, where the LFNST is applied to the top-left 3 4×4 blocks of the sample position (denoted by Figure 6B The remaining sample positions are set to zero.

[0091] By employing a "zeroing" strategy, LFNST significantly reduces the size of the transform compared to a full-size transform applied to all sample positions. However, it is inherently lossy and cannot recover the values ​​at sample positions ignored by LFNST. If the LFNST tool were applied directly to the residual samples, the loss would be too great, rendering the LFNST tool unusable. However, LFNST is called a secondary transform because it is applied after the separable DCT at the encoder has been performed, acting on the primary transform coefficients to produce secondary transform coefficients. In other words, the DCT can be considered the primary transform. In this disclosure, the leftmost sample position in the primary transform coefficient block corresponds to the horizontal low-frequency DCT, while the topmost sample position corresponds to the vertical low-frequency DCT. By prioritizing the transform and reconstructing the top-left sample position at the decoder, LFNST is able to reconstruct low-frequency information from the original residual. As previously mentioned, the transform yields coding gain due to its energy-concentrating nature, and it has been well demonstrated in practice that the variance (energy) of camera-captured image and video signals is primarily concentrated in the low-frequency DCT coefficients. Therefore, although “zeroing” prevents LFNST from losslessly reconstructing arbitrary residual blocks, in practice the loss can be minimal for most classes of image and video signals.

[0092] The second modification is that for both the small LFNST kernel and the large LFNST kernel, the applied transform is not a span transform. From the encoder's perspective, the number of output (secondary transform) coefficients is less than the number of input (main transform) coefficients. For example, the smaller LFNST kernel takes 4×4=16 main transform coefficients as input, but only produces 8 output secondary transform coefficients. The larger LFNST kernel takes 3×4×4=48 main transform coefficients as input and outputs 8 secondary transform coefficients. The use of non-span transforms produces further reconstruction losses. However, a trade-off can be made between this loss and the complexity reduction achieved in a controlled manner. A spanned non-separable transform can first be designed using the above-mentioned KLT method. By following this method, the basis vectors of the transform correspond to the eigenvectors of the covariance matrix calculated based on the representative set of the residual block. These eigenvectors can be sorted according to the importance of their corresponding eigenvalues, where the most important eigenvectors are selected to construct a non-spanned non-separable transform. For example, the 8 eigenvectors with the largest eigenvalues ​​can be selected to form a non-spanned transform for the smaller LFNST kernel.

[0093] In summary, the above two modifications significantly reduce the complexity of the LFNST kernel compared to spanning non-separable transforms. For smaller blocks, using a smaller LFNST kernel reduces the potential complexity from (4×N)×(4×N) multiplications per transform block to 16×8 multiplications (for N≥4). For larger blocks, using a larger LFNST kernel reduces the potential complexity from (8×N)×(8×N) multiplications per transform block to 48x8 multiplications (for N≥8).

[0094] The LFNST kernel does not consist of just one transform matrix. To achieve good coding gain across a variety of image and video signals, multiple transform matrices are learned. The number of different transform matrices is the product of the third and fourth dimensions of the LFNST kernel. The dimensions of the smaller LFNST kernel are 16×8×2×4, while the larger LFNST kernel has dimensions of 48×8×2×4. The LFNST kernel is represented using two additional dimensions because the specific transform matrix for a transform block is chosen through a mixture of explicit signaling and implicit selection.

[0095] Explicit signaling is performed via an LFNST index transmitted in the bitstream. The LFNST index can take the values ​​0, 1, or 2, where 0 indicates that LFNST is not used for the transform block, while values ​​1 or 2 indicate the selection of the third dimension of the LFNST kernel. This explicit signaling mechanism overcomes the potential reconstruction loss caused by zeroing and non-spanning simplification. In cases where using LFNST results in excessive reconstruction loss for the transform block, the LFNST tool can be disabled by signaling an LFNST index of 0.

[0096] Implicit selection of LFNST is enabled by limiting it to coding blocks using intra prediction. Intra prediction generates a prediction block for a coding block based on adjacent neighboring reference samples to the top and left of the current block. The specific method for constructing the prediction block is signaled in the bitstream via the intra prediction mode. Simple methods for intra prediction include taking the average of the reference samples ("DC" mode) or performing affine interpolation between a number of reference samples ("planar" mode). However, most intra prediction modes are reserved for signaling intra angular direction, where the prediction block is constructed by assuming that reference sample values ​​repeat along a specific direction. When intra angular direction is used, this can be a strong hint about the directional nature of the residual block. Implicit selection of the LFNST transform is performed by mapping the intra prediction mode to one of four possible values ​​of a "transform set index," which is used to index into the fourth dimension of the LFNST kernel. The mapping used in VVC is shown in Table 1 below (i.e., mapping from intra prediction mode to LFNST transform set index).

[0097]

[0098]

[0099] Table 1

[0100] In addition, in the post-VVC exploratory activities, an extension to LFNST was proposed and integrated into the enhanced compression test model (ECM). The LFNST tool in ECM reduces the complexity reduction requirements of the original LFNST tool used in VVC to achieve the improvement in coding gain. ECM has three LFNST cores. Similar to the LFNST tool in VVC, in most cases, a significant portion of the transform block is set to zero. In addition, refer to 7A to 7C The shaded area indicates the primary transform coefficient positions in the ECM where LFNST is applied, while the white area indicates which transform coefficient positions are set to zero. For blocks of size 4×N or N×4 (where N≥4), a small LFNST kernel is used on the top-left 4×4 primary transform coefficients. For blocks of size 8×N or N×8 (where N≥8), a medium LFNST kernel is used on the four top-left 4×4 primary transform coefficient blocks. For blocks of 16×16 or larger, a large LFNST kernel is used on the six top-left 4×4 primary transform coefficient blocks.

[0101] The size of the LFNST kernel in ECM can be 16×16×3×35 for a small LFNST kernel, 64×32×3×35 for a medium LFNST kernel, and 96×32×3×35 for a large LFNST kernel. Compared to the LFNST tool in VVC, the range of the LFNST index indicated by signaling is increased from 2 to 3, and the number of LFNST transform sets is increased from 4 to 35. The mapping from intra prediction mode to LFNST transform set index is shown in Table 2 below (i.e., mapping from intra prediction mode to LFNST transform set index in ECM).

[0102] predModeIntra lfnstTrSetIdx predModeIntra<0 3 0<=predModeIntra<=34 predModeIntra 35<=predModeIntra<=66 68-predModeIntra 67<=predModeIntra<=80 2

[0103] Table 2

[0104] The complexity overhead of the LFNST tool can be evaluated in three ways. First, the additional memory load imposed on the decoder, which must store the LFNST kernel. Second, the worst-case number of multiplications per sample that the decoder must perform if the LFNST tool is applied. Third, the additional multiplications per sample that the encoder would require if a full search on the LFNST tool were performed. Based on these three parameters, the extended LFNST proposed in ECM is more complex than VVC's LFNST. However, in terms of the total number of multiplications per sample, the worst-case decoder complexity is still likely to be lower than the worst-case decoder complexity of other transforms.

[0105] For a matrix multiplication implementation of the DCT applied to each M×N transform, the number of multiplications per sample is (M+N). Therefore, the worst-case complexity occurs for the maximum value of (M+N). In practice, the complexity can be reduced by alternative implementations of the DCT, such as butterfly factorization, but it is still convenient to evaluate the complexity of the matrix multiplication implementation. In ECM, the separable DCT has been extended so that the largest transform is a 128-point DCT. The worst-case complexity of the separable DCT is then 128+128=256 multiplications per sample.

[0106] The worst-case decoder complexity of LFNST in ECM can be evaluated by considering several different block sizes. For a fair comparison, the evaluation includes the cost of performing the main transform. For a 4×4 block, the main transform involves 4+4=8 multiplications per sample. LFNST involves 16×16 matrix multiplications, which is 16 multiplications per sample. Therefore, the total cost of LFNST for a 4×4 block is 24 multiplications per sample.

[0107] For a 4×8 block, a naive implementation of the main transform would typically require eight 4×4 transforms along the short dimension and four 8×8 transforms along the long dimension, resulting in a total of 4+8=12 multiplications per sample. However, since LFNST only reconstructs the non-zero coefficient values ​​in the 4×4 block to the upper left of the main transform coefficient position, an optimized decoder will be able to take advantage of performing only four 4×4 transforms along the short dimension, followed by four 4×8 transforms along the long dimension, resulting in a total of 2+4=6 multiplications per sample. Since the order of separable transforms is generally fixed, in the worst case, the decoder may need to first perform four 4×8 transforms along the long dimension, followed by eight 4×4 transforms along the short dimension, resulting in 4+4=8 multiplications per sample. LFNST is still a 16×16 matrix multiplication whose cost is amortized over the larger block, resulting in 8 multiplications per sample. However, for a 4×8 block, the worst-case cost of LFNST is 16 multiplications per sample. The same principle generally applies to 4xN or Nx4 block sizes.Thus, the number of multiplications per sample for a 4xN or Nx4 block will always be less than or equal to the number of multiplications per sample for a 4x4 block.

[0108] For an 8x8 block, the main transform includes 8+8=16 multiplications per sample. LFNST includes 64x32 matrix multiplications, which is 32 multiplications per sample. The total cost of LFNST for an 8x8 block is then 48 multiplications per sample.

[0109] For 8×16 blocks, it is again assumed that the optimized decoder will exploit the zeroing property of LFNST reconstruction. Only the upper left 8×8 block of the main transform coefficient position is non-zero. The optimized decoder can then exploit this simply by performing 8 8×8 transforms along the short dimension and then 8 8×16 transforms along the long dimension, resulting in a total number of multiplications per sample of 4+8=12. Alternatively, the optimized decoder can first perform 8 8x16 transforms along the long dimension and then 16 8×8 transforms along the short dimension, resulting in a total number of multiplications per sample of 8+8=16. LFNST adds an additional (64×32) / (8×16)=16 multiplications per sample, resulting in an overall worst-case complexity of 32 multiplications per sample. As mentioned earlier, the number of multiplications per sample for an 8×N or N×8 block is always less than or equal to the number of multiplications per sample for an 8×8 block.

[0110] For 16×16 blocks, the zeroing property of LFNST reconstruction means that Figure 7C Only six 4x4 blocks of primary transform coefficients have non-zero values ​​in the pattern shown. For simplicity, a more relaxed pattern is assumed where the 12x12 block to the upper left of the primary transform position may have non-zero values. An optimized decoder can then exploit this simply by first performing 12 12x16 transforms along one dimension, and then 16 12x16 transforms along the second dimension, involving 9+12=21 multiplications per sample. LFNST involves (96x32) / (16x16)=12 multiplications per sample, resulting in an overall complexity of 33 multiplications per sample.

[0111] For M×N blocks (where M, N ≥ 16), an optimized decoder can first perform 12 12×M transforms along one dimension, followed by M 12×N transforms along the second dimension, resulting in a per-sample multiplication count of (12×12) / N+12 to perform a separable DCT. Then, at the minimum value of N=16, the worst-case complexity of 21 multiplications per sample occurs, which is equal to the complexity of a 16×16 block. LFNST adds an additional (96×32) / (M×N) multiplications per sample, which is always less than or equal to the per-sample multiplication count for a 16×16 block. Therefore, for larger M×N block sizes, the overall complexity of LFNST in ECM is always equal to or less than the per-sample multiplication count for a 16×16 block.

[0112] After an exhaustive evaluation of the decoder complexity of LFNST in ECM at different block sizes, the worst-case complexity was found to be 48 multiplications per sample (occurring in the case of 8×8 blocks). Somewhat surprisingly, this worst-case complexity includes the cost of performing a separable DCT, but due to optimizations possible from LFNST zeroing, it is significantly lower than the worst-case complexity of performing the separable DCT alone (which was estimated to be 256 multiplications per sample).

[0113] As can be seen above, the use of a non-separable secondary transform results in a significant reduction in complexity due to the use of zeroing on selected regions of the primary transform coefficients. However, further encoding is possible with NSPT. Preliminary studies on the non-separable primary transform found that significant gains (3.43% average bitrate reduction according to the Bjontegaard metric) can be achieved despite the complexity of the implemented transform and the kernel weights being obtained by overfitting the test dataset.

[0114] A practical implementation of NSPT is proposed. In this proposal, NSPT is applied only to a small set of block sizes such as 4×4, 4×8, 8×4, and 8×8. For these block sizes, NSPT replaces LFNST. Like LFNST, the NSPT kernel is trained, where the process of selecting the appropriate matrix for a particular block is guided by a combination of an index indicated by signaling and an implicit selection made by the intra prediction mode. Three types of NSPT kernels are proposed. For 4×4 blocks, a small NSPT kernel of dimensions 16×16×3×35 is used. For 4×8 and 8×4 blocks, a medium NSPT kernel of dimensions 32×20×3×35 is used. For 8x8 blocks, a large NSPT kernel of dimensions 64×32×3×35 is used.

[0115] In general, it is possible to not use zero-set transforms in the proposed NSPT, so the first dimension of each kernel is always equal to the number of samples in the block. For medium and large NSPT kernels, the second dimension is smaller than the first dimension, which means that NSPT is a lossy transform in these cases. As with LFNST, the NSPT index is signaled in the bitstream and can take values ​​0, 1, 2, or 3, where 0 indicates that NSPT is not used to transform the block, and values ​​1 to 3 indicate a selection among the corresponding NSPT kernels along the third dimension. The selection along the 4th dimension of the NSPT kernel is determined by mapping the intra prediction mode in the same way as for the extended LFNST in ECM.

[0116] The block sizes and kernel sizes for which NSPT is enabled are carefully designed to enable NSPT. This can be confirmed by comparing the complexity of NSPT for each block size with the complexity of the corresponding LFNST that NSPT replaces. For 4×4 blocks, the complexity of NSPT is 16 multiplications per sample, while the complexity of LFNST is 24 multiplications per sample. For 4×8 and 8×4 blocks, the complexity of NSPT is 20 multiplications per sample, while the complexity of LFNST is 16 multiplications per sample. For 8×8 blocks, the complexity of NSPT is 32 multiplications per sample, while the complexity of LFNST is 48 multiplications per sample. Therefore, the worst-case decoder complexity is not increased.

[0117] In the proposed implementation of NSPT described above, the NSPT core is designed to replace LFNST for a specific block size so that the decoder complexity in terms of the worst-case number of multiplications per sample does not increase. The following embodiments provide several schemes for improving the gain of NSPT while maintaining the worst-case decoder complexity.

[0118] Therefore, return Figures 1 to 3 In step S340, in response to the determined block size and the determined NSPT index, the transform module 204 may perform NSPT on the residual block to generate a plurality of coefficients based on the kernel. In one embodiment of the present invention, in response to the block size of the residual block being 8×16, an NSPT transform matrix is ​​selected from a kernel of dimensions A×B×C×D, where the value of parameter A is 128, the value of parameter C is 3, and the value of parameter D is 35. It should be noted that in the present disclosure, dimension "A" represents the number of input samples from the residual block, dimension "B" represents the number of transform coefficients generated by NSPT, dimension "C" represents the number of index sets, and dimension "D" represents the number of transform sets. The order of the dimensions of the kernel described in the present disclosure is provided only as an example for describing how to select the NSPT transform matrix, and it may be different in the implementation of NSPT.

[0119] The NSPT transform matrix is ​​selected by indexing the "C" dimension and the "D" dimension of the kernel, resulting in an NSPT transform matrix of size A×B. The value of the NPST index can be 1, 2 or 3, and the value of the NSPT index is used to index the "C" dimension of the kernel. The value of the NSPT index can be determined by the following operation: the video encoder 100 performs rate-distortion optimization (RDO) on all possible values ​​of the NSPT index and selects the value that minimizes the RD cost. As described above with reference to step S320, the prediction block is generated by intra-frame prediction associated with the intra-frame prediction mode. The intra-frame prediction mode is mapped to the transform set index through the mapping process described above with reference to Table 2. The transform set index is used to index the "D" dimension of the kernel.

[0120] In one example of this embodiment of the present invention, NSPT is performed on the residual block by matrix multiplication. The samples of the residual block can be arranged in a one-dimensional column vector R of 128×1 samples in raster scan order. Let the selected NSPT transformation matrix be T with size A×B, which is 128×B in this embodiment. Then, NSPT can be performed by performing matrix multiplication on the samples of the residual block using the transpose of the NSPT transformation matrix, that is, P=T T R, so that the result of the matrix multiplication P is a one-dimensional column vector of B×1 transform coefficients. Note that in matrix multiplication, T T Each column of corresponds to a sample position in the column vector R, and the sample positions in the column vector R correspond to the spatial positions in the residual block according to the above raster scan order. The above raster scan order for constructing v is only an example, and in different implementations, if the NSPT kernel is modified so that T T If the columns of P correspond to the same spatial positions in the residual block, different scan orders can be used. The column vector P can then be inserted into a two-dimensional transform coefficient block of the same size as the residual block in a hierarchical diagonal scan order. Either the one-dimensional vector P or the two-dimensional transform coefficient block can be referred to as a plurality of coefficients.

[0121] In another embodiment of the present invention, in response to the block size of the residual block being 16×8, the NSPT transform matrix T is selected from a kernel of dimensions A×B×C×D, where the value of parameter A is 128, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, NSPT is performed on the residual block by matrix multiplication, P=T TR, where the samples of the residual block are arranged in raster scan order into a one-dimensional column vector R of 128×1 samples. The column vector P can then be inserted into a two-dimensional transform coefficient block of the same size as the residual block in a hierarchical diagonal scan order. The one-dimensional vector P or the two-dimensional transform coefficient block can be referred to as a plurality of coefficients.

[0122] Specifically, NSPT can be extended for use with 8×16 and 16×8 block sizes and replaces LFNST for these block sizes. For these block sizes, additional NSPT kernels with dimensions 128×B×C×D are trained, where parameter B is the number of transform coefficients produced by NSPT from the encoder's perspective. Parameter C can be set to 3, and parameter D can be set to 35 to match the current dimensions of kernels used in both the extended LFNST kernel in ECM and the proposed NSPT, although these dimensions may change if the overall design of LFNST and NSPT is further modified. The number of transform coefficients is set so that the worst-case decoder complexity when using the extended NSPT and the overall encoder complexity when using the extended NSPT are not inferior to the corresponding complexities of the LFNST tool that the NSPT replaces. The worst-case LFNST complexity for 8×16 and 16×8 blocks is 32 multiplications per sample. In one embodiment of the present invention, parameter B = 32. However, in another embodiment of the present invention, parameter B may be set to a value close to but not equal to 32, with the exact value being chosen based on empirical verification such that the value of parameter B does not increase the reference encoder complexity.

[0123] In another embodiment of the present invention, in response to the block size of the residual block being 16×16, the NSPT transform matrix T is selected from a kernel of dimensions A×B2×C×D, where the value of parameter A is 256, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, NSPT is performed on the residual block by matrix multiplication, i.e., P=T T R, where the samples of the residual block are arranged in raster scan order into a one-dimensional column vector R of 256×1 samples. The column vector can then be inserted into a two-dimensional transform coefficient block of the same size as the residual block in a hierarchical diagonal scan order. The one-dimensional vector P or the two-dimensional transform coefficient block can be referred to as a plurality of coefficients.

[0124] Specifically, NSPT can be extended for use with 8×16, 16×8, and 16×16 block sizes and replace LFNST for these block sizes. For 8×16 and 16×8 block sizes, an additional NSPT kernel with dimensions 128×B1×C×D is trained, whereas for 16×16 block size, another additional NSPT kernel with dimensions 256×B2×C×D is trained. Parameter C can be set to 3 and parameter D to 35 to match the current dimensions of the kernels used in both the extended LFNST kernel in ECM and the proposed NSPT, however these dimensions may change if the overall design of LFNST and NSPT is further modified. The number of transform coefficients (B1 and B2) of each additional NSPT kernel is set so that the worst-case decoder complexity when using the extended NSPT and the overall complexity of the encoder when using the extended NSPT are not inferior to the corresponding complexities of the LFNST tool replaced by the NSPT. The worst-case LFNST complexity for 8×16 and 16×8 blocks is 32 multiplications per sample, while the worst-case LFNST complexity for 16×16 blocks is approximately 33 multiplications per sample (depending on the degree to which the main transform is optimized for zeroed transform coefficient positions). Thus, in one embodiment of the present invention, parameter B1 is 32, and parameter B2 is 32. In another embodiment of the present invention, parameters B1 and B2 can be set to values ​​close to, but not equal to, 32, with the exact values ​​being chosen based on empirical verification so that these values ​​do not increase the reference encoder complexity.

[0125] In step S350, the quantization module 207 may quantize the multiple coefficients to generate multiple quantized coefficients. In step S360, the entropy coding module 208 may encode the multiple quantized coefficients into an output bitstream. In step S370, the entropy coding module 208 may encode the NSPT index into the output bitstream. Thus, the video encoder 100 and its video coding method can implement low-complexity NSPT to compress residual data using the kernel matrix designed above.

[0126] Figure 8 Schematic diagram of a zeroing area according to an embodiment of the present invention. Figures 1 to 3 as well as Figure 8, the prediction module 203 can also perform zeroing on the residual block so that the residual block after zeroing has a zeroing area. From the encoder perspective, zeroing can be applied to the input side of the NSPT kernel (residual sample position) to further reduce its complexity. With the LFNST tool, the retained DCT transform coefficient positions occupy the upper left (low-frequency) part of the transform block, while the zeroing area is located in the lower right (high-frequency) part of the transform block. In contrast, the retained residual sample positions of NSPT are located in the lower right part of the residual block, while the zeroing area is the sample position at the upper left edge of the residual block. The reason for this is that intra-frame prediction is based on a set of reference samples occupying the upper left neighborhood of the coding block, and the correlation between the reference samples and the coding block samples is highly dependent on spatial proximity. At sample positions close to the reference neighborhood, the amplitude of the residual samples (which is equivalent to the prediction error) is likely to be the smallest. These positions are at the upper left edge of the residual block, and therefore, if they are completely ignored by zeroing, they are least likely to cause significant reconstruction errors.

[0127] For example, Figure 8 As shown, Figure 8 An example arrangement of 8×8 blocks is shown. In this example, the top left edge of the sample positions with a thickness of 1 is zeroed. From the encoder perspective, the bottom right 7×7 sample positions (e.g. Figure 8 The upper left edge of the sample position ( Figure 8 The NSPT kernel for 4×4, 4×8, and 8×4 block sizes is represented by a region of zeros (shown as the white area in the middle) that is ignored, or “zeroed”. From the decoder’s perspective, the inverse NSPT is applied to produce the lower right 7×7 residual samples, with the remaining residual samples padded with zeros. This kernel has a complexity of 24.5 multiplications per sample, compared to the NSPT kernel for 8×8 blocks, which has a complexity of 32 multiplications per sample, a 24% reduction. A similar zeroing region can be defined along the upper left edge of the NSPT kernel for 4×4, 4×8, and 8×4 block sizes, reducing the NSPT kernel to 9×9×3×35 (5 multiplications per sample, a 69% reduction in complexity) for 4×4 blocks and 21×20×3×35 (13.1 multiplications per sample, a 35% reduction in complexity) for 4×8 or 8×4 blocks.

[0128] In another embodiment of the present invention, Figure 8 Compared to the example of [ 001 ], a reduced zeroing region is applied to some NSPT kernels. For example, the zeroing region can be reduced to only a single top-left sample position of the residual block. The size of the zeroing region can be designed for each NSPT kernel to trade off complexity and reconstruction error.

[0129] In another embodiment of the present invention, Figure 8Compared to the example of , an enlarged zeroing region with a thickness N is applied to some NSPT kernels. For example, the thickness of the zeroing region can be increased to the upper left edge of the sample position 2 sample widths.

[0130] In another embodiment of the present invention, the intra-frame prediction mode can be selected according to the Figure 8 Compared to the reduced zeroed area, for example, when the intra prediction mode is mainly in the horizontal direction (indicating that the CU is substantially predicted by left reference samples), the reduced zeroed area may be the left edge of the residual block.

[0131] See also Figures 1 to 3 In response to the size of the residual block being larger than the block size supported by the maximum NSPT kernel, the transform module 204 may further divide the residual block into a plurality of smaller transform blocks. Specifically, the maximum NSPT kernel size is limited to being applied to blocks of size M×N. For example, M may be 16 and N may be 16. However, for block sizes up to and including PxQ (where P>M and Q>N), the NSPT tool is enabled and replaces the LFNST. Let the coding block size be RxS. In this embodiment, the coding block is larger than the block size supported by the maximum NSPT kernel, i.e., R≥M and S≥N. The intra-frame prediction mode is indicated for the coding block by signaling, and from the encoder's perspective, a prediction value is determined according to the intra-frame prediction mode, and the prediction value is subtracted from the coding block sample to generate a residual. In this regard, a non-zero NSPT index is indicated by signaling, thereby indicating that NSPT will be applied to the residual, while a zero NSPT index indicates that NSPT will not be applied to the residual. In this embodiment of the present invention, the transform is not directly applied to the residual block, but the residual block is implicitly partitioned or divided into smaller transform blocks before applying NSPT. Therefore, the NSPT of the above embodiment can be extended to a larger range of block sizes.

[0132] Figure 9A FIG. 4 is a schematic diagram of segmenting a residual block according to an embodiment of the present invention. Figure 9B FIG. 1 is a schematic diagram of segmenting a residual block according to another embodiment of the present invention. Figures 1 to 3 、 Figure 9A and Figure 9BIn one embodiment of the present invention, the RxS residual block of the above embodiment can be implicitly split into transform blocks of size MxN, when possible. When R and S are not integer multiples of M and N, respectively, smaller blocks of dimensions (R%M) and (S%N) appear, where % represents the modulo or "remainder" operator. In one embodiment of the present invention, the implicit split is aligned with the upper left corner of the residual block, so that smaller blocks appear at the bottom and right of the residual block (when smaller blocks exist). In another embodiment of the present invention, the implicit split is aligned with the lower right corner of the residual block, so that smaller blocks appear at the top and left of the residual block (when smaller blocks exist). Figure 9A and Figure 9B Two arrangements of implicit partitioning are shown in . Furthermore, each transform block is processed by an NSPT kernel for the corresponding block size, where the choice of a specific transform matrix is ​​determined by the signaled NSPT index and by mapping the intra prediction mode to a transform set index.

[0133] The advantage can be understood by considering the implicit partitioning into an integer multiple of MxN transform blocks. In such a case, the number of multiplications per sample is equal to the number of output transform coefficients B generated by the selected NSPT. However, the total number of transform coefficients generated for the coding block is B per transform block. If NSPT is applied directly to the coding block without implicit partitioning, then in order to achieve the same number of multiplications per sample, the NSPT kernel will also need to include B output transform coefficients. However, in this case, the total number of transform coefficients for the entire coding block is B. This number of transform coefficients may not be enough to reconstruct the coding block without very large reconstruction errors, thus making the direct application of NSPT to large blocks useless for video coding.

[0134] In addition, Figure 8 In the case of the embodiment of FIG5 , the zeroing can be applied in a restrictive manner only to the transform blocks located at the upper edge or the left edge of the residual block. This means that depending on the position of the transform block in the residual, either a zeroed version or a full-size version of the same NSPT kernel can be applied.

[0135] In one embodiment of the present invention, the NSPT index may be signaled for each transform block. This increases the signaling cost of the NSPT tool, but potentially reduces reconstruction error, making it possible to improve the overall rate-distortion tradeoff.

[0136] In one embodiment of the present invention, an NSPT index can be signaled for each unique transform block size. Transform blocks of the same block size share a common NSPT index. Therefore, depending on whether the residual block is an integer multiple of the maximum block size supported by the NSPT kernel, the total number of NSPT indices signaled can be 1, 2, or 4.

[0137] In one embodiment of the present invention, an intra prediction mode is signaled for a coding block, and then a non-zero NSPT index is signaled, indicating that NSPT is to be applied. Unlike the above-described embodiments, when the coding block is larger than the block size supported by the maximum NSPT kernel, from the encoder's perspective, the coding block is implicitly split into smaller prediction blocks before applying intra prediction. This implicit splitting follows the same strategy as described above with reference to FIG9 , except that prediction blocks are generated instead of transform blocks. Each prediction block is predicted according to the intra prediction mode indicated by the signaling, resulting in a residual block from each prediction block. However, each prediction value is determined based on a reference sample directly adjacent to the upper left boundary of the corresponding prediction block. Figure 10 An example of this prediction process is shown for one prediction block. Figure 10 Figure 2 is a schematic diagram of local prediction according to another embodiment of the present invention. From the decoder's perspective, this means that the prediction value cannot be determined until the top and left prediction blocks are fully reconstructed. This can reduce the size of the residual because the prediction is performed locally, but at the expense of increased sequence dependency. However, the sequence dependency is no worse than the worst-case scenario that the decoder may already encounter when the CTU is divided into the smallest possible CU size.

[0138] Figure 11 Schematic diagram of a video decoder provided by an embodiment of the present invention. Figure 11 The video decoder 1100 includes a processor 1110, a storage device 1120, a communication interface 1130, and a data bus 1140. The processor 1110 is electrically connected to the storage device 1120 and the communication interface 1130 via the data bus 1140. The storage device 1120 can store relevant instructions and can also store the relevant video decoder of the algorithm. The processor 1110 can receive a bitstream from the communication interface 1130. The processor 1110 can execute the relevant video decoder and / or relevant instructions to implement the video decoding method of the present invention.

[0139] In one embodiment of the present invention, the video decoder 1100 may be implemented by one or more PCs, one or more server computers, and one or more workstation computers, or may be composed of multiple computing devices, but the present invention is not limited thereto. In another embodiment of the present invention, the video decoder 1100 may include more processors for executing relevant video encoders and / or related instructions to implement the video decoding method of the present invention. Furthermore, in one embodiment of the present invention, the video decoder 1100 may include more processors for executing relevant video decoders and / or related instructions to implement the video decoding method of the present invention. The video decoder 1100 may be used to implement a video codec and may perform the video decoding function of the present invention.

[0140] In one embodiment of the present invention, the processor 1110 may include, for example, a CPU, a GPU, or other programmable general-purpose or special-purpose microprocessor, a DSP, an ASIC, a PLD, or other similar processing circuits, or a combination of these devices. In one embodiment of the present invention, the storage device 1120 may be a non-transitory computer-readable recording medium, such as a ROM, EPROM, EEPROM, or NVM, but the present invention is not limited thereto.

[0141] In one embodiment of the present invention, the relevant video decoder and / or related instructions may also be stored in a non-transitory computer-readable recording medium of one device and executed by a processor of another device. Communication interface 1130 may be, for example, a network card supporting wired network connections such as Ethernet, a wireless network card supporting wireless communication standards such as IEEE 802.11n / b / g / ac / ax / be, or any other network connection device, but the embodiment is not limited thereto. Communication interface 1130 is configured to obtain an input bitstream.

[0142] Figure 12 Schematic diagram of the video decoding process provided by an embodiment of the present invention. Figure 11 and Figure 12 , the video decoder 1100 can execute Figure 12 The video decoding process decodes the input bitstream into an output video. In this embodiment of the present invention, the storage device 120 may store algorithms for the entropy decoding module 1201, the prediction module 1202, the inverse quantization module 1203, the inverse transform module 1204, the reconstruction module 1205 (e.g., an addition module or a subtraction module), the filtering module 1206, and the decoded picture buffer module 1207. The processor 1110 may execute the above modules to perform the video decoding process.

[0143] In one embodiment of the present invention, the processor 1110 may receive an input bitstream from an external video source. The entropy decoding module 1201 may parse the input bitstream and obtain syntax element values ​​from the input bitstream. The entropy decoding module 1201 may decode the input bitstream to obtain multiple quantized coefficients and may decode entropy-encoded syntax elements from the input bitstream. The entropy decoding module 1201 may convert the binary representation of the entropy-encoded syntax elements into numerical values. The entropy decoding module 1201 may transmit the syntax element values ​​and one or more variables set or determined based on the syntax element values ​​to various modules within the video decoder 1200 to obtain one or more decoded pictures. The prediction module 1202 may determine a prediction block for the current coding block (CB). Note that although this process involves decoding, these blocks are still referred to as coding blocks. The prediction module 1202 may include an intra-frame prediction module. When an intra-frame prediction mode is indicated for decoding the current CB, the prediction module 1202 may transmit relevant parameters from the entropy decoding module 1201 to the intra-frame prediction module to obtain an intra-frame prediction block.

[0144] The inverse quantization module 1203 can inverse quantize the multiple quantized coefficients to generate multiple reconstructed coefficients. The inverse transform module 1204 can perform inverse NSPT on the multiple reconstructed coefficients to generate a reconstructed residual block based on the (NSPT) kernel. The reconstruction module 1205 can generate a reconstructed CB based on the reconstructed residual block and the prediction block. In addition, the reconstructed CB can also be sent to the prediction module 1202 for use as a reference for other blocks encoded in intra-frame prediction mode.

[0145] After all CBs in the current picture or current sub-picture have been reconstructed, the reconstructed CBs are merged to generate a reconstructed picture or reconstructed sub-picture. The filtering module 1206 can perform in-loop filtering on the reconstructed picture or reconstructed sub-picture. The filtering module 1206 may include one or more in-loop filtering operations, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, a luminance mapping and chroma scaling (LMCS) filter, and a neural network-based loop filter (NNLF). The filtering module 1206 can output a decoded picture or a decoded sub-picture, and the decoded picture or the decoded sub-picture is cached in the decoded picture cache module 1207. The decoded picture cache module 1207 can output a decoded picture or a decoded sub-picture according to timing and control information. The picture stored in the decoded picture cache module 1207 can be used as a reference for the prediction module 1202 to perform inter-frame prediction or intra-frame prediction.

[0146] Figure 13 Flowchart of the video decoding method provided by the embodiment of the present invention. Figures 11 to 13, the video decoder 1100 may perform the following steps S1310 to S1370 to implement the video decoding method. In step S1310, the video decoder 1100 may receive an input code stream. In step 1320, the prediction module 1202 may determine a prediction block of a current coding block (CB) based on an intra-frame prediction mode. The intra-frame prediction mode may be determined by a prediction mode flag decoded from the input code stream. In step 1330, the entropy decoding module 1201 may decode the input code stream to obtain a plurality of quantized coefficients. In step 1340, the entropy decoding module 1201 may decode the input code stream to obtain an NSPT index having a value of 1, 2, or 3. In step S1350, the inverse quantization module 1203 may inverse quantize the plurality of quantized coefficients to generate a plurality of reconstructed coefficients. In step S1360 , in response to the determined block size and the decoded NSPT index, the inverse transform module 1204 may perform an inverse non-separable primary transform (NSPT) on the plurality of reconstruction coefficients to generate a reconstructed residual block based on the kernel.

[0147] In one embodiment of the present invention, in response to the block size of the current CB being 8×16, an NSPT transform matrix is ​​selected from a kernel of dimensions A×B×C×D, where the value of parameter A is 128, the value of parameter C is 3, and the value of parameter D is 35. The size of the current CB can be determined by a partitioning mode flag previously decoded from the input code stream. It should be noted that in the present disclosure, dimension "A" represents the number of input samples from the residual block, dimension "B" represents the number of transform coefficients generated by the NSPT, dimension "C" represents the number of index sets, and dimension "D" represents the number of transform sets. The order of the dimensions of the kernel described in the present disclosure is provided only as an example for describing how to select the NSPT transform matrix, and it may be different in the implementation of NSPT.

[0148] The NSPT transform matrix is ​​selected by indexing the "C" and "D" dimensions of the kernel, resulting in an NSPT transform matrix of size A×B. The value of the NSPT index is used to index the "C" dimension of the kernel. As described above with reference to step S1320, a prediction block is generated by intra prediction associated with the intra prediction mode. The intra prediction mode is mapped to a transform set index through the mapping process described above with reference to Table 2. The transform set index is used to index the "D" dimension of the kernel.

[0149] In one example of this embodiment of the present invention, inverse NSPT is performed on multiple reconstruction coefficients by matrix multiplication. The reconstruction coefficients are typically decoded in a hierarchical reverse diagonal scan order, which means that the coefficients corresponding to high spatial frequencies are decoded before the coefficients corresponding to low spatial frequencies. The reconstruction coefficients can then be arranged into a one-dimensional column vector P of B×1 samples in reverse order (i.e., a hierarchical forward diagonal scan order). Let the selected NSPT transform matrix be T of size A×B, where A×B is 128×B in this embodiment. The inverse NSPT can then be performed by performing matrix multiplication on the reconstruction coefficients using the NSPT transform matrix, i.e., R=TP, so that the result of the matrix multiplication R is a one-dimensional column vector of 128×1 reconstructed residual coefficients. The column vector R can then be arranged into a two-dimensional reconstructed residual block of the same size as the current CB in a raster scan order. Note that in the matrix multiplication, each row of T corresponds to a sample position in the column vector R, which corresponds to a spatial position in the reconstructed residual block in the above-mentioned raster scan order. The raster scan order described for constructing the reconstructed residual block is only an example, and in different implementations, different scan orders may be used if the NSPT kernel is modified so that the columns of T correspond to the same spatial positions in the reconstructed residual block.

[0150] In another embodiment of the present invention, in response to the block size of the current CB being 16×8, an NSPT transform matrix T is selected from a kernel of dimensions A×B×C×D, where the value of parameter A is 128, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, an inverse NSPT is performed on a plurality of reconstruction coefficients by matrix multiplication, i.e., R=TP, where the samples of R are arranged in raster scan order to generate a two-dimensional reconstructed residual block.

[0151] Specifically, NSPT can be extended for use with 8×16 and 16×8 block sizes and replaces LFNST for these block sizes. For these block sizes, an additional NSPT kernel with dimensions 128×B×C×D is trained, where parameter B is the number of transform coefficients produced by NSPT from the encoder's perspective. The additional NSPT kernel is the same as the additional NSPT kernel available to video encoder 100. In one embodiment of the present invention, parameter B=32. However, in another embodiment of the present invention, parameter B can be set to a value close to, but not equal to, 32, the exact value of which is selected based on empirical verification so that this value of parameter B does not increase the reference encoder complexity.

[0152] In another embodiment of the present invention, in response to the block size of the current CB being 16×16, an NSPT transform matrix T is selected from a kernel of dimensions A×B×C×D, where the value of parameter A is 256, the value of parameter C is 3, and the value of parameter D is 35. The value of the NSPT index is used to index the "C" dimension of the kernel, and the transform set index is used to index the "D" dimension of the kernel. In one example of this embodiment of the present invention, an inverse NSPT is performed on a plurality of reconstruction coefficients by matrix multiplication, i.e., R=TP, where the samples of R are arranged in raster scan order to generate a two-dimensional reconstructed residual block.

[0153] Specifically, NSPT can be extended to be used for 8×16, 16×8, and 16×16 block sizes and replace LFNST for these block sizes. For 8×16 and 16×8 block sizes, additional NSPT kernels with dimensions 128×B1×C×D are trained, while for 16×16 block size, another additional NSPT kernel with dimensions 256×B2×C×D is trained. The additional NSPT kernels are the same as those available to video encoder 100. In one embodiment of the present invention, parameters B1=32, B2=32. In another embodiment of the present invention, parameter B1 can be set to a value close to but not equal to 32, the exact value of which is selected based on empirical verification so that these values ​​of parameter B1 do not increase the complexity of the reference encoder.

[0154] In step S1370, the reconstruction module 1205 may generate a reconstructed CB by sample-by-sample summation of the reconstructed residual block and the prediction block. The video decoder 1100 may combine the reconstructed CB with other reconstructed CBs to generate a reconstructed picture or a reconstructed sub-picture. After the filtering module 1206 performs in-loop filtering to generate a decoded picture or a decoded sub-picture, the video decoder 1100 may output the decoded picture or the decoded sub-picture to the output video.

[0155] In summary, the video encoding method, video decoding method, video encoder, and video decoder of the present invention propose several methods for achieving low-complexity NSPT in video encoding and decoding. The video encoding method, video decoding method, video encoder, and video decoder of the present invention can effectively improve the coding performance of video coding by introducing a larger NSPT core with a complexity comparable to that of the corresponding LFNST core, a new zeroing strategy for the NSPT core, and / or extending the NSPT core to larger block sizes through implicit partitioning. The video encoding method, video decoding method, video encoder, and video decoder of the present invention can be used in future video coding standards.

[0156] It will be apparent to those skilled in the art that various modifications and variations may be made to the disclosed embodiments without departing from the scope or spirit of the present disclosure. In view of the foregoing, the present disclosure is intended to cover modifications and variations as long as they fall within the scope of the appended claims and their equivalents.

[0157] Reference Signs List

[0158] 100: Video encoder

[0159] 110, 1110: Processor

[0160] 120, 1120: Storage devices

[0161] 130, 1130: Communication interface

[0162] 140: Data bus

[0163] 201: Divide modules

[0164] 202: Arithmetic Module

[0165] 203: Prediction Module

[0166] 204: Transformation Module

[0167] 205: Quantization Module

[0168] 206: Entropy Coding Module

[0169] 207: Inverse quantization module

[0170] 208: Inverse quantization module

[0171] 209: Reconstruction Module

[0172] 210: Filter module

[0173] 211: Decoding image cache module

[0174] 400: Input image

[0175] 401: CTU

[0176] 402:CU

[0177] 1100: Video decoder

[0178] 1201: Entropy decoding module

[0179] 1202: Prediction module

[0180] 1203: Inverse quantization module

[0181] 1204: Inverse transform module

[0182] 1205: Rebuild module

[0183] 1206: Filter module

[0184] 1207: Decoded image cache module

[0185] S310~S370, S1310~S1370: steps.

Claims

1. A video encoder, characterized in that include: a partitioning module for receiving an input video and generating a plurality of coding blocks (CBs) of the input video; a prediction module, coupled to the partitioning module, for generating a prediction block of the current CB; an arithmetic module, coupled to the partitioning module and the prediction module, configured to calculate a residual block based on the current CB and the prediction block; a transform module, coupled to the arithmetic module, and in response to the residual block having an 8×16 block size, a 16×8 block size, or a 16×16 block size, and the determined NSPT index having a value of 1, 2, or 3, the transform module for performing a non-separable primary transform (NSPT) on the residual block to generate a plurality of coefficients based on a kernel; a quantization module, coupled to the transform module, configured to quantize the plurality of coefficients to generate a plurality of quantized coefficients; as well as An entropy coding module, coupled to the quantization module, is configured to encode the plurality of quantization coefficients and the NSPT index into an output bitstream.

2. The video encoder according to claim 1, wherein The partitioning module is configured to partition an input picture of the input video into a plurality of coding tree units (CTUs), and to partition each CTU into one or more CUs to generate a plurality of CUs and equivalently generate the plurality of CBs.

3. The video encoder according to claim 1, wherein The prediction module is used to generate a prediction block of the current CB through intra-frame prediction.

4. The video encoder according to claim 1, wherein In response to the residual block having an 8×16 block size, the transform module is used to perform the NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, wherein parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35. The video encoder according to claim 1 , wherein: In response to the residual block having a 16×8 block size, the transform module is used to perform the NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, wherein parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35. The video encoder according to claim 1 , wherein: In response to the residual block having a 16×16 block size, the transform module is used to perform the NSPT using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D, wherein parameter B2 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

7. The video encoder according to claim 4, wherein The parameter B1 is 32.

8. The video encoder according to claim 4, wherein The parameter B1 is close to but not equal to 32.

9. The video encoder according to claim 5, wherein The parameter B1 is 32.

10. The video encoder according to claim 5, wherein The parameter B1 is close to but not equal to 32.

11. The video encoder according to claim 6, wherein The parameter B1 and the parameter B2 are 32.

12. The video encoder according to claim 6, wherein The parameter B1 and the parameter B2 are close to but not equal to 32.

13. The video encoder according to claim 1, wherein The prediction module further performs zeroing on the residual block, so that the residual block after zeroing has a zeroing area.

14. The video encoder according to claim 1, wherein In response to a size of the residual block being larger than a block size supported by a maximum NSPT kernel, the transform module is further configured to divide the residual block into a plurality of smaller transform blocks.

15. The video encoder according to claim 14, wherein For each transform block in the plurality of smaller transform blocks, an NSPT index is indicated by signaling.

16. The video encoder according to claim 14, wherein: For each unique transform block size, an NSPT index is signaled, and transform blocks with the same block size in the plurality of smaller transform blocks share a common NSPT index.

17. The video encoder according to claim 1, wherein In response to the size of the residual block being larger than a block size supported by a maximum NSPT kernel, the current CB is implicitly partitioned into a plurality of smaller prediction blocks before prediction.

18. A video encoding method, characterized in that: include: receiving an input video and generating a plurality of coding blocks (CBs) of the input video; Generate the prediction block of the current CB; Calculating a residual block according to the current CB and the prediction block; In response to the residual block having an 8×16 block size, a 16×8 block size, or a 16×16 block size, and the determined NSPT index having a value of 1, 2, or 3, performing a non-separable primary transform (NSPT) on the residual block to generate a plurality of coefficients based on a kernel; quantizing the plurality of coefficients to generate a plurality of quantized coefficients; Encoding the plurality of quantized coefficients into an output bitstream by entropy coding; as well as The NSPT index is encoded into the output code stream.

19. The video encoding method according to claim 18, wherein: Also includes: Dividing an input picture of the input video into a plurality of coding tree units (CTUs), and Each CTU is divided into one or more CUs to generate multiple CUs and equivalently generate multiple coding blocks (CBs).

20. The video encoding method according to claim 18, wherein: The prediction block of the current CB is generated by intra prediction.

21. The video encoding method according to claim 18, wherein: In response to the residual block having an 8×16 block size, the NSPT is performed using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, where parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

22. The video encoding method according to claim 18, wherein: In response to the residual block having a 16×8 block size, the NSPT is performed using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, where parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

23. The video encoding method according to claim 18, wherein: In response to the residual block having a 16×16 block size, the NSPT is performed using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D, where parameter B2 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

24. The video encoding method according to claim 21, wherein: The parameter B1 is 32.

25. The video encoding method according to claim 21, wherein: The parameter B1 is close to but not equal to 32.

26. The video encoding method according to claim 22, wherein: The parameter B1 is 32.

27. The video encoding method according to claim 22, wherein: The parameter B1 is close to but not equal to 32.

28. The video encoding method according to claim 23, wherein: The parameter B1 and the parameter B2 are 32.

29. The video encoding method according to claim 23, wherein: The parameter B1 and the parameter B2 are close to but not equal to 32.

30. The video encoding method according to claim 18, wherein: Also includes: Zeroing is performed on the residual block, so that the residual block after zeroing has a zeroing area.

31. The video encoding method according to claim 18, wherein: In response to a size of the residual block being larger than a block size supported by a maximum NSPT kernel, the residual block is divided into a plurality of smaller transform blocks.

32. The video encoding method according to claim 31, wherein: For each transform block in the plurality of smaller transform blocks, an NSPT index is indicated by signaling.

33. The video encoding method according to claim 31, wherein: For each unique transform block size, an NSPT index is signaled, and transform blocks with the same block size in the plurality of smaller transform blocks share a common NSPT index.

34. The video encoding method according to claim 18, wherein: In response to the size of the residual block being larger than a block size supported by a maximum NSPT kernel, the current CB is implicitly partitioned into a plurality of smaller prediction blocks before prediction.

35. A video decoder, characterized in that include: An entropy decoding module, configured to receive an input code stream and decode the input code stream to obtain a plurality of quantization coefficients and NSPT indices; a prediction module, coupled to the entropy decoding module, configured to determine a prediction block; an inverse quantization module, coupled to the entropy decoding module, configured to inverse quantize the plurality of quantized coefficients to generate a plurality of reconstruction coefficients; an inverse transform module, coupled to the inverse quantization module, and in response to a current coding block (CB) having an 8×16 block size, a 16×8 block size, or a 16×16 block size, and the NSPT index having a value of 1, 2, or 3, the inverse transform module being configured to perform an inverse non-separable primary transform (NSPT) on the plurality of reconstruction coefficients to generate a reconstructed residual block based on a kernel; as well as A reconstruction module is coupled to the prediction module and the inverse transform module, and is configured to generate a reconstructed CB according to the residual block and the prediction block.

36. The video decoder according to claim 35, wherein: The reconstruction module is configured to combine the reconstructed CB with a plurality of reconstructed CBs to generate the output video.

37. The video decoder according to claim 35, wherein: In response to a flag or a flag set indicating that intra prediction is applied, the prediction module is configured to determine a prediction block of the current CB by intra prediction.

38. The video decoder according to claim 35, wherein: In response to the current CB having an 8×16 block size, the inverse transform module is used to perform inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, wherein parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

39. The video decoder according to claim 35, wherein: In response to the current CB having a 16×8 block size, the inverse transform module is used to perform inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, wherein parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

40. The video decoder according to claim 35, wherein In response to the current CB having a 16×16 block size, the inverse transform module is used to perform inverse NSPT using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D, wherein parameter B2 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

41. The video decoder according to claim 38, wherein The parameter B1 is 32.

42. The video decoder according to claim 38, wherein The parameter B1 is close to but not equal to 32.

43. The video decoder according to claim 39, wherein: The parameter B1 is 32.

44. The video decoder according to claim 39, wherein: The parameter B1 is close to but not equal to 32.

45. The video decoder according to claim 40, wherein: The parameter B1 and the parameter B2 are 32.

46. ​​The video decoder of claim 40, wherein: The parameter B1 and the parameter B2 are close to but not equal to 32.

47. A video decoding method, characterized in that: include: Receive input code stream; Determine a prediction block according to the input code stream; Performing entropy decoding on the input bitstream to obtain a plurality of quantization coefficients and NSPT indexes; Inverse quantizing the plurality of quantized coefficients to generate a plurality of reconstruction coefficients; In response to a current coding block (CB) having an 8×16 block size, a 16×8 block size, or a 16×16 block size, and the NSPT index having a value of 1, 2, or 3, performing an inverse non-separable primary transform (NSPT) on the plurality of reconstruction coefficients to generate a reconstructed residual block based on the kernel; as well as A reconstructed CB is generated according to the residual block and the prediction block.

48. The video decoding method according to claim 47, wherein: Also includes: The reconstructed CB is combined with a plurality of reconstructed CBs to generate the output video.

49. The video decoding method according to claim 47, wherein: In response to a flag or a flag set indicating that intra prediction is applied, a prediction block of the current CB is determined by intra prediction.

50. The video decoding method according to claim 47, wherein: In response to the current CB having an 8×16 block size, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, where parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

51. The video decoding method according to claim 47, wherein: In response to the current CB having a 16×8 block size, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of dimensions 128×B1×C×D, where parameter B1 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

52. The video decoding method according to claim 47, wherein: In response to the current CB having a 16×16 block size, the inverse NSPT is performed using an NSPT transform matrix selected from a kernel of dimensions 256×B2×C×D, where parameter B2 is the number of transform coefficients generated by the NSPT from the encoder perspective, parameter C is 3, and parameter D is 35.

53. The video decoding method according to claim 50, wherein: The parameter B1 is 32.

54. The video decoding method according to claim 50, wherein: The parameter B1 is close to but not equal to 32.

55. The video decoding method according to claim 51, wherein: The parameter B1 is 32.

56. The video decoding method according to claim 51, wherein: The parameter B1 is close to but not equal to 32.

57. The video decoding method according to claim 52, wherein: The parameter B1 and the parameter B2 are 32.

58. The video decoding method according to claim 52, wherein: The parameter B1 and the parameter B2 are close to but not equal to 32.

Images