Decoding method, encoding method, decoder, and encoder

Abstract

The embodiment of the invention provides a decoding method, an encoding method, a decoder and an encoder. The embodiment of the invention relates to the technical field of image video encoding and decoding. In the decoding method provided by the embodiment of the invention, first transformation is carried out on a first transformation coefficient of a current block through a transformation set corresponding to a first intra-frame prediction mode; and designing the first intra prediction mode to include any one of the following: using an intra prediction mode derived by the DIMD mode for a prediction block of the current block, and using an intra prediction mode derived by the DIMD mode for an output vector for predicting an optimal MIP mode of the current block, by using an intra prediction mode derived from a DIMD mode and an intra prediction mode derived from a TIMD mode for a reconstructed sample in a first template region adjacent to the current block, the decompression performance of the current block can be improved.

Description

Decoding method, encoding method, decoder and encoder Technical Field

[0001] The embodiments of the present application relate to the technical field of image and video coding and decoding, and more specifically, to a decoding method, an encoding method, a decoder, and an encoder. Background Art

[0002] Digital video compression technology primarily compresses large amounts of digital video data for easier transmission and storage. With the surge in Internet video usage and increasing demand for higher-quality video, while existing digital video compression standards are capable of implementing video decompression technology, there is still a need for better digital video decompression technologies to improve compression efficiency.

[0003] Summary of the Invention

[0004] The embodiments of the present application provide a decoding method, an encoding method, a decoder, and an encoder, which can improve compression efficiency.

[0005] In a first aspect, an embodiment of the present application provides a decoding method, comprising:

[0006] Parsing the code stream of the current sequence to obtain the first transform coefficient of the current block;

[0007] determining a first intra prediction mode;

[0008] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0009] Performing a first transform on the first transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a second transform coefficient of the current block;

[0010] Performing a second transform on the second transform coefficient to obtain a residual block of the current block;

[0011] A reconstructed block of the current block is determined based on the prediction block of the current block and the residual block of the current block.

[0012] In a second aspect, an embodiment of the present application provides an encoding method, including:

[0013] Get the residual block of the current block in the current sequence;

[0014] Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;

[0015] determining a first intra prediction mode;

[0016] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0017] Performing a fourth transform on the third transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a fourth transform coefficient of the current block;

[0018] The fourth transform coefficients are encoded.

[0019] In a third aspect, an embodiment of the present application provides a decoder, including:

[0020] A parsing unit, configured to parse a code stream of a current sequence to obtain a first transform coefficient of a current block;

[0021] Transformation unit, used to:

[0022] determining a first intra prediction mode;

[0023] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0024] Performing a first transform on the first transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a second transform coefficient of the current block;

[0025] Performing a second transform on the second transform coefficient to obtain a residual block of the current block;

[0026] A reconstruction unit is configured to determine a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.

[0027] In a fourth aspect, an embodiment of the present application provides an encoder, including:

[0028] The residual unit is used to obtain the residual block of the current block in the current sequence;

[0029] Transformation unit, used to:

[0030] Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;

[0031] determining a first intra prediction mode;

[0032] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0033] Performing a fourth transform on the third transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a fourth transform coefficient of the current block;

[0034] A coding unit is used to encode the fourth transformation coefficient.

[0035] In a fifth aspect, an embodiment of the present application provides a decoder, including:

[0036] a processor adapted to implement computer instructions; and,

[0037] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the decoding method in the above-mentioned first aspect or its various implementations.

[0038] In one implementation, there are one or more processors and one or more memories.

[0039] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.

[0040] In a sixth aspect, an embodiment of the present application provides an encoder, including:

[0041] a processor adapted to implement computer instructions; and,

[0042] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the encoding method in the above-mentioned second aspect or its various implementations.

[0043] In one implementation, there are one or more processors and one or more memories.

[0044] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.

[0045] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the decoding method involved in the first aspect or the encoding method involved in the second aspect.

[0046] In an eighth aspect, an embodiment of the present application provides a code stream, which is the code stream involved in the first aspect or the code stream involved in the second aspect.

[0047] Based on the above technical solution, by introducing a first intra-frame prediction mode and performing a first transform on the first transform coefficient of the current block based on the transform set corresponding to the first intra-frame prediction mode, the decompression performance of the current block can be improved. In particular, when the decoder predicts the current block using a non-traditional intra-frame prediction mode, it can avoid directly using the transform set corresponding to the planar mode for the first transform. The transform set corresponding to the first intra-frame prediction mode can, to a certain extent, reflect the texture direction of the current block, thereby improving the decompression performance of the current block. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG1 is a schematic block diagram of a coding framework provided in an embodiment of the present application.

[0049] FIG2 is a schematic diagram of the MIP mode provided in an embodiment of the present application.

[0050] FIG3 is a schematic diagram of a DIMD-based prediction model provided in an embodiment of the present application.

[0051] FIG4 is a schematic diagram of deriving a prediction block based on DIMD according to an embodiment of the present application.

[0052] FIG5 is a schematic diagram of a template used by the TIMD provided in an embodiment of the present application.

[0053] FIG6 is an example of LFNST provided by an embodiment of the present application.

[0054] FIG7 is an example of a transformation set of LFNST provided in an embodiment of the present application.

[0055] FIG8 is a schematic block diagram of a decoding framework provided in an embodiment of the present application.

[0056] FIG9 is a schematic flowchart of a decoding method provided in an embodiment of the present application.

[0057] FIG10 is a schematic flowchart of the encoding method provided in an embodiment of the present application.

[0058] FIG11 is a schematic block diagram of a decoder provided in an embodiment of the present application.

[0059] FIG12 is a schematic block diagram of an encoder provided in an embodiment of the present application.

[0060] FIG13 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0062] The solution provided in the embodiment of the present application can be applied to the field of digital video coding technology, for example, including but not limited to: image coding and decoding field, video coding and decoding field, hardware video coding and decoding field, dedicated circuit video coding and decoding field and real-time video coding and decoding field. In addition, the solution provided in the embodiment of the present application can be combined with the audio and video coding standard (Audio Video Coding Standard, AVS), the second generation AVS standard (AVS2) or the third generation AVS standard (AVS3). For example, including but not limited to: H.264 / Audio Video Coding (AVC) standard, H.265 / High Efficiency Video Coding (HEVC) standard and H.266 / Versatile Video Coding (VVC) standard. In addition, the solution provided in the embodiment of the present application can be used for lossy compression of images, and can also be used for lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0063] Video coding standards all use a block-based hybrid coding framework. Specifically, each image in a video is divided into square largest coding units (LCUs) or coding tree units (CTUs) of the same size (e.g., 128x128, 64x64, etc.). Each LCU or CTU can be divided into rectangular coding units (CUs) according to a rule. Coding units may also be divided into prediction units (PUs), transform units (TUs), etc. The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra prediction and inter prediction. Inter prediction includes motion estimation and motion compensation. Because there is a strong correlation between adjacent pixels in a video image, intra prediction is used in video coding and decoding technology to eliminate spatial redundancy between adjacent pixels. Intra-frame prediction only references information from the same image to predict pixel information within the current block. Due to the strong similarities between adjacent images in a video, inter-frame prediction is used in video coding and decoding to eliminate temporal redundancy between adjacent images, thereby improving coding efficiency. Inter-frame prediction can reference image information from different frames and use motion estimation to search for the motion vector information that best matches the current block. Transformation converts the predicted image block to the frequency domain, redistributing the energy. Combined with quantization, it removes information that is insensitive to the human eye, eliminating visual redundancy. Entropy coding can eliminate character redundancy based on the current context model and the probabilistic information of the binary bitstream.

[0064] During digital video encoding, the encoder may first read a black-and-white or color image from the original video sequence and then encode the black-and-white or color image. A black-and-white image may include pixels of the luminance component, while a color image may include pixels of the chrominance component. Optionally, the color image may also include pixels of the luminance component. The color format of the original video sequence may be a luminance-chrominance (YCbCr, YUV) format or a red-green-blue (RGB) format. Specifically, after reading the black-and-white or color image, the encoder divides it into blocks and uses intra-frame prediction or inter-frame prediction on the current block to generate a prediction block for the current block. The prediction block is subtracted from the original block of the current block to obtain a residual block. The residual block is transformed and quantized to obtain a quantization coefficient matrix. The quantization coefficient matrix is ​​entropy encoded and output to the bitstream. During digital video decoding, the decoder uses intra-frame prediction or inter-frame prediction on the current block to generate a prediction block for the current block. Furthermore, the decoder decodes the bitstream to obtain a quantization coefficient matrix, dequantizes and inversely transforms the quantization coefficient matrix to obtain a residual block. The prediction block and the residual block are added together to obtain a reconstructed block. The reconstructed blocks can be used to compose a reconstructed image. The decoding end performs loop filtering on the reconstructed image based on an image or a block to obtain a decoded image.

[0065] The current block may be a current coding unit (CU) or a current prediction unit (PU), etc.

[0066] It should be noted that the encoder also requires similar operations as the decoder to obtain a decoded image. The decoded image can serve as a reference image for inter-frame prediction of subsequent images. The block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode information or parameter information determined by the encoder need to be output to the bitstream if necessary. The decoder determines the same block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode information or parameter information as the encoder by parsing and analyzing existing information, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical. The decoded image obtained by the encoder is also commonly called a reconstructed image. During prediction, the current block can be divided into prediction units, and during transform, the current block can be divided into transform units. The division of prediction units and transform units can be the same or different. Of course, the above is only the basic process of a video codec within a block-based hybrid coding framework. With technological advancement, some modules of the framework or some steps of the process may be optimized. This application applies to the basic process of a video codec within this block-based hybrid coding framework.

[0067] To facilitate understanding, the coding framework provided in this application is briefly introduced first.

[0068] FIG1 is a schematic block diagram of a coding framework 100 provided in an embodiment of the present application.

[0069] As shown in FIG1 , the coding framework 100 may include an intra-frame prediction unit 180, an inter-frame prediction unit 170, a residual unit 110, a transform and quantization unit 120, an entropy coding unit 130, an inverse transform and inverse quantization unit 140, and a loop filter unit 150. Optionally, the coding framework 100 may further include a decoded image buffer unit 160. The coding framework 100 may also be referred to as a hybrid framework coding mode.

[0070] The intra-frame prediction unit 180 or the inter-frame prediction unit 170 may predict the image block to be encoded and output a prediction block. The residual unit 110 may calculate a residual block based on the prediction block and the image block to be encoded, i.e., the difference between the prediction block and the image block to be encoded. The transform and quantization unit 120 is configured to perform operations such as transform and quantization on the residual block to remove information that is insensitive to the human eye, thereby eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform and quantization unit 120 may be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform and quantization unit 120 may be referred to as a frequency residual block or a frequency-domain residual block. After receiving the transform and quantization coefficients output by the transform and quantization unit 120, the entropy coding unit 130 may output a bitstream based on the transform and quantization coefficients. For example, the entropy coding unit 130 may eliminate character redundancy based on a target context model and probability information of the binary bitstream. For example, the entropy coding unit 130 may be configured for context-based adaptive binary arithmetic entropy coding (CABAC). The entropy coding unit 130 may also be referred to as a header information coding unit. Optionally, in this application, the image block to be encoded may also be referred to as an original image block or a target image block, the prediction block may also be referred to as a prediction image block or an image prediction block, or may also be referred to as a prediction signal or prediction information, and the reconstruction block may also be referred to as a reconstructed image block or an image reconstruction block, or may also be referred to as a reconstruction signal or reconstruction information. In addition, for the encoding end, the image block to be encoded may also be referred to as a coding block or a coding image block, and for the decoding end, the image block to be encoded may also be referred to as a decoding block or a decoding image block. The image block to be encoded may be a CTU or a CU.

[0071] The coding framework 100 calculates the residual between the prediction block and the image block to be coded, generates a residual block, and transmits the residual block to the decoder through processes such as transformation and quantization. Correspondingly, the decoder receives and decodes the bitstream, then performs inverse transformation and inverse quantization to obtain a residual block. The predicted block, obtained by the decoder, is then superimposed on the residual block to create a reconstructed block.

[0072] It should be noted that the inverse transform and inverse quantization unit 140, the loop filter unit 150, and the decoded image buffer unit 160 in the coding framework 100 can be used to form a decoder. This means that the intra-frame prediction unit 180 or the inter-frame prediction unit 170 can predict the image block to be coded based on the existing reconstructed block, thereby ensuring that the encoder and decoder have a consistent understanding of the reference image. In other words, the encoder can replicate the decoder's processing loop, thereby generating the same prediction as the decoder. Specifically, the quantized transform coefficients are inversely transformed and inversely quantized by the inverse transform and inverse quantization unit 140 to replicate the approximate residual block at the decoder. This approximate residual block, after adding the prediction block, can pass through the loop filter unit 150 to smooth out block artifacts caused by block-based processing and quantization. The image block output by the loop filter unit 150 can be stored in the decoded image buffer unit 160 for use in subsequent image prediction.

[0073] It should be understood that FIG1 is merely an example of the present application and should not be construed as limiting the present application.

[0074] For example, the loop filtering unit 150 in the coding framework 100 may include a deblocking filter (DBF) and a sample adaptive offset (SAO) filter. The function of DBF is to remove the blocking effect, and the function of SAO is to remove the ringing effect. In other embodiments of the present application, the coding framework 100 may adopt a neural network-based loop filtering algorithm to improve the compression efficiency of the video. In other words, the coding framework 100 may be a video coding hybrid framework based on a deep learning neural network. In one implementation, a convolutional neural network-based model may be used to calculate the result of pixel filtering on the basis of the deblocking filter and the sample adaptive offset filter. The network structure of the loop filtering unit 150 on the luminance component and the chrominance component may be the same or different. Considering that the luminance component contains more visual information, the luminance component may also be used to guide the filtering of the chrominance component to improve the reconstruction quality of the chrominance component.

[0075] The following describes the intra-frame prediction and inter-frame prediction.

[0076] For inter-frame prediction, inter-frame prediction can refer to image information of different frames and use motion estimation to search for motion vector information that best matches the image block to be encoded to eliminate temporal redundancy; the frames used for inter-frame prediction can be P frames and / or B frames, P frames refer to forward prediction frames, and B frames refer to bidirectional prediction frames.

[0077] Intra-frame prediction only references information from the same image, predicting pixel information within the image block to be encoded to eliminate spatial redundancy. The frame used for intra-frame prediction can be an I-frame. For example, based on the left-to-right, top-to-bottom encoding order, the image block to be encoded can reference the upper left image block, the upper image block, and the left image block as reference information to predict the image block to be encoded. The image block to be encoded then serves as reference information for the next image block. In this way, the entire image can be predicted. If the input digital video is in a color format, such as YUV 4:2:0, each four pixels in each image frame of the digital video are composed of four Y components and two UV components. The encoding framework can encode the Y component (i.e., the luminance block) and the UV component (i.e., the chrominance block) separately. Similarly, the decoding end can also perform decoding according to the format.

[0078] Intra-frame prediction uses both angular and non-angular prediction modes to predict the image block to be coded. This generates a prediction block. Based on the rate-distortion information calculated between the prediction block and the image block to be coded, the optimal prediction mode for the image block to be coded is selected and transmitted to the decoder via the bitstream. The decoder parses the prediction mode, predicts the prediction block for the target decoding block, and then overlays it with the time-domain residual block obtained via the bitstream to reconstruct the block.

[0079] Throughout the development of digital video codec standards, non-angular prediction modes have remained relatively stable, including the mean and planar modes. Angular prediction modes have continued to increase with the evolution of digital video codec standards. For example, the H-series of international digital video coding standards, H.264 / AVC, has only eight angular prediction modes and one non-angular prediction mode; H.265 / HEVC has expanded this to 33 angular prediction modes and two non-angular prediction modes. In H.266 / VVC, intra-frame prediction modes have been further expanded, with a total of 67 traditional prediction modes and a non-traditional prediction mode, matrix-weighted intra-frame prediction (MIP), for luma blocks. These 67 traditional prediction modes include planar and DC modes, and 65 angular prediction modes. Planar modes are typically used for blocks with gradual texture changes, while DC modes, as the name suggests, are typically used for flat areas. Angular prediction modes are typically used for blocks with pronounced angular texture.

[0080] It should be noted that, in this application, the current block used for intra-frame prediction can be a square block or a rectangular block.

[0081] Furthermore, since intra-frame prediction blocks are all square, the probability of using each angle prediction mode is equal. When the length and width of the current block are unequal, the probability of using the reference pixel on the top of a horizontal block (width greater than height) is greater than the probability of using the reference pixel on the left, and the probability of using the reference pixel on the top of a vertical block (height greater than width) is less than the probability of using the reference pixel on the left. When predicting a rectangular block, the traditional angle prediction mode is converted to a wide-angle prediction mode. When the wide-angle prediction mode is used to predict the rectangular block, the prediction angle range of the current block is greater than the prediction angle range when the traditional angle prediction mode is used to predict the rectangular block. Optionally, when using the wide-angle prediction mode, the index of the traditional angle prediction mode can still be used to send a signal. Accordingly, the decoding end can convert the traditional angle prediction mode back to the wide-angle prediction mode after receiving the signal. As a result, the total number of intra-frame prediction modes and the encoding method of the intra-frame mode can remain unchanged.

[0082] Furthermore, the intra-frame prediction mode to be performed can be determined or selected based on the size of the current block; for example, a wide-angle prediction mode can be determined or selected based on the size of the current block to perform intra-frame prediction on the current block; for example, when the current block is a rectangular block (with different sizes in width and height), the wide-angle prediction mode can be used to perform intra-frame prediction on the current block. Among them, the aspect ratio of the current block can be used to determine the angular prediction mode replaced by the wide-angle prediction mode and the angular prediction mode after the replacement. For example, when predicting the current block, any intra-frame prediction mode with an angle that does not exceed the diagonal angle of the current block (from the lower left corner to the upper right corner of the current block) can be selected as the angular prediction mode after the replacement.

[0083] The following is an introduction to other intra-frame prediction modes involved in this application:

[0084] (1) Matrix based Intra Prediction (MIP) mode.

[0085] The MIP mode can also be called the Matrix Weighted Intra Prediction mode. The process involved in the MIP mode can be divided into three main steps, namely the downsampling process, the matrix multiplication process, and the upsampling process. Specifically, the downsampling process first downsamples spatially adjacent reconstructed samples, and then the downsampled sample sequence is used as the input vector of the matrix multiplication process, that is, the output vector of the downsampling process is used as the input vector of the matrix multiplication process, multiplied with a pre-set matrix and added with a bias vector, and the calculated sample vector is output; the output vector of the matrix multiplication process is used as the input vector of the upsampling process, and the final prediction block is obtained by upsampling.

[0086] FIG2 is a schematic diagram of the MIP mode provided in an embodiment of the present application.

[0087] As shown in Figure 2, the MIP mode obtains the upper adjacent down-sampled reconstructed sample vector bdry by averaging the reconstructed samples above the current coding unit during the down-sampling process. top , by averaging the left adjacent reconstructed samples, we get the left adjacent down-sampling reconstructed sample vector bdry left . Get bdry top and bdry left After that, it is used as the input vector bdry of the matrix multiplication process red Specifically, based on bdry red The top row vector bdry top red 、bdry left 、A k ·bdry red +b k Get the sample vector, where A k is a pre-set matrix, b k is a pre-set bias vector, and k is the index of the MIP mode. After obtaining the sample vector, it is upsampled by linear interpolation to obtain a prediction sample block that matches the number of samples in the actual coding unit.

[0088] In other words, to predict a block of width W and height H, MIP requires H reconstructed pixels in the column to the left of the current block and W reconstructed pixels in the row above the current block as input. MIP generates the prediction block using three steps: reference pixel averaging, matrix vector multiplication, and interpolation. The core of MIP is matrix multiplication, which can be thought of as a process of generating a prediction block from input pixels (reference pixels) using a matrix multiplication approach. MIP offers a variety of matrices, and different prediction methods are reflected in different matrices; using different matrices for the same input pixels will produce different results. The reference pixel averaging and interpolation processes are a compromise between performance and complexity. For larger blocks, reference pixel averaging can achieve a similar downsampling effect, allowing the input to fit into a smaller matrix, while interpolation achieves an upsampling effect. This eliminates the need to provide MIP matrices for every block size; instead, matrices of one or a few specific sizes are sufficient. With increasing demands for compression performance and improved hardware capabilities, even more complex MIPs may appear in next-generation standards.

[0089] The MIP mode can be simplified from a neural network. For example, the matrix it uses can be derived from training. Therefore, the MIP mode has strong generalization capabilities and prediction results that traditional prediction modes cannot achieve. The MIP mode can be a model derived from a neural network-based intra-frame prediction model through multiple hardware and software complexity simplifications. Based on a large number of training samples, multiple prediction modes represent a variety of models and parameters, which can effectively cover the texture conditions of natural sequences.

[0090] MIP is somewhat similar to the planar mode, but it is obviously more complex and more flexible than the planar mode.

[0091] It should be noted that the number of MIP modes may be different for coding units of different block sizes. For example, for a coding unit of size 4x4, the MIP mode has 16 prediction modes; for a coding unit of size 8x8, width equal to 4 or height equal to 4, the MIP mode has 8 prediction modes; for coding units of other sizes, the MIP mode has 6 prediction modes. At the same time, the MIP mode has a transposition function. For the prediction mode that meets the current size, the MIP mode can attempt transposition calculation on the encoder side. Therefore, the MIP mode not only requires a usage flag to indicate whether the current coding unit uses the MIP mode, but also, if the current coding unit uses the MIP mode, an additional transposition flag and index flag need to be transmitted to the decoder. The transposition flag can be binarized by the fixed-length coding method (FL) with a length of 1. The index flag is binarized using a truncated binary (TB) encoding method. Taking a 4x4 coding unit as an example, the MIP mode has 16 prediction modes, and the index flag can be a 5- or 6-bit truncated binary identifier.

[0092] (2) Decoder side Intra Mode Derivation (DIMD) mode.

[0093] The main core point of the DIMD mode is that the decoder uses the same method as the encoder to derive the intra-frame prediction mode, thereby avoiding transmitting the intra-frame prediction mode index of the current coding unit in the bitstream and achieving the purpose of saving bit overhead.

[0094] The specific process of DIMD mode can be divided into the following two main steps:

[0095] Step 1: Derive the prediction model.

[0096] FIG3 is a schematic diagram of a DIMD-based prediction model provided in an embodiment of the present application.

[0097] As shown in FIG3(a), DIMD uses pixels in a template within a reconstructed region (reconstructed pixels to the left and above the current block) to derive a prediction mode. For example, the template may include three adjacent rows of reconstructed samples above the current block, three adjacent columns of reconstructed samples to the left, and corresponding adjacent reconstructed samples to the upper left of the current block. Based on this, multiple gradient values ​​corresponding to multiple adjacent reconstructed samples can be determined within the template according to a window (e.g., as shown in FIG3(a) or FIG3(b)). Each gradient value can be used to adapt an intra-frame prediction mode (IPM) that is compatible with its gradient direction. Based on this, the encoder can use the prediction mode adapted to the maximum and second-largest gradient values ​​among the multiple gradient values ​​as the derived prediction mode. For example, as shown in (b) of FIG3 , for a 4×4 block, all adjacent reconstructed samples for which gradient values ​​need to be determined are analyzed and a corresponding gradient histogram is obtained. For example, as shown in (c) of FIG3 , for blocks of other sizes, all adjacent reconstructed samples for which gradient values ​​need to be determined are analyzed and a corresponding gradient histogram is obtained. Finally, the prediction modes corresponding to the largest and second largest gradients in the gradient histogram are used as the derived prediction modes.

[0098] Of course, the gradient histogram in this application is merely an example for determining the derived prediction mode. A variety of simple forms can be used in specific implementations, and this application does not impose any specific limitations on this. Furthermore, this application does not limit the method for calculating the gradient histogram. For example, the Sobel operator or other methods can be used to calculate the gradient histogram. Furthermore, in other alternative embodiments, the gradient values ​​referred to in this application can also be equivalently replaced with gradient amplitude values, and this application does not impose any specific limitations on this.

[0099] Step 2: Derive the prediction block.

[0100] FIG4 is a schematic diagram of deriving a prediction block based on DIMD according to an embodiment of the present application.

[0101] As shown in Figure 4, the encoder can weight the prediction values ​​of the three intra-frame prediction modes (planar mode and two intra-frame prediction modes derived based on DIMD). The codec uses the same prediction block derivation method to obtain the prediction block of the current block. Assuming that the prediction mode corresponding to the maximum gradient value is prediction mode 1 and the prediction mode corresponding to the second largest gradient value is prediction mode 2, the encoder determines the following two conditions:

[0102] 1. The gradient of prediction mode 2 is not 0;

[0103] 2. Prediction mode 1 and prediction mode 2 are neither planar mode nor DC prediction mode.

[0104] If neither of the above two conditions holds, only prediction mode 1 is used to calculate the predicted sample values ​​for the current block, applying the normal prediction process to prediction mode 1. Otherwise, if both conditions hold, a weighted average is used to derive the prediction block for the current block. Specifically, the planar mode accounts for 1 / 3 of the weight, and the remaining 2 / 3 is the combined weight of prediction modes 1 and 2. For example, the weight for prediction mode 1 is calculated by dividing the gradient amplitude of prediction mode 1 by the sum of the gradient amplitudes of prediction mode 1 and prediction mode 2, and the weight for prediction mode 2 is calculated by dividing the gradient amplitude of prediction mode 2 by the sum of the gradient amplitudes of prediction mode 1 and prediction mode 2. The prediction blocks obtained based on the three prediction modes—prediction blocks 1, 2, and 3 obtained based on planar mode, prediction mode 1, and prediction mode 2, respectively—are weighted averaged to obtain the prediction block for the current coding unit. The decoder uses the same steps to obtain the prediction block.

[0105] In other words, the specific weight calculation in step 2 above is as follows:

[0106] Weight(PLANAR)=1 / 3;

[0107] Weight(mode1)=2 / 3*(amp1 / (amp1+amp2));

[0108] Weight(mode2)=1-Weight(PLANAR)-Weight(mode1);

[0109] Wherein, mode1 and mode2 represent prediction mode 1 and prediction mode 2 respectively, amp1 and amp2 represent the gradient amplitude value of prediction mode 1 and the gradient amplitude value of prediction mode 2 respectively. The DIMD mode requires a flag to be transmitted to the decoder, which is used to indicate whether the current coding unit uses the DIMD mode.

[0110] Of course, the above-mentioned weighted averaging method is only an example of the present application and should not be understood as a limitation of the present application.

[0111] In summary, DIMD uses gradient analysis of reconstructed pixels to select the intra prediction mode, and can weight two intra prediction modes plus the planar mode based on the analysis results. The advantage of DIMD is that if DIMD mode is selected for the current block, the bitstream no longer needs to indicate the specific intra prediction mode used. Instead, the decoder automatically derives the selected intra prediction mode through the above process, which reduces overhead to a certain extent.

[0112] (3) Template based Intra Mode Derivation (TIMD) mode.

[0113] The technical principle of the TIMD mode is similar to that of the aforementioned DIMD mode. Both utilize the same codec operation to derive the prediction mode to save transmission mode index overhead. The TIMD mode can be understood as two main parts. First, the cost information of each prediction mode is calculated based on the template. The prediction mode corresponding to the minimum cost and the second minimum cost will be selected. The prediction mode corresponding to the minimum cost is recorded as prediction mode 1, and the prediction mode corresponding to the second minimum cost is recorded as prediction mode 2. If the ratio of the value of the second minimum cost (costMode2) to the value of the minimum cost (costMode1) meets the preset conditions, such as costMode2<2*costMode1, the prediction blocks corresponding to prediction mode 1 and prediction mode 2 can be weightedly fused according to the corresponding weights of prediction mode 1 and prediction mode 2 to obtain the final prediction block.

[0114] Exemplarily, the weights corresponding to prediction mode 1 and prediction mode 2 are determined according to the following method:

[0115] weight1=costMode2 / (costMode1+costMode2);

[0116] weight2=1-weight1;

[0117] Where weight1 is the weight of the prediction block corresponding to prediction mode 1, and weight2 is the weight of the prediction block corresponding to prediction mode 2. However, if the ratio of the value of the second smallest cost, costMode2, to the value of the smallest cost, costMode1, does not meet the preset conditions, weighted fusion between the prediction blocks is not performed, and the prediction block corresponding to prediction mode 1 is the TIMD prediction block.

[0118] It should be noted that when using TIMD mode for intra prediction of the current block, if the reconstructed sample template of the current block does not contain available adjacent reconstructed samples, TIMD mode selects planar mode for intra prediction of the current block, that is, no unweighted fusion is performed. Similar to DIMD mode, TIMD mode requires a flag to be transmitted to the decoder to indicate whether the current coding unit uses TIMD mode.

[0119] The encoder or decoder calculates the cost information of each prediction mode mainly as follows: based on the reconstructed samples adjacent to the upper or left side of the template area, intra-frame prediction is performed on the samples within the template area. The prediction process is the same as the original intra-frame prediction mode. For example, when using the DC mode to perform intra-frame prediction on the samples within the template area, the mean of the entire coding unit is calculated. For example, when using the angle prediction mode to perform intra-frame prediction on the samples within the template area, the corresponding interpolation filter is selected according to the mode and the predicted samples are interpolated according to the rules. At this time, the distortion between the predicted samples and the reconstructed samples in the template area can be calculated based on the predicted samples and the reconstructed samples in the area, which is the cost information of the current prediction mode.

[0120] FIG5 is a schematic diagram of a template used by the TIMD provided in an embodiment of the present application.

[0121] As shown in Figure 5, if the current block is a coding unit with a width equal to M and a height equal to N, the codec can calculate the template of the current block based on the reference template (Reference of template) of the current block selected in the coding unit with a width equal to 2(M+L1)+1 and a height equal to 2(N+L2)+1. If the template of the current block does not contain available adjacent reconstructed samples, the TIMD mode selects the plane mode to perform intra-frame prediction on the current block. For example, the template of the current block can be the samples adjacent to the left and top sides of the current CU in Figure 5, that is, there are no available reconstructed samples in the diagonal filling area. In other words, if there are no available adjacent reconstructed samples in the diagonal filling area, the TIMD mode selects the plane mode to perform intra-frame prediction on the current block.

[0122] It's worth noting that, except for edge cases, when encoding and decoding the current block, reconstructed values ​​can theoretically be obtained for the left and top sides of the current block. This means that the template of the current block contains available adjacent reconstructed samples. In practice, the decoder can use a specific intra prediction mode to predict on the template and compare the predicted value with the reconstructed value to obtain the cost of that intra prediction mode on the template. This can be measured using the sum of absolute differences (SAD), the sum of absolute transformed differences (SATD), or the sum of squares for error (SSE). Because the template and the current block are adjacent, the reconstructed samples within the template are correlated with the pixels within the current block. Therefore, the performance of a prediction mode on the template can be used to estimate its performance on the current block. TIMD predicts several candidate intra prediction modes on the template, obtains their costs on the template, and replaces the one or two intra prediction modes with the lowest costs as the intra prediction value for the current block. If the cost difference between the two intra prediction modes on the template is not large, taking a weighted average of the prediction values ​​of the two intra prediction modes can improve the compression performance. Optionally, the weights of the prediction values ​​of the two prediction modes are related to the above-mentioned cost, for example, the weights are inversely proportional to the cost.

[0123] In summary, TIMD uses the prediction performance of intra-frame prediction modes on a template to select the appropriate intra-frame prediction mode, and can weight the two intra-frame prediction modes based on their cost on the template. The advantage of TIMD is that if TIMD mode is selected for the current block, the bitstream no longer needs to indicate the specific intra-frame prediction mode used. Instead, the decoder automatically derives the selected intra-frame prediction mode through the above process, which reduces overhead to a certain extent.

[0124] Through the above brief introduction to several intra-frame prediction modes, it is not difficult to find that the technical principles of the DIMD mode are similar to those of the TIMD mode. Both use the decoder to perform the same operations as the encoder to infer the prediction mode of the current coding unit. This prediction mode can save the transmission of the prediction mode index when the complexity is acceptable, thus saving overhead and improving compression efficiency. However, due to the limitations of the reference information and the fact that there is not much improvement in the prediction quality, the DIMD mode and the TIMD mode are more effective in areas with consistent texture characteristics over a large area. If the texture changes slightly or the template area cannot be covered, the prediction effect of this prediction mode is poor.

[0125] In addition, regardless of the DIMD mode or the TIMD mode, the prediction blocks obtained based on multiple traditional prediction modes are fused or weighted. The fusion of prediction blocks can generate effects that cannot be achieved by a single prediction mode. Although the DIMD mode introduces the planar mode as an additional weighted prediction mode to increase the spatial correlation between adjacent reconstructed samples and prediction samples, thereby improving the prediction effect of intra-frame prediction, due to the relatively simple prediction principle of the planar mode, for some prediction blocks with obvious difference between the upper right corner and the lower left corner, using the planar mode as an additional weighted prediction mode may have a counterproductive effect.

[0126] The following describes the content related to the transformation of the residual block.

[0127] During encoding, a prediction is first performed on the current block. This prediction exploits spatial or temporal correlation to produce an image identical or similar to the current block. While it's possible for the predicted block to be identical to the current block for a given block, it's difficult to guarantee this for all blocks in a video. This is particularly true for natural video, or video captured by a camera, where complex textures and noise often result in the predicted block being similar to the current block, but with some differences. Furthermore, irregular video motion, distortion, occlusion, and brightness changes make it difficult to fully predict the current block. Therefore, a hybrid coding framework subtracts the predicted image from the original image of the current block to produce a residual image, or, in other words, the current block minus the predicted block to produce a residual block. This residual block is typically much simpler than the original image, so prediction can significantly improve compression efficiency. The residual block is not directly encoded, but is usually first transformed. This transform transforms the residual image from the spatial domain to the frequency domain, removing any correlation. After the residual image is transformed into the frequency domain, the non-zero coefficients are mostly concentrated in the upper left corner, as most of the energy is concentrated in the low-frequency area. Quantization is then used to further compress the residual block. Optionally, a larger quantization step size can be used in the high-frequency area, as the human eye is less sensitive to high frequencies.

[0128] Image transformation technology transforms an original image to enable it to be represented using an orthogonal function or matrix. This transformation is linearly reversible in two dimensions. The original image is generally referred to as a spatial domain image, and the transformed image is called a transform domain image (also called a frequency domain image). The transform domain image can be reverse-transformed back to a spatial domain image. Image transformation not only more effectively reflects the image's characteristics, but also concentrates energy on a small amount of data, facilitating image storage, transmission, and processing.

[0129] The following describes the transformation-related technologies involved in this application.

[0130] In the field of image and video coding, after obtaining the residual block, the encoder can transform the residual block. Transformation methods include primary transform and secondary transform. Primary transform methods include, but are not limited to, discrete cosine transform (DCT) and discrete sine transform (DST). DCTs that can be used in video coding and decoding include, but are not limited to, DCT2 and DCT8; DSTs that can be used in video coding and decoding include, but are not limited to, DST7. Because DCT has a strong energy concentration characteristic, after the original image is transformed by DCT, only certain areas (such as the upper left corner) have non-zero coefficients. Of course, in video coding and decoding, images are divided into blocks for processing, so the transformation is also performed on a block basis.

[0131] It is worth noting that since images are all two-dimensional, the computational complexity and memory overhead of directly performing a two-dimensional transformation are unacceptable to the hardware conditions. Therefore, the above-mentioned DCT2, DCT8, and DST7 transformations are usually split into one-dimensional transformations in the horizontal and vertical directions, that is, they are performed in two steps. For example, the horizontal transformation is performed first and then the vertical transformation, or the vertical transformation is performed first and then the horizontal transformation. The above-mentioned transformation method is more effective for horizontal and vertical textures, but the effect is less effective for diagonal textures. Since horizontal and vertical textures are the most common, the above-mentioned transformation method is very useful for improving compression efficiency.

[0132] The encoder can perform a secondary transform based on the primary transform to further improve the compression efficiency.

[0133] The primary transform (also known as the base transform) can be used to process horizontal and vertical textures. For example, primary transforms include, but are not limited to, the aforementioned DCT2, DCT8, and DST7 transforms. The secondary transform (LFNST) is used to process diagonal textures. At the encoder, the secondary transform is applied after the primary transform and before quantization. At the decoder, the secondary transform is applied after inverse quantization and before the inverse of the primary transform.

[0134] FIG6 is an example of LFNST provided by an embodiment of the present application.

[0135] As shown in Figure 6, at the encoder, LFNST performs a secondary transform on the low-frequency coefficients in the upper left corner after the base transform. The primary transform decorrelates the image, concentrating energy in the upper left corner. The secondary transform further decorrelates the low-frequency coefficients of the primary transform. At the encoder, when 16 coefficients are input to a 4x4 LFNST, 8 coefficients are output; when 64 coefficients are input to an 8x8 LFNST, 16 coefficients are output. At the decoder, when 8 coefficients are input to a 4x4 inverse LFNST, 16 coefficients are output; when 16 coefficients are input to an 8x8 inverse LFNST, 64 coefficients are output.

[0136] When the encoder performs a secondary transform on the current block in the current image, it can use a transform kernel in the selected transform set to transform the residual block of the current block. Taking the secondary transform as LFNST as an example, the transform kernel can refer to a set of transform kernels for transforming a certain oblique texture, or the transform set can include a set of transform kernels for transforming certain similar oblique textures. Of course, in other alternative embodiments, the transform kernel can also be referred to as or equivalently replaced by terms with similar or identical meanings such as transform matrix, transform kernel type or basis function, and the transform set can also be referred to as or equivalently replaced by terms with similar or identical meanings such as transform matrix group, transform kernel type group or basis function group, and the present application does not make specific limitations on this.

[0137] FIG7 is an example of a transformation set of LFNST provided in an embodiment of the present application.

[0138] As shown in Figures 7(a) to 7(d), LFNST can have four transform sets, and the transform kernels in the same transform set have similar oblique textures. For example, the transform set shown in Figure 7(a) can be a transform set with an index of 0, the transform set shown in Figure 7(b) can be a transform set with an index of 1, the transform set shown in Figure 7(c) can be a transform set with an index of 2, and the transform set shown in Figure 7(d) can be a transform set with an index of 3.

[0139] The following describes the application of LFNST to intra-coded blocks.

[0140] Intra-frame prediction uses the reconstructed pixels around the current block as reference to predict the current block. Since current videos are encoded from left to right and from top to bottom, the reference pixels that can be used by the current block are usually on the left and top. Angular prediction tiles the reference pixels to the current block as predicted values ​​at a specified angle, which means that the predicted block will have obvious directional texture, and the residual of the current block after angular prediction will also statistically show obvious angular characteristics. Therefore, the transform set selected by LFNST can be bound to the intra-frame prediction mode. That is, after determining the intra-frame prediction mode, LFNST can use a transform set (Transform set) whose texture direction is adapted to the angular characteristics of the intra-frame prediction mode to save bit overhead.

[0141] For example, it is assumed that LFNST has a total of 4 transform sets, each of which has 2 transform kernels. Table 1 shows the correspondence between intra prediction modes and transform sets.

[0142] Table 1

[0143]

[0144] As shown in Table 1, intra prediction modes 0 to 81 may be associated with the indices of four transform sets.

[0145] It is worth noting that the cross-component prediction modes used for chroma intra prediction are 81 to 83, while these modes are not available for luma intra prediction. The transform set of LFNST can be transposed to handle more angles with one transform set. For example, intra prediction modes 13 to 23 and intra prediction modes 45 to 55 both correspond to transform set 2. However, intra prediction modes 13 to 23 are obviously close to horizontal modes, while intra prediction modes 45 to 55 are obviously close to vertical modes. The transforms corresponding to intra prediction modes 45 to 55 are adapted by transposition.

[0146] In practice, since LFNST has four transform sets, the encoder can determine which transform set to use based on the intra prediction mode used by the current block, and then determine the transform kernel to use within that set. This effectively leverages the correlation between the intra prediction mode and the LFNST transform set, reducing the amount of time required to select the LFNST transform set in the bitstream. Whether LFNST is used for the current block, and if so, whether the first or second transform kernel within a transform set is used, can be determined based on the bitstream and certain conditions.

[0147] Of course, considering that there are 67 common intra-frame prediction modes and LFNST only has four transform sets, multiple similar angle prediction modes can only correspond to one transform set. This is a trade-off between performance and complexity, as each transform set requires storage space to store the coefficients of the transform kernels in the transform set. As the requirements for compression efficiency increase and hardware capabilities improve, LFNST can also be designed to be more complex, such as using larger transform sets, more transform sets, and more transform kernels per transform set.

[0148] Exemplarily, Table 2 shows another correspondence between intra prediction modes and transform sets.

[0149] Table 2

[0150] Intra prediction mode -14-13-12-11-10-9-8-7-6-5-4-3-2-1 Transform set index 2222222222222 Intra prediction mode 012345678910111213 Transform set index 012345678910111213 Intra prediction mode 1415161718192021222324252627 Transform set index 1415161718192021222324252627 Intra prediction mode 28293031323334353 63738394041 Transform set index 2829303132333433323130292827 Intra prediction mode 4243444546474849505152535455 Transform set index 2625242322212019181716151413 Intra prediction mode 5657585960616263646566676869 Transform set index 12111098765432222 Intra prediction mode 7071727374757677787980 Transform set index 22222222222

[0151] As shown in Table 2, 35 transform sets are used, each with three transform kernels. The correspondence between transform sets and intra prediction modes can be implemented as follows: For intra prediction modes 0-34, the positive correspondence is to transform sets 0-34, that is, the larger the prediction mode number, the larger the transform set index; for intra prediction modes 35-67, due to transposition, the negative correspondence is to transform sets 2-33, that is, the larger the prediction mode number, the smaller the transform set index; for the remaining prediction modes, they can all be uniformly mapped to the transform set with an index of 2. In other words, if transposition is not considered, each intra prediction mode corresponds to a transform set. According to this design, the residual corresponding to each intra prediction mode can obtain a more suitable transform set, and compression performance will also be improved.

[0152] Of course, the wide-angle prediction mode can theoretically be one-to-one with the transform set, but this design is less cost-effective and will not be described in detail in this application. It should be noted that LFNST is only an example of a secondary transform and should not be construed as a limitation on secondary transforms. For example, LFNST is a non-separable secondary transform. In other alternative embodiments, a separable secondary transform can also be used to improve the compression efficiency of the residual error of the oblique texture. This application does not specifically limit this.

[0153] FIG8 is a schematic block diagram of a decoding framework 200 provided in an embodiment of the present application.

[0154] As shown in FIG8 , the decoding framework 200 may include an entropy decoding unit 210, an inverse transform and dequantization unit 220, a residual unit 230, an intra-frame prediction unit 240, an inter-frame prediction unit 250, a loop filter unit 260, and a decoded image buffer unit 270. The entropy decoding unit 210 receives and parses the bitstream to obtain a prediction block and a frequency domain residual block. The frequency domain residual block is inversely transformed and dequantized by the inverse transform and dequantization unit 220 to obtain a time domain residual block. The residual unit 230 superimposes the prediction block obtained by the intra-frame prediction unit 240 or the inter-frame prediction unit 250 onto the time domain residual block that has been inversely transformed and dequantized by the inverse transform and dequantization unit 220 to obtain a reconstructed block.

[0155] FIG9 is a schematic flow chart of a decoding method 300 provided in an embodiment of the present application. It should be understood that the decoding method 300 can be performed by a decoder. For example, the decoding method 300 can be performed by the decoding framework 200 shown in FIG6 . For ease of description, the following description uses a decoder as an example.

[0156] As shown in FIG9 , the decoding method 300 may include:

[0157] S310, parsing the code stream of the current sequence to obtain the first transform coefficient of the current block;

[0158] S320, determining a first intra prediction mode;

[0159] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0160] S330, performing a first transform on the first transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a second transform coefficient of the current block;

[0161] S340: Perform a second transform on the second transform coefficient to obtain a residual block of the current block;

[0162] S350 : Determine a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.

[0163] Exemplarily, when the decoder uses the intra-frame prediction mode derived from the DIMD mode for the reconstructed samples within the first template area, it may first calculate the gradient values ​​of the reconstructed samples within the first template area, and then determine the intra-frame prediction mode that matches the gradient direction of the reconstructed sample with the largest gradient value among the reconstructed samples within the first template area as the intra-frame prediction mode derived using the DIMD mode. In other words, the decoder may calculate the gradient value corresponding to each intra-frame prediction mode by traversing the intra-frame prediction modes based on the reconstructed samples within the first template area, and determine the intra-frame prediction mode with the largest gradient value as the intra-frame prediction mode derived using the DIMD mode.

[0164] Exemplarily, when the decoder uses the intra-frame prediction mode derived from the DIMD mode for the prediction block of the current block (or the output vector of the optimal MIP mode), it can first calculate the gradient value of the prediction sample within the prediction block of the current block (or the output vector of the optimal MIP mode), and then determine the intra-frame prediction mode that matches the gradient direction of the prediction sample with the largest gradient value among the prediction samples within the prediction block of the current block (or the output vector of the optimal MIP mode) as the intra-frame prediction mode derived using the DIMD mode. In other words, the decoder can calculate the gradient value corresponding to each intra-frame prediction mode by traversing the intra-frame prediction modes based on the prediction samples within the prediction block of the current block (or the output vector of the optimal MIP mode), and determine the intra-frame prediction mode with the largest gradient value as the intra-frame prediction mode derived using the DIMD mode.

[0165] Exemplarily, the first transformation is used to process texture in the current block along an oblique direction.

[0166] Exemplarily, the second transformation is used to process textures in the horizontal direction and textures in the vertical direction in the current block.

[0167] It should be understood that the first transform is the inverse transform of the secondary transform at the encoding end, and the second transform is the inverse transform of the base transform at the encoding end. For example, the first transform may be an inverse (inverse) LFNST, and the second transform may be an inverse (inverse) DCT2 type, an inverse (inverse) DCT8 type, or an inverse (inverse) DST7 type, etc.

[0168] Of course, the adaptation method of TMMIP technology and LFNST is also applicable to other secondary transformation methods. For example, LFNST is an inseparable secondary transformation. In other alternative embodiments, TMMIP technology can also be applied to separable secondary transformations, which is not specifically limited in this application.

[0169] It is worth noting that when the encoder or decoder predicts the current block, it may use the transform set corresponding to the planar mode for LFNST. The reason is that the transform kernel used by LFNST is obtained by deep learning training of the data set of the traditional intra prediction mode. Therefore, in the normal intra prediction process, the transform kernel used by LFNST is usually also the transform kernel selected from the transform set of LFNST corresponding to the traditional intra prediction mode. However, the encoder or decoder may use a non-traditional intra prediction mode to predict the current block. At this time, considering that the planar mode is usually used to process some blocks with gradient textures, and LFNST is used to process oblique textures, the texture information of the predicted block output by the planar mode and the texture information of the planar mode in the traditional intra prediction mode are usually treated as a type of texture. That is, when the encoder or decoder uses a non-traditional intra prediction mode to predict the current block, the transform set corresponding to the planar mode is used for LFNST. For example, when the encoder uses the MIP mode to predict the current block, the transform set corresponding to the planar mode is used for LFNST. However, since the meaning of the MIP mode is different from that of the traditional intra-frame prediction mode, that is, the traditional intra-frame prediction mode has obvious directionality, while the MIP mode is only an index of the matrix coefficient, therefore, although the planar mode is used to process some blocks with gradient textures, it does not necessarily conform to the texture information of the current block, that is, the texture direction of the transform set used by LFNST does not necessarily conform to the texture direction of the current block, which reduces the decompression performance of the current block.

[0170] In view of this, embodiments of the present application can further improve the decompression performance of the current block by introducing a first intra-frame prediction mode and performing a first transform on the first transform coefficients of the current block based on the transform set corresponding to the first intra-frame prediction mode. In particular, when the decoder uses a non-traditional intra-frame prediction mode to predict the current block, it can avoid directly using the transform set corresponding to the planar mode for the first transform. The transform set corresponding to the first intra-frame prediction mode can, to a certain extent, reflect the texture direction of the current block, thereby improving the decompression performance of the current block.

[0171] The beneficial effects of the solution provided in this application are described below in conjunction with the test results in Tables 1 to 3.

[0172] Among them, Table 1 is the result of testing on a test sequence when the optimal MIP mode and the suboptimal MIP mode are used to perform weighted prediction on the current block, and the first intra-frame prediction mode is designed to be the intra-frame prediction mode derived by the DIMD mode for the prediction block of the current block. Table 2 is the result of testing on a test sequence when the optimal MIP mode and the intra-frame prediction mode derived by the DIMD mode for the reconstructed samples in the first template area adjacent to the current block are used to perform weighted prediction on the current block, and the first intra-frame prediction mode is designed to be the intra-frame prediction mode derived by the DIMD mode for the prediction block of the current block.

[0173] Table 1

[0174]

[0175] Table 2

[0176]

[0177] As shown in Tables 1 and 2, the incremental bit rate (BD-rate) is negative, which represents the performance improvement of the solution provided by this application relative to the test results of ECM2.0. From the test results, it can be seen that under the general test conditions, the test results of Tables 1 and 2 can provide an average brightness performance gain of 0.20%, and the 4K sequence performs well. It is worth noting that the TIMD prediction mode integrated in ECM2.0 has a higher complexity based on ECM1.0, and has only a 0.4% performance gain. In the current situation where intra-frame coding performance is becoming increasingly difficult to obtain, the solution provided by this application can bring good performance gains without increasing the complexity of the decoder, especially for 4K type video sequences, the performance gain is obvious. In addition, due to server load reasons, even if the encoding and decoding time fluctuates slightly, the decoding time will basically not increase in theory.

[0178] In some embodiments, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode before upsampling; or, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode after upsampling.

[0179] In other words, the process of using the intra-frame prediction mode derived from the DIMD mode on the output vector of the optimal MIP mode can be performed before upsampling the vector output by the optimal MIP mode, or after upsampling the vector output by the optimal MIP mode. This application does not make any specific restrictions on this.

[0180] After the decoder inputs the prediction matrix of the optimal MIP mode into the reference sample and obtains the output vector, the vector output by the optimal MIP mode has a maximum of 64 prediction samples. Compared to the upsampled prediction block having a maximum of several thousand prediction samples, the decoder uses the first intra-frame prediction mode derived by the DIMD mode before upsampling the vector output by the optimal MIP mode, which can reduce computational complexity and thereby improve decompression performance of the current block. For example, using the DIMD mode to calculate the gradient magnitude value of each traditional prediction mode before upsampling can effectively reduce computational complexity.

[0181] In some embodiments, the S350 may include:

[0182] The decoder determines the first intra prediction mode based on a prediction mode used to predict the current block.

[0183] Exemplarily, the decoder determines the first intra prediction mode based on a mode type of a prediction mode used to predict the current block.

[0184] Exemplarily, the decoder determines the first intra prediction mode based on a derived mode of a prediction mode used to predict the current block.

[0185] Exemplarily, the derived modes of the prediction mode used to predict the current block include but are not limited to: MIP mode, the DIMD mode and the TIMD mode.

[0186] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block, the decoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0187] In other words, if the decoder uses the optimal MIP mode and the suboptimal MIP mode to perform weighted prediction on the current block, the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0188] In this embodiment, when the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode, when the decoder preferentially uses the intra-frame prediction mode derived from the DIMD mode for the prediction block of the current block as the first intra-frame prediction mode, the texture direction of the transform set corresponding to the first intra-frame prediction mode can simultaneously fit the texture characteristics exhibited by the prediction block of the current block by the optimal MIP mode and the texture characteristics exhibited by the prediction block of the current block by the suboptimal MIP mode, thereby maximizing the decompression performance of the current block; when the decoder preferentially uses the intra-frame prediction mode derived from the DIMD mode for the output vector of the optimal MIP mode as the first intra-frame prediction mode, the output vector of the optimal MIP mode can be directly obtained in the process of determining the optimal MIP mode. This is equivalent to enabling the texture direction of the transform set corresponding to the first intra-frame prediction mode to fit the texture characteristics exhibited by the prediction block of the current block by the optimal MIP mode on the basis of reducing the decompression complexity, thereby maximizing the decompression performance of the current block.

[0189] Of course, in other alternative embodiments, when the prediction mode for predicting the current block includes the optimal MIP mode and the suboptimal MIP mode, the decoder may also determine the intra-frame prediction mode derived from the DIMD mode or the intra-frame prediction mode derived from the TIMD mode for the reconstructed samples within the first template area as the first intra-frame prediction mode, and this application does not make any specific limitations on this.

[0190] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode, the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived from the TIMD mode.

[0191] In other words, if the second intra-frame prediction mode is an intra-frame prediction mode derived using the TIMD mode, the decoder determines that the first intra-frame prediction mode is an intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines that the first intra-frame prediction mode is an intra-frame prediction mode derived from the TIMD mode.

[0192] In this embodiment, when the second intra-frame prediction mode includes the intra-frame prediction mode derived from the TIMD mode, when the decoder preferentially uses the intra-frame prediction mode derived from the TIMD mode for the prediction block of the current block as the first intra-frame prediction mode, the texture direction of the transform set corresponding to the first intra-frame prediction mode can simultaneously fit the texture characteristics of the prediction block of the current block exhibited by the optimal MIP mode and the texture characteristics of the prediction block of the current block exhibited by the intra-frame prediction mode derived from the TIMD mode, thereby improving the decompression performance of the current block as much as possible; when the decoder preferentially uses the intra-frame prediction mode derived from the TIMD mode as the first intra-frame prediction mode, the second intra-frame prediction mode can be directly determined as the first intra-frame prediction mode, which is equivalent to, on the basis of reducing the decompression complexity, making the texture direction of the transform set corresponding to the first intra-frame prediction mode fit the texture characteristics of the prediction block of the current block exhibited by the optimal MIP mode, thereby improving the decompression performance of the current block as much as possible.

[0193] Of course, in other alternative embodiments, when the second intra-frame prediction mode is the intra-frame prediction mode derived from the TIMD mode, the decoder may also determine to use the intra-frame prediction mode derived from the DIMD mode for the output vector of the optimal MIP mode and to use the intra-frame prediction mode derived from the DIMD mode for the reconstructed samples within the first template area as the first intra-frame prediction mode. This application does not make any specific limitations on this.

[0194] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, the decoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0195] In other words, if the second intra-frame prediction mode is an intra-frame prediction mode derived using the DIMD mode, the decoder determines that the first intra-frame prediction mode is an intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines that the first intra-frame prediction mode is an intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0196] In this embodiment, when the second intra-frame prediction mode includes the intra-frame prediction mode derived by the DIMD mode for the reconstructed samples within the first template area, the decoder prioritizes the intra-frame prediction mode derived by the DIMD mode for the prediction block of the current block as the first intra-frame prediction mode, so that the texture direction of the transform set corresponding to the first intra-frame prediction mode can simultaneously fit the texture characteristics of the prediction block of the current block exhibited by the optimal MIP mode and the texture characteristics of the prediction block of the current block exhibited by the intra-frame prediction mode derived by the DIMD mode, thereby improving the decompression performance of the current block as much as possible; the decoder prioritizes the intra-frame prediction mode derived by the DIMD mode for the reconstructed samples within the first template area as the first intra-frame prediction mode, and can directly determine the second intra-frame prediction mode as the first intra-frame prediction mode, which is equivalent to, on the basis of reducing the decompression complexity, making the texture direction of the transform set corresponding to the first intra-frame prediction mode fit the texture characteristics of the prediction block of the current block exhibited by the optimal MIP mode, thereby improving the decompression performance of the current block as much as possible.

[0197] Of course, in other alternative embodiments, when the second intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, the decoder may also determine to use the intra-frame prediction mode derived by the DIMD mode or the intra-frame prediction mode derived by the TIMD mode for the output vector of the optimal MIP mode as the first intra-frame prediction mode, and this application does not make any specific limitation on this.

[0198] In some embodiments, the method 300 may further include:

[0199] determining a second intra prediction mode;

[0200] The second intra prediction mode includes any one of the following: a suboptimal MIP mode for predicting the current block, an intra prediction mode derived from the DIMD mode using the reconstructed samples within the first template area, and an intra prediction mode derived from the TIMD mode;

[0201] The current block is predicted based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block.

[0202] Exemplarily, the process by which the decoder predicts the current block based on the optimal MIP mode and the determination of the second intra-frame prediction mode is referred to as Template Matching MIP (TMMIP) technology, a MIP prediction mode derivation method based on template matching, or TMMIP fusion enhancement technology; that is, after obtaining the residual block of the current block, the decoder can enhance the performance of the prediction process of the current block based on the derived optimal MIP mode and the second intra-frame prediction mode. In other words, the TMMIP technology can enhance the performance of the prediction process of the current block by using the optimal MIP prediction mode and at least one of the following: a suboptimal MIP prediction mode, an intra-frame prediction mode derived from the TIMD mode, and an intra-frame prediction mode derived using the DIMD mode for reconstructed samples within the first template area adjacent to the current block.

[0203] In this embodiment, the decoder predicts the current block based on the optimal MIP mode and the second intra-frame prediction mode, and designs the optimal MIP mode to be the optimal MIP mode for predicting the current block determined based on the distortion costs of multiple MIP modes, and designs the second intra-frame prediction mode to include at least one of the following: a suboptimal MIP mode for predicting the current block determined based on the distortion costs of the multiple MIP modes, an intra-frame prediction mode derived using the DIMD mode for reconstructed samples in a first template area adjacent to the current block, and an intra-frame prediction mode derived from the TIMD mode. This is equivalent to avoiding the decoder from obtaining the MIP mode by parsing the bitstream. Compared with traditional MIP technology, it can effectively reduce the bit overhead at the coding unit level, thereby improving the decompression efficiency of the current block.

[0204] Specifically, the MIP mode has a larger bit overhead than other intra-frame prediction modes. It requires not only a usage flag to indicate whether the MIP mode is used, but also a transposition flag to indicate whether the MIP mode is transposed. Finally, and this is the largest overhead, it requires the use of a truncated binary code to represent the MIP mode index. The MIP mode is a simplified technology based on neural network technology and is significantly different from traditional interpolation filter prediction technology. For some special textures, the MIP mode often performs better than the traditional intra-frame prediction mode. However, its large flag bit overhead is a drawback of the MIP mode. Taking a 4x4 coding unit as an example, the MIP mode has 16 prediction modes, but its bit overhead includes one MIP mode usage flag, one MIP mode transposition flag, and a 5- or 6-bit truncated binary flag. In view of this, the present application utilizes a method in which the decoder autonomously determines the optimal MIP mode for predicting the current block and determines the intra-frame prediction mode for the current block based on the optimal MIP mode. This method can save up to 5 or 6 bits of overhead, effectively reducing the bit overhead at the coding unit level and thereby improving decompression efficiency.

[0205] Furthermore, the assumption for saving up to 5 or 6 bits of overhead per coding unit is that the derivation algorithm based on template matching prediction mode is sufficiently accurate. If the accuracy of the derivation algorithm based on template matching prediction mode is too low, the MIP mode derived by the encoder side will be different from the MIP mode derived by the encoder side, thereby reducing codec performance. In other words, codec performance depends on the accuracy of the derivation algorithm based on template matching prediction mode.

[0206] However, the accuracy of both the traditional intra-frame prediction mode template-based derivation algorithm and the inter-frame template matching derivation algorithm is unsatisfactory. Although they can save bit overhead and improve compression efficiency, as the number of prediction modes of the derivation algorithm based on the template matching prediction mode increases, the additional bit overhead at the coding unit level brought by the derivation algorithm based on the template matching prediction mode is no longer sufficient for subsequent technologies to rely solely on the derivation algorithm based on the template matching prediction mode to improve compression efficiency. Therefore, the derivation algorithm based on the template matching prediction mode urgently needs to improve the encoding and decoding performance on the basis of improving compression efficiency. As a possible implementation method, while striving to save bit overhead at the coding unit level, different novel prediction blocks can be created to ensure prediction diversity and selection diversity, thereby improving encoding and decoding performance. In view of this, the present application fuses the optimal MIP mode and the second intra-frame prediction mode, that is, performs a fusion prediction on the current block based on the optimal MIP mode and the second intra-frame prediction mode, avoiding the optimal MIP mode completely replacing the optimal prediction mode calculated based on the rate-distortion cost, and can balance prediction accuracy and prediction diversity, thereby improving decompression performance.

[0207] In particular, because the TMMIP technology predicts the current block using a combination of the optimal MIP mode and the second intra-frame prediction mode, and the prediction blocks obtained by predicting the current block using different prediction modes may have different texture characteristics, if the TMMIP technology is selected for the current block, the optimal MIP mode may cause the prediction block of the current block to exhibit one texture characteristic, while the second intra-frame prediction mode may cause the prediction block of the current block to exhibit another texture characteristic. In other words, after predicting the current block, from a statistical perspective, the residual block of the current block will also exhibit two texture characteristics, that is, the residual block of the current block does not necessarily conform to the rules that can be reflected by a single prediction mode. In this case, for the TMMIP technology, when the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, the texture direction of the transform set corresponding to the first intra-frame prediction mode can simultaneously align with the texture characteristics exhibited by the prediction block of the current block by the optimal MIP mode and the texture characteristics exhibited by the prediction block of the current block by the second intra-frame prediction mode, thereby improving the decompression performance of the current block. Furthermore, when the first intra-frame prediction mode is the intra-frame prediction mode derived from the DIMD mode for the reconstructed samples in the first template area or the intra-frame prediction mode derived from the TIMD mode, the first intra-frame prediction mode can be directly determined in the process of determining the second intra-frame prediction mode. This is equivalent to reducing the decompression complexity so that the texture direction of the transform set corresponding to the first intra-frame prediction mode can simultaneously fit the texture characteristics of the prediction block of the current block exhibited by the optimal MIP mode and the texture characteristics of the prediction block of the current block exhibited by the second intra-frame prediction mode, thereby improving the decompression efficiency.

[0208] In some embodiments, the decoder first predicts the current block based on the optimal MIP mode to obtain a first prediction block; then predicts the current block based on the second intra-frame prediction mode to obtain a second prediction block; and then performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode to obtain a prediction block of the current block.

[0209] In some embodiments, before performing weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra prediction mode to obtain the prediction block of the current block, the method 300 may further include:

[0210] If the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block or the intra-frame prediction mode derived from the TIMD mode, the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are determined based on the distortion cost of the optimal MIP mode and the distortion cost of the second intra-frame prediction mode; if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are determined to be preset values.

[0211] In some embodiments, the decoder predicts the current block based on the optimal MIP mode to obtain a first prediction block; and predicts the current block based on the second intra-frame prediction mode to obtain a second prediction block; then, the decoder performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode to obtain a prediction block for the current block.

[0212] Exemplarily, the decoder may directly perform intra-frame prediction on the current block based on the optimal MIP mode to obtain the first prediction block. In addition, the decoder may directly obtain the optimal prediction mode and the suboptimal prediction mode based on the TIMD mode, perform prediction on the current block, and obtain the second prediction block. For example, if neither the optimal prediction mode nor the suboptimal prediction mode is a DC mode (also known as a mean mode) or a plane mode (also known as a flat mode), and the distortion cost of the suboptimal prediction mode is less than twice the distortion cost of the optimal prediction mode, a prediction block fusion operation is required; that is, the decoder may first perform intra-frame prediction on the current block according to the optimal prediction mode to obtain the optimal prediction block; secondly, perform intra-frame prediction on the current block according to the suboptimal prediction mode to obtain the suboptimal prediction block; then use the ratio between the distortion cost of the optimal prediction mode and the distortion cost of the suboptimal prediction mode to calculate the weight value of the optimal prediction block and the weight value of the suboptimal prediction block; finally, the optimal prediction block and the suboptimal prediction block are weightedly fused to obtain the second prediction block. For another example, if the optimal prediction mode or the suboptimal prediction mode is a planar mode or a DC mode, or if the distortion cost of the optimal prediction mode is greater than twice the distortion cost of the optimal prediction mode, then no prediction block fusion operation is required. That is, only the optimal prediction block obtained based on the optimal prediction mode can be directly used as the second prediction block. After the decoder obtains the first prediction block and the second prediction block, it performs weighted processing on the first prediction block and the second prediction block to obtain the prediction block of the current block.

[0213] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block or the intra-frame prediction mode derived from the TIMD mode, the decoder determines the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the second intra-frame prediction mode; if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, the decoder determines that the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are both preset values.

[0214] In some embodiments, the S320 may include:

[0215] The decoder parses the code stream of the current sequence to obtain a first identifier; if the first identifier is used to indicate that the image block in the current sequence is allowed to be predicted using the optimal MIP mode and the second intra-frame prediction mode, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0216] Exemplarily, if the value of the first identifier is a first numerical value, it is used to indicate that the image blocks in the current sequence are allowed to be predicted using the optimal MIP mode and the second intra-frame prediction mode; if the value of the first identifier is a second numerical value, it is used to indicate that the image blocks in the current sequence are not allowed to be predicted using the optimal MIP mode and the second intra-frame prediction mode. In one implementation, the first numerical value is 1 and the second numerical value is 0. In another implementation, the first numerical value is 0 and the second numerical value is 1. Of course, the first numerical value and the second numerical value may also be other numerical values, which is not limited in this application.

[0217] Exemplarily, if the first identifier is true, it is used to indicate that the image blocks in the current sequence are allowed to be predicted using the optimal MIP mode and the second intra-frame prediction mode; if the first identifier is false, it is used to indicate that the image blocks in the current sequence are not allowed to be predicted using the optimal MIP mode and the second intra-frame prediction mode.

[0218] Exemplarily, the decoder parses the block-level identifier, and if the current block adopts the intra-frame prediction mode, parses or obtains the first identifier, and if the first identifier is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0219] Exemplarily, the first flag is recorded as sps_timd_enable_flag. At this time, the decoder parses or obtains sps_timd_enable_flag. If the sps_timd_enable_flag is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0220] Exemplarily, the first identifier is a sequence-level identifier.

[0221] It should be noted that the first identifier is used to identify that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the image blocks in the current sequence, and may also be replaced by a description with a similar or identical meaning. For example, in other alternative embodiments, the first identifier is used to identify that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the image blocks in the current sequence, and may also be replaced by any one of the following: the first identifier is used to identify that the intra-frame prediction mode of the image blocks in the current sequence is allowed to be determined using the TMMIP technology, the first identifier is used to identify that the TMMIP technology is allowed to be used to perform intra-frame prediction on the image blocks in the current sequence, the first identifier is used to identify that the TMMIP technology is allowed to be used for the image blocks in the current sequence, and the first identifier is used to identify that the MIP mode determined based on the multiple MIP modes is allowed to be used to predict the image blocks in the current sequence.

[0222] In addition, in other alternative embodiments, when TMMIP technology is combined with other technologies, the permission flag of the other technology can also be used to indirectly indicate whether the current sequence allows the use of TMMIP technology. For example, using TIMD technology as an example, if the first flag is used to indicate that the current sequence allows the use of TIMD technology, it means that the current sequence also allows the use of TMMIP technology; or, if the first flag is used to indicate that the current sequence allows the use of TIMD technology, it means that the current sequence allows the use of both TIMD technology and TMMIP technology, thereby further saving bit overhead.

[0223] In some embodiments, if the first identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, the decoder parses the code stream to obtain the second identifier; if the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0224] Exemplarily, the decoder parses the block-level identifier. If the current block adopts the intra-frame prediction mode, the decoder parses or obtains the first identifier. If the first identifier is true, the decoder parses or obtains the second identifier. If the second identifier is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0225] Exemplarily, if the third value of the second identifier is set, it is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block; if the fourth value of the second identifier is set, it is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are not allowed to be used to predict the current block. In one implementation, the third value is 1 and the fourth value is 0. In another implementation, the third value is 0 and the fourth value is 1. Of course, the third value and the fourth value may also be other values, and this application does not specifically limit this.

[0226] Exemplarily, if the second identifier is true, it is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block; if the second identifier is false, it is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are not allowed to be used to predict the current block.

[0227] Exemplarily, the first identifier is recorded as sps_timd_enable_flag, and the second identifier is recorded as cu_timd_enable_flag. At this time, the decoder parses or obtains sps_timd_enable_flag. If the sps_timd_enable_flag is true, the decoder can parse or obtain cu_timd_enable_flag. If the cu_timd_enable_flag is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0228] Exemplarily, the second identifier is a block-level identifier or a coding unit-level identifier.

[0229] It should be noted that the second identifier is used to identify that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block, and may also be replaced by a description with a similar or identical meaning. For example, in other alternative embodiments, the second identifier is used to identify that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block, and may also be replaced by any one of the following: the second identifier is used to identify that the intra-frame prediction mode of the current block is allowed to be determined using the TMMIP technology, the second identifier is used to identify that the intra-frame prediction of the current block is allowed to be performed using the TMMIP technology, the second identifier is used to identify that the image block in the current block is allowed to use the TMMIP technology, and the second identifier is used to identify that the MIP mode determined based on the multiple MIP modes is allowed to be used to predict the current block.

[0230] Furthermore, in alternative embodiments, when TMMIP is combined with other technologies, the permission flag of the other technologies can be used to indirectly indicate whether the current block is allowed to use TMMIP. For example, using TIMD as an example, if the second flag is used to indicate that the current block is allowed to use TIMD, then it indicates that the current block is also allowed to use TMMIP. Alternatively, if the second flag is used to indicate that the current block is allowed to use TIMD, then it indicates that the current block is allowed to use both TIMD and TMMIP, thereby further saving bit overhead.

[0231] In addition, when the decoding end parses the second identifier, it can parse the second identifier before parsing the residual block of the current block, or it can parse the second identifier after parsing the residual block of the current block. This application does not make specific restrictions on this.

[0232] In some embodiments, the method 300 may further include:

[0233] The decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes;

[0234] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a second template area adjacent to the current block using the multiple MIP modes.

[0235] Exemplarily, the decoder needs to calculate the distortion cost of each of the multiple MIP modes before determining the optimal MIP mode for predicting the current block based on the distortion costs of multiple MIP modes, and sort the multiple MIP modes according to the distortion cost of each MIP mode. The MIP mode with the smallest cost is the optimal prediction result.

[0236] It is worth noting that the distortion cost involved in the decoder in this application is different from the rate-distortion cost (RDcost) involved in the encoder. The rate-distortion cost is the distortion cost used by the encoder to determine a certain intra-frame prediction technology among multiple intra-frame prediction technologies. The rate-distortion cost can be the cost value obtained by comparing the distorted image and the original image. Since the decoder cannot obtain the original image, the distortion cost involved in the decoder can be the distortion cost between the reconstructed sample and the predicted sample, such as the sum of absolute transformed differences (SATD) between the reconstructed sample and the predicted sample or other costs that can be used to calculate the difference between the reconstructed sample and the predicted sample.

[0237] Of course, in other alternative embodiments, the decoder first determines the arrangement order of the multiple MIP modes based on the distortion cost of the MIP modes; then determines the encoding method used by the optimal MIP mode based on the arrangement order of the multiple MIP modes; and then decodes the code stream of the current sequence based on the encoding method used by the optimal MIP mode to obtain the index of the optimal MIP mode.

[0238] For example, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable-length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method. Exemplarily, n can be any value greater than or equal to 1. It should be noted that for traditional MIP technology, the index of the MIP mode is usually binarized and written in a truncated binary manner. This encoding method is closer to equal probability encoding, that is, it divides all prediction modes into two segments, one segment is represented by N codewords, and the other is represented by N+1 codewords. In view of this, in the present application, before the decoder determines the optimal MIP mode for predicting the current block based on the distortion costs of multiple MIP modes, it can first calculate the distortion cost of each MIP mode in the multiple MIP modes, and sort the multiple MIP modes according to the distortion cost of each MIP mode. Finally, the decoder can choose to use a more flexible variable-length coding method based on the sorting of the multiple MIP modes. Compared with the equal probability coding method, by flexibly setting the coding method of the MIP mode, it is beneficial to save the bit overhead of the index of the MIP mode.

[0239] For another example, the arrangement order is the order obtained by the decoder arranging the multiple MIP modes in order of distortion cost from small to large. Since the smaller the distortion cost of the MIP mode, the greater the probability that the encoder will use it to perform intra-frame prediction on the current block, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is designed to be smaller than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the encoding method used by the first n MIP modes is designed to be a variable-length encoding method and the encoding method used by the MIP modes after the nth MIP mode is designed to be a truncated binary encoding method; this is equivalent to using a shorter codeword length or a variable-length encoding method for the MIP mode with a high probability used by the encoder, which can save the bit overhead of the MIP mode index and improve the decompression performance.

[0240] In some embodiments, the method 300 may further include:

[0241] If the second intra-frame prediction mode is the suboptimal MIP mode, the decoder determines whether to adopt the suboptimal MIP mode to predict the current block based on the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode; if it is determined not to adopt the suboptimal MIP mode, the decoder may directly predict the current block based on the optimal MIP mode; if it is determined to adopt the suboptimal MIP mode, the decoder may predict the current block based on the optimal MIP mode and the suboptimal MIP mode to obtain a prediction block of the current block.

[0242] Exemplarily, when the second intra-frame prediction mode is a suboptimal MIP mode, if the ratio between the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode is less than or equal to a preset ratio, the decoder may directly predict the current block based on the optimal MIP mode to obtain a predicted block for the current block. Alternatively, when the second intra-frame prediction mode is a suboptimal MIP mode, if the ratio between the distortion cost of the suboptimal MIP mode and the distortion cost of the optimal MIP mode is greater than or equal to a preset ratio, the decoder may directly predict the current block based on the optimal MIP mode to obtain a predicted block for the current block. For example, if the distortion cost of the suboptimal MIP mode is greater than or equal to a certain multiple (e.g., twice) of the distortion cost of the optimal MIP mode, it can be interpreted that the suboptimal MIP mode already has a large distortion and is not suitable for the current block, that is, the fusion enhancement technology can be omitted and only the optimal MIP mode can be used to predict the current block.

[0243] In this embodiment, the decoder determines whether to adopt the suboptimal MIP mode to predict the current block based on the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode. This is equivalent to the decoder determining whether to adopt the suboptimal MIP mode to enhance the performance of the optimal MIP mode based on the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode. This avoids carrying an identifier for determining whether to adopt the suboptimal MIP mode to enhance the performance of the optimal MIP mode in the bitstream, saves bit overhead, and can further enhance decompression performance.

[0244] In some embodiments, the second template region is the same as or different from the first template region.

[0245] Exemplarily, the size of the second template area can be predefined based on the size of the current block. For example, the width of the upper area of ​​the second template area adjacent to the current block is the width of the current block, and its height is the height of at least one row of samples. For another example, the height of the left area of ​​the second template area adjacent to the left side of the current block is the height of the current block, and its width is the width of two rows of samples. Of course, in other alternative embodiments, the second template area can also be implemented as a second template area of ​​other sizes or dimensions, and this application does not specifically limit this.

[0246] In some embodiments, the method 300 may further include:

[0247] The decoder predicts samples within the second template area based on the third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes in each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode; the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes in each state of the third identifier.

[0248] Exemplarily, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the decoder predicts samples in the second template area based on a third identifier and the multiple MIP modes to obtain the distortion costs of the multiple MIP modes in each state of the third identifier.

[0249] As mentioned above, traditional MIP technology has a higher bit overhead than other intra-frame prediction tools. It not only requires a flag to indicate whether MIP technology is used, but also a flag to indicate whether MIP is transposed. Finally, and this is the largest overhead, it requires the use of truncated binary code to represent the MIP prediction mode. MIP is a technology simplified based on neural network technology and is significantly different from traditional interpolation filter prediction technology. For some special textures, although MIP prediction is better than traditional intra-frame prediction mode, its large flag overhead is a drawback of MIP technology. Taking a 4x4 coding unit as an example, there are 16 prediction samples, but its bit overhead includes 1 MIP use flag, 1 MIP transposition flag, and 5 or 6 bits of truncated binary flags. In view of this, when determining the optimal MIP mode, this application takes into account the transposition function of the MIP mode by traversing each state of the third flag, which can save the overhead of 1 MIP transposition flag, thereby improving decompression efficiency.

[0250] Exemplarily, the decoder traverses each state of the third identifier and the multiple MIP modes, determines the distortion cost of the multiple MIP modes in each state of the third identifier, and determines the optimal MIP mode based on the distortion cost of the multiple MIP modes in each state of the third identifier; or the decoder traverses each state of the third identifier and the multiple MIP modes, determines the distortion cost of the multiple MIP modes in each state of the third identifier, and determines the optimal MIP mode based on the distortion cost of the multiple MIP modes in each state of the third identifier. In other words, the decoding end can first traverse the multiple MIP modes, or first traverse the state of the third identifier.

[0251] For example, if the value of the third identifier is the fifth numerical value, it is used to identify the input vector and output vector of the transposed MIP mode; if the value of the third identifier is the sixth numerical value, it is used to identify the input vector and output vector of the non-transposed MIP mode. In this case, each state of the third identifier can also be replaced by each value of the third identifier. In one implementation, the fifth numerical value is 1 and the sixth numerical value is 0. In another implementation, the fifth numerical value is 0 and the sixth numerical value is 1. Of course, the fifth and sixth numerical values ​​can also be other numerical values, and this application is not limited to this.

[0252] Exemplarily, if the third flag is true, it is used to identify the input vector and output vector of the transposed MIP mode; if the third flag is false, it is used to identify the input vector and output vector of the non-transposed MIP mode. In this case, whether the third flag is true or false is a state of the third flag.

[0253] Exemplarily, the third identifier is a sequence-level identifier, a block-level identifier, or a coding unit-level identifier.

[0254] Exemplarily, the third identifier may also be referred to as transposition information, a transposition identifier, or a MIP transposition identifier.

[0255] It should be noted that the third flag is used to identify whether the input vector and output vector of the MIP mode are transposed, and can also be replaced by a description with a similar or identical meaning. For example, in other alternative embodiments, the third flag is used to identify whether the input and output of the MIP mode need to be transposed, the third flag is used to identify whether the input vector and output vector of the MIP mode are transposed vectors, and the third flag is used to indicate whether transposition is performed.

[0256] In some embodiments, the method 300 may further include:

[0257] If the size of the current block is a preset size, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0258] Exemplarily, the preset size may include a size having a preset width and a preset height. That is, if the width of the current block is the preset width and the height is the preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0259] Exemplarily, the preset size can be implemented by pre-storing corresponding codes, tables, or other methods that can be used to indicate relevant information in a device (e.g., including a decoder and an encoder). This application does not limit the specific implementation method. For example, the preset size can refer to a size defined in a protocol. Alternatively, the "protocol" can refer to a standard protocol in the field of codec technology, such as a VCC or ECM protocol.

[0260] Of course, in other alternative embodiments, the decoder may also determine whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes based on the preset size in other ways, and this application does not make specific limitations on this.

[0261] For example, the decoder may determine whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes based solely on the width or height of the current block. In one implementation, if the width of the current block is a preset width or the height is a preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. For another example, the decoder may determine whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes by comparing the size of the current block with the preset size. In one implementation, if the size of the current block is greater than or less than the preset size, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. In another implementation, if the width of the current block is greater than or less than the preset width, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. In another implementation, if the height of the current block is greater than or less than the preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0262] In some embodiments, the method 300 may further include:

[0263] If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0264] Exemplarily, if the image frame containing the current block is an I-frame, the width of the current block is a preset width, and the height of the current block is a preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. In other words, only when the image frame containing the current block is an I-frame does the decoder determine, based on the size of the current block, whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0265] In some embodiments, the method 300 may further include:

[0266] If the image frame where the current block is located is a B frame, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.

[0267] For example, if the image frame containing the current block is a B-frame, the decoder may determine the optimal MIP mode directly based on the distortion costs of the multiple MIP modes. In other words, when the image frame containing the current block is a B-frame, regardless of the size of the current block, the decoder may determine the optimal MIP mode directly based on the distortion costs of the multiple MIP modes.

[0268] In some embodiments, before S320, the method 300 may further include:

[0269] The decoder obtains the MIP mode used by the neighboring blocks adjacent to the current block; and the decoder determines the MIP mode used by the neighboring blocks as the multiple MIP modes.

[0270] Exemplarily, the adjacent block may be an image block adjacent to at least one of the upper side, left side, lower left, upper right, and upper left of the current block. For example, the decoder may determine the image blocks obtained in the order of the upper side, left side, lower left, upper right, and upper left of the current block as the adjacent block. Optionally, the multiple MIP modes may be used to construct an available MIP mode or a list of available MIP modes that the decoder determines to be used for predicting the current block, so that the decoder determines the optimal MIP mode from the available MIP modes or the list of available MIP modes by predicting samples within the second template area.

[0271] In some embodiments, the method 300 may further include:

[0272] The decoder performs reconstructed sample filling on a reference area adjacent to the second template area to obtain reference rows and reference columns of the second template area; the decoder uses the reference rows and the reference columns as input and uses the multiple MIP modes to predict samples in the second template area respectively to obtain multiple prediction blocks corresponding to the multiple MIP modes; the decoder determines the distortion costs of the multiple MIP modes based on the multiple prediction blocks and the reconstructed blocks in the second template area.

[0273] Exemplarily, before the decoder determines the optimal MIP mode based on the distortion costs of multiple MIP modes, the decoder performs reconstructed sample padding on the adjacent reference area outside the second template area.

[0274] Exemplarily, the width of the region adjacent to the upper side of the second template region in the reference region is equal to the width of the second template region, and the height of the region adjacent to the left side of the second template region is equal to the width of the second template region. If the width of the region adjacent to the upper side of the second template region in the reference region is greater than the width of the second template region, the decoder may downsample or perform dimensionality reduction processing on the region adjacent to the upper side of the second template region in the reference region to obtain the reference row. If the height of the region adjacent to the left side of the second template region in the reference region is greater than the width of the second template region, the decoder may downsample or perform dimensionality reduction processing on the region adjacent to the left side of the second template region in the reference region to obtain the reference column.

[0275] Exemplarily, the second template area may be the template area used in the TIMD mode mentioned above, and the reference area may be the reference template used in the TIMD mode. For example, in conjunction with FIG5 , if the current block is a coding unit with a width equal to M and a height equal to N, the decoder fills the reference area composed of coding units with a width equal to 2(M+L1)+1 and a height equal to 2(N+L2)+1 with reconstructed samples, and downsamples or reduces the dimension of the filled reference area to obtain the reference rows and the reference columns, and then constructs the input vector of the MIP mode based on the reference rows and the reference columns.

[0276] Exemplarily, after obtaining the reference row and the reference column, the decoder uses the reference row and the reference column as input, and uses the multiple MIP modes to predict the samples in the second template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes; that is, the decoder predicts the samples in the second template area of ​​the current block based on the reconstructed samples in the reference template of the current block by traversing the multiple MIP modes. Taking the current traversal MIP mode as an example, the decoder takes the reference row, the reference column, the index of the current traversal MIP mode, and the third identifier mentioned above as input to obtain the prediction block corresponding to the current traversal MIP mode; wherein the reference row and the reference column are used to construct the input vector of the current traversal MIP mode; the index of the current traversal MIP mode is used to determine the matrix and / or bias vector of the current traversal MIP mode; the third identifier is used to identify whether to transpose the input vector and output vector of the MIP mode; for example, if the third identifier is used to identify the input vector and output vector of the MIP mode that are not transposed, the reference column is spliced ​​after the reference row to form the input vector of the current traversal MIP mode; if the third identifier is used to identify the input vector and output vector of the transposed MIP mode, the reference row is spliced ​​after the reference column to form the input vector of the current traversal MIP mode. Accordingly, if the third identifier is used to identify the input vector and output vector of the transposed MIP mode, the decoder transposes the output of the currently traversed MIP mode to obtain a prediction block for the second template region. After the decoder traverses the multiple MIP modes to obtain multiple prediction blocks corresponding to the multiple MIP modes, the decoder can select the MIP mode with the lowest cost based on the distortion costs between the multiple prediction blocks and the reconstructed samples in the second template region according to the principle of minimizing distortion cost, and determine it as the optimal MIP mode among the template matching-based MIP modes for the current block.

[0277] In some embodiments, when the decoder uses the multiple MIP modes to predict the samples in the second template area respectively, it first downsamples the reference rows and the reference columns to obtain an input vector; then, using the input vector as input, it predicts the samples in the second template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; finally, it upsamples the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.

[0278] Exemplarily, the reference row and the reference column meet the input conditions of the multiple MIP modes. If the reference row and the reference column do not meet the input conditions of the multiple MIP modes, the reference row and / or the reference column may be first processed into input samples that meet the input conditions of the multiple MIP modes, and then the input vectors of the multiple MIP modes are determined based on the input samples that meet the input conditions of the multiple MIP modes. For example, taking the input condition as a specified number of input samples as an example, if the reference row and the reference column do not meet the number of input samples of the MIP mode, the decoder may perform Haar downsampling or other methods on the reference row and / or the reference column to reduce the dimension to the specified number of input samples, and determine the input vectors of the multiple MIP modes based on the specified number of input samples after the dimension reduction.

[0279] In some embodiments, the S320 may include:

[0280] The decoder determines the optimal MIP mode based on the sum of absolute transform differences (SATD) of the multiple MIP modes in the second template area.

[0281] In this embodiment, when the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes in the second template area, the distortion costs of the multiple MIP modes are designed as the SATDs of the multiple MIP modes. Compared with directly calculating the rate-distortion costs of the multiple MIP modes, not only can the optimal MIP mode be determined based on the distortion costs of the multiple MIP modes in the second template area, but the calculation complexity of the distortion costs of the multiple MIP modes can also be simplified, thereby improving the decompression performance of the decoder.

[0282] In summary, the solution proposed in this application proposes the concept of fusion enhancement based on the optimal MIP mode. Specifically, the decoder not only needs to determine the optimal MIP mode for predicting the current block, but also needs to fuse it with other prediction blocks to achieve different prediction effects. This not only saves bit overhead but also creates a new prediction technology. The fusion process is actually because the optimal MIP mode cannot completely replace the optimal prediction mode calculated by the encoder based on the rate-distortion cost. Therefore, the fusion method is used to balance prediction accuracy and prediction diversity.

[0283] For example, the main ideas of the decoder's MIP mode derivation method based on template matching can be divided into the following parts:

[0284] First, fill the reference area (e.g., the reference template shown in FIG5 ) with reconstructed samples, i.e., the reference reconstructed samples required for predicting the samples in the second template area (e.g., the template shown in FIG5 ). Optionally, the width and height of the reference area do not need to exceed the width and height of the second template area. If the reference area is filled with samples that exceed the width and height of the second template area, downsampling or other dimensionality reduction methods are required to meet the MIP input dimension requirements.

[0285] Then, the decoder uses the reference reconstructed samples in the reference area, the indexes of the multiple MIP modes, and the MIP transposition flag as input to predict the samples in the second template area to obtain the prediction blocks corresponding to the multiple MIP modes. Optionally, the reference reconstructed samples in the reference area need to meet the MIP input conditions, such as Haar downsampling and other input samples that are reduced to a specified number of dimensions. The indexes of the multiple MIP modes are used to determine the matrix index of the MIP technology, and then obtain the MIP prediction matrix coefficients. The MIP transposition flag is used to identify whether the input and output need to be transposed.

[0286] Then, for the prediction blocks corresponding to the multiple MIP modes, all combinations of MIP modes and whether or not the MIP is transposed can be traversed to obtain the prediction samples of the second template area under each MIP mode and each state of the MIP transposition flag, and the distortion between the prediction samples and the reconstructed samples in the second template area is calculated and the cost information is recorded; finally, according to the principle of minimum distortion, the MIP mode with the minimum cost and its corresponding MIP transposition information are selected, which is the optimal MIP mode under the MIP prediction derivation mode based on template matching for the current block.

[0287] Finally, the decoder predicts the current block using the second intra-frame prediction mode of the optimal MIP prediction mode to obtain a first prediction block and a second prediction block, and performs weighted calculation on the first prediction block and the second prediction block according to the weighted weight of the second intra-frame prediction mode of the optimal MIP prediction mode to obtain a prediction block of the current block.

[0288] It's worth noting that some of the calculations involved in this application can be replaced by lookup tables and shifting. Although the lookup table approach may produce some errors compared to direct division, it facilitates hardware implementation and reduces encoding and decoding costs. For example, calculations related to distortion cost calculations or determining the optimal MIP mode can be used.

[0289] The decoding method according to the embodiment of the present application is described in detail above from the perspective of the decoder. The encoding method according to the embodiment of the present application will be described below from the perspective of the encoder in conjunction with Figure 17.

[0290] FIG10 is a schematic flow chart of an encoding method 400 provided in an embodiment of the present application. It should be understood that the encoding method 400 can be performed by an encoder. For example, it can be applied to the encoding framework 100 shown in FIG1 . For ease of description, the following description uses an encoder as an example.

[0291] As shown in FIG10 , the encoding method 400 may include:

[0292] S410, obtaining a residual block of a current block in a current sequence;

[0293] S420, performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;

[0294] S430, determining a first intra prediction mode;

[0295] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0296] S440, performing a fourth transform on the third transform coefficient based on the transform set corresponding to the first intra prediction mode to obtain a fourth transform coefficient of the current block;

[0297] S450: Encode the fourth transform coefficient.

[0298] It should be understood that the first transform at the decoding end is the inverse transform of the fourth transform at the encoding end, and the second transform at the decoding end is the inverse transform of the third transform at the encoding end. For example, the third transform is the basic transform or main transform mentioned above, and the fourth transform is the secondary transform mentioned above. Accordingly, the first transform is the inverse transform (or inverse transform) of the secondary transform, and the second transform can be the inverse transform (or inverse transform) of the basic transform or the main transform. For example, the first transform can be an inverse (inverse) LFNST, and the second transform can be an inverse (inverse) DCT2 type, an inverse (inverse) DCT8 type, or an inverse (inverse) DST7 type, etc.; accordingly, the third transform can be a DCT2 type, a DCT8 type, or a DST7 type, etc., and the fourth transform can be a LFNST.

[0299] In some embodiments, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode before upsampling; or, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode after upsampling.

[0300] In some embodiments, the S430 may include:

[0301] The first intra prediction mode is determined based on a prediction mode used to predict the current block.

[0302] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block, the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0303] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode, the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived from the TIMD mode.

[0304] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0305] In some embodiments, the second template region is the same as or different from the first template region.

[0306] In some embodiments, the S410 may include:

[0307] determining a second intra prediction mode;

[0308] The second intra prediction mode includes any one of the following: a suboptimal MIP mode for predicting the current block, an intra prediction mode derived from the DIMD mode using the reconstructed samples within the first template area, and an intra prediction mode derived from the TIMD mode;

[0309] Predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block;

[0310] Based on the prediction block of the current block, a residual block of the current block is obtained.

[0311] In some embodiments, the current block is predicted based on the optimal MIP mode to obtain a first prediction block; the current block is predicted based on the second intra-frame prediction mode to obtain a second prediction block; and the first prediction block and the second prediction block are weighted based on the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode to obtain a prediction block of the current block.

[0312] In some embodiments, if the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block or the intra-frame prediction mode derived from the TIMD mode, the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are determined based on the distortion cost of the optimal MIP mode and the distortion cost of the second intra-frame prediction mode; if the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are determined to be preset values.

[0313] In some embodiments, the encoder obtains a first identifier; if the first identifier is used to indicate that the image block in the current sequence is allowed to be predicted using the optimal MIP mode and the second intra-frame prediction mode, then determining the second intra-frame prediction mode; wherein S450 may include:

[0314] The fourth transform coefficient and the first identifier are encoded.

[0315] In some embodiments, if the first identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, then predicting the current block based on the optimal MIP mode and the second intra-frame prediction mode to obtain a first rate-distortion cost; predicting the current block based on at least one intra-frame prediction mode to obtain at least one rate-distortion cost; if the first rate-distortion cost is less than or equal to a minimum value of the at least one rate-distortion cost, determining a prediction block obtained by predicting the current block based on the optimal MIP mode and the second intra-frame prediction mode as the prediction block of the current block; wherein, S450 may include:

[0316] encoding the fourth transform coefficient, the first identifier, and the second identifier;

[0317] Wherein, if the first rate-distortion cost is less than or equal to the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block; if the first rate-distortion cost is greater than the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are not allowed to be used to predict the current block.

[0318] In some embodiments, the method 400 may further include:

[0319] Determining the optimal MIP mode based on the distortion costs of multiple MIP modes;

[0320] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a second template area adjacent to the current block using the multiple MIP modes.

[0321] In some embodiments, the second template region is the same as or different from the first template region.

[0322] In some embodiments, samples within the second template area are predicted based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes in each state of the third identifier; the third identifier is used to identify an input vector and an output vector of whether to transpose the MIP mode; and the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.

[0323] In some embodiments, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the method 400 may further include:

[0324] Obtaining a MIP mode used by a neighboring block adjacent to the current block;

[0325] The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.

[0326] In some embodiments, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the method 400 may further include:

[0327] Reconstructed samples are filled in a reference area adjacent to the second template area to obtain reference rows and reference columns of the second template area; the reference rows and the reference columns are used as input, and the samples within the second template area are respectively predicted using the multiple MIP modes to obtain multiple prediction blocks corresponding to the multiple MIP modes; and distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the second template area.

[0328] In some embodiments, the reference rows and the reference columns are downsampled to obtain an input vector; with the input vector as input, samples within the second template area are predicted by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; the output vectors of the multiple MIP modes are upsampled to obtain prediction blocks corresponding to the multiple MIP modes.

[0329] In some embodiments, the optimal MIP mode is determined based on a sum of absolute transformation differences (SATD) of the multiple MIP modes in the second template area.

[0330] It should be understood that the encoding method can be understood as the inverse process of the decoding method. Therefore, the specific scheme of the encoding method 400 can refer to the relevant content of the decoding method 300. For the sake of ease of description, this application will not go into details.

[0331] The solution of this application is described below with reference to specific embodiments.

[0332] Example 1:

[0333] In this embodiment, the second intra prediction mode mentioned above is the suboptimal MIP mode, that is, the encoder or decoder may perform intra prediction on the current block based on the optimal MIP mode and the suboptimal MIP mode to obtain a prediction block of the current block.

[0334] The encoder iterates through the prediction modes. If the current block is in intra mode, the encoder obtains the sequence-level permission flag, which indicates whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current encoder allows the use of TMMIP technology.

[0335] Exemplarily, the encoder process can be implemented as follows:

[0336] Step 1:

[0337] If sps_tmmip_enable_flag is true, the encoder tries the TMMIP technology, that is, executes step 2; if sps_tmmip_enable_flag is false, the encoder does not try the TMMIP technology, that is, skips step 2 and directly executes step 3.

[0338] Step 2:

[0339] First, the encoder fills the adjacent rows and columns outside the second template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the encoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, all available reconstructed samples are filled in turn; if all reconstructed samples are unavailable, all are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the encoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.

[0340] Secondly, the encoder takes the reconstructed samples outside the filled second template area as input and uses the allowed MIP mode to predict the samples in the second template area.

[0341] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.

[0342] Exemplarily, the specific prediction calculation process includes: the encoder first performs Haar downsampling on the reconstructed samples, for example, the encoder determines the downsampling step size based on the block size. Then, the encoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced ​​after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced ​​after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the encoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the encoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the second template area.

[0343] Next, the encoder calculates the distortion cost based on the prediction block of the second template area obtained by traversing each MIP mode and the reconstructed samples in the second template area, and records the distortion cost value under each prediction mode and transposition information. After traversing all allowed prediction modes and transposition information, the optimal MIP mode and its corresponding transposition information, as well as the suboptimal MIP mode and its corresponding transposition information are selected according to the principle of minimum cost. The encoder determines whether fusion enhancement is required based on the relationship between the cost value of the optimal MIP mode and the cost value of the suboptimal MIP mode. If the cost value of the suboptimal MIP mode is less than twice the cost value of the optimal MIP mode, the optimal MIP prediction block and the suboptimal MIP prediction block need to be fused and enhanced. If the cost value of the suboptimal prediction mode is greater than or equal to twice the cost value of the optimal MIP mode, fusion enhancement is not required.

[0344] Finally, if fusion enhancement is required, the encoder obtains the prediction block corresponding to the optimal MIP mode and the prediction block corresponding to the suboptimal MIP mode based on the optimal MIP mode, the suboptimal MIP mode, the transposed information of the optimal MIP mode, and the transposed information of the suboptimal MIP mode. Specifically, the encoder first optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them as an input vector based on the transposed information. The encoder then reads the matrix coefficients of the current mode using the MIP mode as an index and obtains an output vector by calculating the input vector and the matrix coefficients. The encoder then transposes the output based on the transposed information and upsamples the output vector based on the size of the current block and the number of samples in the output vector to obtain the optimal MIP prediction block and the suboptimal MIP prediction block of the same size as the current block. The encoder then performs a weighted average of the optimal MIP prediction block and the suboptimal MIP prediction block based on the calculated weight values ​​of the optimal MIP mode and the suboptimal MIP mode to obtain a new prediction block as the final prediction block for the current block. If fusion enhancement is not required, the encoder can calculate the optimal MIP prediction block based on the optimal MIP mode and its transposed information. The calculation process is the same as described above. Finally, the encoder uses the optimal MIP prediction block as the prediction block of the current block.

[0345] In addition, the encoder obtains the rate-distortion cost of the current block, which is recorded as cost1.

[0346] In addition, the encoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the encoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0347] Step 3:

[0348] The encoder continues to traverse other intra prediction techniques and calculates the corresponding rate-distortion costs, which are recorded as cost2…costN.

[0349] Step 4:

[0350] If cost1 is the minimum among all rate-distortion costs, TMMIP is used for the current block, and the encoder sets the TMMIP usage flag for the current block to true and writes it into the bitstream. If cost1 is not the minimum rate-distortion cost, another intra-frame prediction technique is used for the current block, and the encoder sets the TMMIP usage flag for the current block to false and writes it into the bitstream. It should be understood that information such as the flags or indices of other intra-frame prediction techniques is transmitted according to definition and is not elaborated on here.

[0351] Step 5:

[0352] The encoder determines a residual block for the current block based on the predicted block and the original block of the current block. The encoder then performs a base transform on the residual block of the current block and performs a secondary transform on the transform coefficients after the base transform based on the first intra-frame prediction mode. The encoder then performs operations such as quantization, entropy coding, and loop filtering on the transform coefficients after the secondary transform. It should be understood that the specific quantization process can be found in the relevant content above and will not be repeated here to avoid repetition.

[0353] The following describes the relevant solutions of the decoder in this embodiment.

[0354] The decoder parses the block-level type flag. If it is intra mode, it parses or obtains the sequence-level permission flag, which is used to indicate whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current decoder allows the use of TMMIP technology.

[0355] Exemplarily, the decoder process can be implemented as follows:

[0356] Step 1:

[0357] If sps_tmmip_enable_flag is true, the decoder parses the TMMIP usage flag for the current block. Otherwise, the current decoding process does not need to decode the block-level TMMIP usage flag, and the block-level TMMIP usage flag defaults to false. If the TMMIP usage flag for the current block is true, proceed to step 2; otherwise, proceed to step 3.

[0358] Step 2:

[0359] First, the decoder fills the adjacent rows and columns outside the second template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the decoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, all available reconstructed samples are filled in turn; if all reconstructed samples are unavailable, all are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the decoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.

[0360] Secondly, the decoder takes the reconstructed samples outside the filled second template area as input and uses the allowed MIP mode to predict the samples within the second template area.

[0361] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.

[0362] Exemplarily, the specific prediction calculation process includes: the decoder first performs Haar downsampling on the reconstructed samples, for example, the decoder determines the downsampling step size based on the block size. Then, the decoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced ​​after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced ​​after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the decoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the decoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the second template area.

[0363] Next, the decoder calculates the distortion cost based on the prediction block of the second template area obtained by traversing each MIP mode and the reconstructed samples in the second template area, and records the distortion cost value under each prediction mode and transposition information. After traversing all allowed prediction modes and transposition information, the optimal MIP mode and its corresponding transposition information, as well as the suboptimal MIP mode and its corresponding transposition information are selected according to the principle of minimum cost. The decoder determines whether fusion enhancement is required based on the relationship between the cost value of the optimal MIP mode and the cost value of the suboptimal MIP mode. If the cost value of the suboptimal MIP mode is less than twice the cost value of the optimal MIP mode, the optimal MIP prediction block and the suboptimal MIP prediction block need to be fused and enhanced. If the cost value of the suboptimal prediction mode is greater than or equal to twice the cost value of the optimal MIP mode, fusion enhancement is not required.

[0364] Finally, if fusion enhancement is required, the decoder obtains the prediction block corresponding to the optimal MIP mode and the prediction block corresponding to the suboptimal MIP mode based on the optimal MIP mode, the suboptimal MIP mode, the transposed information of the optimal MIP mode, and the transposed information of the suboptimal MIP mode. Specifically, the decoder first optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them as an input vector based on the transposed information. The decoder then reads the matrix coefficients of the current mode using the MIP mode as an index. The output vector is then calculated by combining the input vector and the matrix coefficients. The decoder can perform an output transposition based on the transposed information and upsample the output vector based on the size of the current block and the number of samples in the output vector to obtain the optimal MIP prediction block and the suboptimal MIP prediction block of the same size as the current block. The decoder then performs a weighted average of the optimal MIP prediction block and the suboptimal MIP prediction block based on the calculated weight values ​​of the optimal MIP mode and the suboptimal MIP mode to obtain a new prediction block as the final prediction block for the current block. If fusion enhancement is not required, the decoder can calculate the optimal MIP prediction block based on the optimal MIP mode and its transposed information. The calculation process is the same as described above. Finally, the decoder uses the optimal MIP prediction block as the prediction block of the current block.

[0365] In addition, the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0366] Step 3:

[0367] The decoder continues to parse information such as usage identification bits or indexes of other intra-frame prediction technologies, and obtains the final prediction block of the current block based on the parsed information.

[0368] Step 4:

[0369] The decoder parses the bitstream and obtains the frequency domain residual block of the current block (also called frequency domain residual information), and performs inverse quantization and inverse transformation on the frequency domain residual block of the current block (first performing the inverse transformation of the secondary transformation based on the first intra-frame prediction mode and then performing the inverse transformation of the basic transformation or the inverse transformation of the main transformation) to obtain the residual block of the current block (also called the time domain residual block or time domain residual information); then the decoder superimposes the prediction block of the current block with the residual block of the current block to obtain a reconstructed sample block.

[0370] Step 5:

[0371] If all reconstructed sample blocks in the current image are processed through loop filtering and other technologies, the final reconstructed image is obtained.

[0372] Optionally, the reconstructed image can be output as a video or used as a reference for subsequent decoding.

[0373] In this embodiment, the size of the second template region used by the encoder or decoder in the TMMIP technique can be predefined based on the size of the current block. For example, the width of the upper region of the second template region adjacent to the current block is the width of the current block, and its height is the height of two rows of samples; the height of the left region of the second template region adjacent to the left side of the current block is the height of the current block, and its width is the width of two rows of samples. Of course, in other alternative embodiments, second template regions of other sizes can also be implemented, and this application does not specifically limit this.

[0374] In this embodiment, through TMMIP technology

[0375] Example 2:

[0376] In this embodiment, the second intra-frame prediction mode mentioned above is the intra-frame prediction mode derived from the TIMD mode, that is, the encoder or decoder can perform intra-frame prediction on the current block based on the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode to obtain a prediction block of the current block.

[0377] That is to say, the MIP mode derivation fusion enhancement technology based on template matching can not only fuse two derived MIP prediction blocks, but also fuse with prediction blocks generated by other derivation technologies based on template matching. This application fuses TMMIP technology with TIMD technology to obtain a method for fusing derived traditional prediction blocks with matrix-based prediction blocks. TIMD uses the idea of ​​template matching at the codec end to derive the optimal traditional intra-frame prediction mode, and the technology can also offset and expand the prediction mode to obtain an updated intra-frame prediction mode. The TMMIP technology also uses the idea of ​​template matching at the codec end to derive the optimal MIP mode. By fusing these two optimal prediction modes, it is possible to take into account the directionality of the traditional prediction block and the unique texture characteristics of the MIP prediction, generate a new prediction block, and improve coding efficiency.

[0378] The encoder iterates through the prediction modes. If the current block is in intra mode, the encoder obtains the sequence-level permission flag, which indicates whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current encoder allows the use of TMMIP technology.

[0379] Exemplarily, the encoder process can be implemented as follows:

[0380] Step 1:

[0381] If sps_tmmip_enable_flag is true, the encoder tries the TMMIP technology, that is, executes step 2; if sps_tmmip_enable_flag is false, the encoder does not try the TMMIP technology, that is, skips step 2 and directly executes step 3.

[0382] Step 2:

[0383] First, the encoder fills the adjacent rows and columns outside the second template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the encoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, all available reconstructed samples are filled in turn; if all reconstructed samples are unavailable, all are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the encoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.

[0384] Secondly, the encoder takes the reconstructed samples outside the filled second template area as input and uses the allowed MIP mode to predict the samples in the second template area.

[0385] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.

[0386] Exemplarily, the specific prediction calculation process includes: the encoder first performs Haar downsampling on the reconstructed samples, for example, the encoder determines the downsampling step size based on the block size. Then, the encoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced ​​after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced ​​after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the encoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the encoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the second template area.

[0387] In addition, the encoder also needs to try the template matching calculation process of TIMD, obtain different interpolation filters according to different prediction mode indexes, and interpolate the reference samples to obtain the predicted samples within the template.

[0388] Next, the encoder calculates the distortion cost based on the predicted samples of the second template area obtained by traversing each MIP mode and the reconstructed samples in the second template area, and records the distortion cost value under each prediction mode and transposition information, and based on the distortion cost value under each prediction mode and transposition information, selects the optimal MIP mode and its corresponding transposition information according to the principle of minimum cost. In addition, the encoder also needs to traverse all intra-frame prediction modes allowed by TIMD, calculate the predicted samples in the template and calculate the distortion cost with the reconstructed samples in the template, and record the optimal prediction mode, suboptimal prediction mode, distortion cost value of the optimal prediction mode and distortion cost value of the suboptimal prediction mode derived by TIMD technology according to the principle of minimum cost.

[0389] Finally, based on the obtained optimal MIP mode and transposition information, the encoder optionally samples the reconstructed samples above and to the left of the current block and concatenates them according to the transposition information as the input vector. The encoder then reads the matrix coefficients of the current mode using the MIP mode as an index and calculates the output vector by combining the input vector and the matrix coefficients. The encoder can perform an output transposition based on the transposition information and upsample the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block.

[0390] For the optimal and suboptimal prediction modes derived from TIMD technology, if neither the optimal nor the suboptimal prediction mode is a mean (DC) or flat (PLANAR) mode, and the distortion cost of the suboptimal prediction mode is less than twice the distortion cost of the optimal prediction mode, the encoder needs to perform a prediction block fusion operation. First, the encoder obtains interpolation filter coefficients based on the optimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the optimal prediction block. Second, the encoder obtains interpolation filter coefficients based on the suboptimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the suboptimal prediction block. Next, the encoder calculates the weights of the optimal and suboptimal prediction blocks using the ratio between the cost values ​​of the optimal and suboptimal prediction modes. Finally, the encoder performs a weighted fusion of the optimal and suboptimal prediction blocks to obtain the prediction block for the current block as the output. If the optimal prediction mode or the suboptimal prediction mode is the mean mode (DC) or the flat mode (PLANAR), or the cost of the suboptimal prediction mode is greater than twice the cost of the optimal prediction mode, the encoder does not need to perform a prediction block fusion operation, and only uses the optimal prediction mode to interpolate and filter the upper and left adjacent reconstructed samples to obtain the optimal prediction block as the optimal TIMD prediction block of the current block.

[0391] Finally, the encoder performs a weighted average of the optimal MIP prediction block and the optimal TIMD prediction block based on the calculated weight value of the optimal MIP mode and the weight value of the prediction mode derived by TIMD technology to obtain a new prediction block, which is the prediction block of the current block.

[0392] In addition, the encoder obtains the rate-distortion cost of the current block, which is recorded as cost1.

[0393] In addition, the encoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the encoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived from the TIMD mode.

[0394] It should be noted that since the template area of ​​the TIMD technology and the second template area (i.e., the template area of ​​the TMMIP technology) can be set to be the same, that is, the template area used to calculate the distortion cost is the same, then the cost information of the template area of ​​the TIMD technology and the cost information of the template area of ​​the TMMIP technology can be equivalent or at the same comparison level. At this time, whether to perform fusion enhancement can also be determined based on the cost information. This application does not make specific restrictions on this.

[0395] Step 3:

[0396] The encoder continues to traverse other intra prediction techniques and calculates the corresponding rate-distortion costs, which are recorded as cost2…costN.

[0397] Step 4:

[0398] If cost1 is the minimum among all rate-distortion costs, then TMMIP is used for the current block, and the encoder sets the TMMIP usage flag for the current block to true and writes it into the bitstream. If cost1 is not the minimum rate-distortion cost, then another intra-frame prediction technique is used for the current block, and the encoder sets the TMMIP usage flag for the current block to false and writes it into the bitstream. It should be understood that the flags or indices of other intra-frame prediction techniques are transmitted according to their definitions and are not elaborated on here.

[0399] Step 5:

[0400] The encoder determines a residual block for the current block based on the predicted block and the original block of the current block. The encoder then performs a base transform on the residual block of the current block and performs a secondary transform on the transform coefficients after the base transform based on the first intra-frame prediction mode. The encoder then performs operations such as quantization, entropy coding, and loop filtering on the transform coefficients after the secondary transform. It should be understood that the specific quantization process can be found in the relevant content above and will not be repeated here to avoid repetition.

[0401] The following describes the relevant solutions of the decoder in this embodiment.

[0402] The decoder parses the block-level type flag. If it is intra mode, it parses or obtains the sequence-level permission flag, which is used to indicate whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current decoder allows the use of TMMIP technology.

[0403] Exemplarily, the decoder process can be implemented as follows:

[0404] Step 1:

[0405] If sps_tmmip_enable_flag is true, the decoder parses the TMMIP usage flag for the current block. Otherwise, the current decoding process does not need to decode the block-level TMMIP usage flag, and the block-level TMMIP usage flag defaults to false. If the TMMIP usage flag for the current block is true, proceed to step 2; otherwise, proceed to step 3.

[0406] Step 2:

[0407] First, the decoder fills the adjacent rows and columns outside the second template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the decoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, all available reconstructed samples are filled in turn; if all reconstructed samples are unavailable, all are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the decoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.

[0408] Secondly, the decoder takes the reconstructed samples outside the filled second template area as input and uses the allowed MIP mode to predict the samples within the second template area.

[0409] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.

[0410] Exemplarily, the specific prediction calculation process includes: the decoder first performs Haar downsampling on the reconstructed samples, for example, the decoder determines the downsampling step size based on the block size. Then, the decoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced ​​after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced ​​after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the decoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the decoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the second template area.

[0411] In addition, the decoder also needs to try the template matching calculation process of TIMD, obtain different interpolation filters according to different prediction mode indexes, and interpolate the reference samples to obtain the predicted samples within the template.

[0412] Next, the decoder calculates the distortion cost based on the predicted samples of the second template area obtained by traversing each MIP mode and the reconstructed samples in the second template area, and records the distortion cost value under each prediction mode and transposition information. Based on the distortion cost value under each prediction mode and transposition information, the decoder selects the optimal MIP mode and its corresponding transposition information according to the principle of minimum cost. In addition, the decoder also needs to traverse all intra-frame prediction modes allowed by TIMD, calculate the predicted samples within the template, and calculate the distortion cost with the reconstructed samples within the template, and record the optimal prediction mode, suboptimal prediction mode, distortion cost value of the optimal prediction mode, and distortion cost value of the suboptimal prediction mode derived by TIMD technology according to the principle of minimum cost.

[0413] Finally, based on the obtained optimal MIP mode and transposition information, the decoder optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them according to the transposition information as the input vector. Using the MIP mode as an index, the decoder reads the matrix coefficients of the current mode and calculates the output vector by combining the input vector and the matrix coefficients. The decoder can perform an output transposition based on the transposition information and upsample the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block.

[0414] For the optimal and suboptimal prediction modes derived from TIMD technology, if neither the optimal nor the suboptimal prediction mode is a mean (DC) or flat (PLANAR) mode, and the distortion cost of the suboptimal prediction mode is less than twice the distortion cost of the optimal prediction mode, the decoder needs to perform prediction block fusion. First, the decoder obtains interpolation filter coefficients based on the optimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the optimal prediction block. Second, the decoder obtains interpolation filter coefficients based on the suboptimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the suboptimal prediction block. Next, the decoder calculates the weights of the optimal and suboptimal prediction blocks using the ratio between the cost values ​​of the optimal and suboptimal prediction modes. Finally, the decoder performs a weighted fusion of the optimal and suboptimal prediction blocks to obtain the prediction block for the current block as output. If the optimal prediction mode or the suboptimal prediction mode is the mean mode (DC) or the flat mode (PLANAR), or the cost of the suboptimal prediction mode is greater than twice the cost of the optimal prediction mode, the decoder does not need to perform the prediction block fusion operation, and only uses the optimal prediction mode to interpolate and filter the upper and left adjacent reconstructed samples to obtain the optimal prediction block as the optimal TIMD prediction block of the current block.

[0415] Finally, the decoder performs a weighted average of the optimal MIP prediction block and the optimal TIMD prediction block based on the calculated weight value of the optimal MIP mode and the weight value of the prediction mode derived by TIMD technology to obtain a new prediction block, which is the prediction block of the current block.

[0416] In addition, the decoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines the first intra-frame prediction mode as the intra-frame prediction mode derived from the TIMD mode.

[0417] Step 3:

[0418] The decoder continues to parse information such as usage identification bits or indexes of other intra-frame prediction technologies, and obtains the final prediction block of the current block based on the parsed information.

[0419] Step 4:

[0420] The decoder parses the bitstream and obtains the frequency domain residual block of the current block (also called frequency domain residual information), and performs inverse quantization and inverse transformation on the frequency domain residual block of the current block (first performing the inverse transformation of the secondary transformation based on the first intra-frame prediction mode and then performing the inverse transformation of the basic transformation or the inverse transformation of the main transformation) to obtain the residual block of the current block (also called the time domain residual block or time domain residual information); then the decoder superimposes the prediction block of the current block with the residual block of the current block to obtain a reconstructed sample block.

[0421] Step 5:

[0422] If all reconstructed sample blocks in the current image are processed through loop filtering and other technologies, the final reconstructed image is obtained.

[0423] Optionally, the reconstructed image can be output as a video or used as a reference for subsequent decoding.

[0424] In this embodiment, the weight calculation process for weighted fusion of TIMD prediction blocks can be found in the description of TIMD technology above and will not be repeated here to avoid repetition. Furthermore, the encoder or decoder can determine whether to perform fusion enhancement based on the optimal prediction mode derived from TIMD. For example, if the optimal prediction mode derived from TIMD is DC or PLANAR, the encoder or decoder may not use fusion enhancement, i.e., only the prediction block generated by the optimal MIP mode derived from TMMIP technology is used as the prediction block for the current block. Furthermore, the size of the second template region used by the encoder or decoder in TMMIP technology can be predefined based on the size of the current block. For example, the definition of the second template region in TMMIP technology can be consistent with or different from the definition of the template region in TIMD technology. For example, if the width of the current block is less than or equal to 8, the height of the upper region adjacent to the current block in the second template region is two rows of samples; otherwise, the height is four rows of samples. Similarly, if the height of the current block is less than or equal to 8, the width of the left region adjacent to the left side of the current block in the second template region is two columns of samples; otherwise, the width is four columns of samples.

[0425] Example 3:

[0426] In this embodiment, the second intra-frame prediction mode mentioned above is the intra-frame prediction mode derived from the DIMD mode, that is, the encoder or decoder can perform intra-frame prediction on the current block based on the optimal MIP mode and the intra-frame prediction mode derived from the DIMD mode for the reconstructed samples in the first template area adjacent to the current block to obtain a predicted block of the current block.

[0427] Similar to Example 2, the TMMIP technology can also be integrated and enhanced with the DIMD technology.

[0428] It should be noted that while both DIMD and TIMD derive prediction modes using traditional intra-frame prediction modes, the resulting prediction modes are not necessarily identical due to different derivation methods. Furthermore, the approach to fusion enhancement using TMMIP and DIMD differs from that of TMMIP and TIMD. For example, since the second template area in TMMIP and TIMD is generally the same size, and the calculated cost information is essentially the sum of absolute transformed differences (SATD), also known as the distortion cost based on the Hadamard transform, TMMIP and TIMD can directly calculate fusion weights based on this cost information. However, the second template area in DIMD is generally different from that in TMMIP (or DIMD), and the criterion for deriving prediction modes using DIMD is based on gradient magnitude, which is not directly equivalent to the SATD cost. Therefore, the weights cannot be simply calculated using the fusion scheme of TMMIP and TIMD.

[0429] The encoder iterates through the prediction modes. If the current block is in intra mode, the encoder obtains the sequence-level permission flag, which indicates whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current encoder allows the use of TMMIP technology.

[0430] Exemplarily, the encoder process can be implemented as follows:

[0431] Step 1:

[0432] If sps_tmmip_enable_flag is true, the encoder tries the TMMIP technology, that is, executes step 2; if sps_tmmip_enable_flag is false, the encoder does not try the TMMIP technology, that is, skips step 2 and directly executes step 3.

[0433] Step 2:

[0434] First, the encoder fills the adjacent rows and columns outside the second template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the encoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, all available reconstructed samples are filled in turn; if all reconstructed samples are unavailable, all are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the encoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.

[0435] Secondly, the encoder takes the reconstructed samples outside the filled second template area as input and uses the allowed MIP mode to predict the samples in the second template area.

[0436] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.

[0437] Exemplarily, the specific prediction calculation process includes: the encoder first performs Haar downsampling on the reconstructed samples, for example, the encoder determines the downsampling step size based on the block size. Then, the encoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced ​​after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced ​​after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the encoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the encoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the second template area.

[0438] In addition, the encoder uses DIMD technology to derive the optimal intra-frame prediction mode, which is the optimal DIMD mode. DIMD technology calculates the gradient value of the reconstructed sample within the first template area based on the Sobel operator, and converts the gradient value according to the angle value of different prediction modes to obtain the amplitude value of the corresponding prediction mode.

[0439] Next, the encoder iterates over the template prediction blocks obtained for each MIP mode, calculates the distortion cost with the reconstructed samples within the template, and records the optimal MIP mode and transposition information based on the principle of minimum cost. Furthermore, the encoder iterates over all allowed intra-frame prediction modes, calculates the amplitude values ​​for each intra-frame prediction mode, and records the optimal DIMD prediction mode based on the principle of maximum amplitude.

[0440] Finally, based on the obtained optimal MIP mode and transposition information, the encoder optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them according to the transposition information as the input vector. Using the MIP mode as an index, the encoder reads the matrix coefficients for the current mode. The output vector is then calculated by combining the input vector and the matrix coefficients. The encoder transposes the output according to the transposition information and upsamples the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block. Furthermore, for the optimal DIMD prediction mode, the encoder obtains the corresponding interpolation filter coefficients and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples for all positions within the current block, which are recorded as the optimal DIMD prediction block. The encoder then performs a weighted average of each prediction sample using the optimal MIP prediction block and the optimal DIMD prediction block according to pre-set weights to obtain a new prediction block, which is the prediction block for the current block.

[0441] In addition, the encoder obtains the rate-distortion cost of the current block, which is recorded as cost1.

[0442] In addition, the encoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the encoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0443] Step 3:

[0444] The encoder continues to traverse other intra prediction techniques and calculates the corresponding rate-distortion costs, which are recorded as cost2…costN.

[0445] Step 4:

[0446] If cost1 is the minimum among all rate-distortion costs, then TMMIP is used for the current block, and the encoder sets the TMMIP usage flag for the current block to true and writes it into the bitstream. If cost1 is not the minimum rate-distortion cost, then another intra-frame prediction technique is used for the current block, and the encoder sets the TMMIP usage flag for the current block to false and writes it into the bitstream. It should be understood that the flags or indices of other intra-frame prediction techniques are transmitted according to their definitions and are not elaborated on here.

[0447] Step 5:

[0448] The encoder determines a residual block for the current block based on the predicted block and the original block of the current block. The encoder then performs a base transform on the residual block of the current block and performs a secondary transform on the transform coefficients after the base transform based on the first intra-frame prediction mode. The encoder then performs operations such as quantization, entropy coding, and loop filtering on the transform coefficients after the secondary transform. It should be understood that the specific quantization process can be found in the relevant content above and will not be repeated here to avoid repetition.

[0449] The following describes the relevant solutions of the decoder in this embodiment.

[0450] The decoder parses the block-level type flag. If it is intra mode, it parses or obtains the sequence-level permission flag, which is used to indicate whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current decoder allows the use of TMMIP technology.

[0451] Exemplarily, the decoder process can be implemented as follows:

[0452] Step 1:

[0453] If sps_tmmip_enable_flag is true, the decoder parses the TMMIP usage flag for the current block. Otherwise, the current decoding process does not need to decode the block-level TMMIP usage flag, and the block-level TMMIP usage flag defaults to false. If the TMMIP usage flag for the current block is true, proceed to step 2; otherwise, proceed to step 3.

[0454] Step 2:

[0455] First, the decoder fills the adjacent rows and columns outside the second template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the decoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, all available reconstructed samples are filled in turn; if all reconstructed samples are unavailable, all are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the decoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.

[0456] Secondly, the decoder takes the reconstructed samples outside the filled second template area as input and uses the allowed MIP mode to predict the samples within the second template area.

[0457] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.

[0458] Exemplarily, the specific prediction calculation process includes: the decoder first performs Haar downsampling on the reconstructed samples, for example, the decoder determines the downsampling step size based on the block size. Then, the decoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced ​​after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced ​​after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the decoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the decoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the second template area.

[0459] In addition, the decoder uses DIMD technology to derive the optimal intra-frame prediction mode, which is the optimal DIMD mode. DIMD technology calculates the gradient value of the reconstructed sample in the first template area based on the Sobel operator, and converts the gradient value according to the angle value of different prediction modes to obtain the amplitude value under the corresponding prediction mode.

[0460] Next, the decoder iterates through the template prediction blocks obtained for each MIP mode, calculates the distortion cost with the reconstructed samples within the template, and records the optimal MIP mode and transposition information based on the principle of minimum cost. Furthermore, the decoder iterates through all allowed intra-frame prediction modes, calculates the amplitude values ​​for each intra-frame prediction mode, and records the optimal DIMD prediction mode based on the principle of maximum amplitude.

[0461] Finally, based on the obtained optimal MIP mode and transposition information, the decoder optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them according to the transposition information as the input vector. Using the MIP mode as an index, the decoder reads the matrix coefficients for the current mode. The decoder then calculates the input vector and the matrix coefficients to obtain the output vector. The decoder transposes the output according to the transposition information and upsamples the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block. Furthermore, for the optimal DIMD prediction mode, the decoder obtains the corresponding interpolation filter coefficients and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples for all positions within the current block, which are recorded as the optimal DIMD prediction block. The decoder then performs a weighted average of each prediction sample using the optimal MIP prediction block and the optimal DIMD prediction block according to pre-set weights to obtain a new prediction block, which is the prediction block for the current block.

[0462] In addition, the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the decoder determines that the first intra-frame prediction mode is the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0463] Step 3:

[0464] The decoder continues to parse information such as usage identification bits or indexes of other intra-frame prediction technologies, and obtains the final prediction block of the current block based on the parsed information.

[0465] Step 4:

[0466] The decoder parses the bitstream and obtains the frequency domain residual block of the current block (also called frequency domain residual information), and performs inverse quantization and inverse transformation on the frequency domain residual block of the current block (first performing the inverse transformation of the secondary transformation based on the first intra-frame prediction mode and then performing the inverse transformation of the basic transformation or the inverse transformation of the main transformation) to obtain the residual block of the current block (also called the time domain residual block or time domain residual information); then the decoder superimposes the prediction block of the current block with the residual block of the current block to obtain a reconstructed sample block.

[0467] Step 5:

[0468] If all reconstructed sample blocks in the current image are processed through loop filtering and other technologies, the final reconstructed image is obtained.

[0469] Optionally, the reconstructed image can be output as a video or used as a reference for subsequent decoding.

[0470] In this embodiment, the calculation process of the optimal DIMD prediction block can refer to the content described in the above introduction to the DIMD technology. To avoid repetition, it will not be repeated here. In addition, the fusion weight of the optimal MIP prediction block and the optimal DIMD prediction block can be a pre-set value, such as the optimal MIP prediction block accounts for 5 / 9, and the optimal DIMD prediction block accounts for 4 / 9. Of course, in other alternative embodiments, the fusion weight of the optimal MIP prediction block and the optimal DIMD prediction block can also be other values, and this application does not make specific restrictions on this. In addition, the above-mentioned second template area and the above-mentioned first template area can be the same or different, and this application does not make specific restrictions on this.

[0471] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will no longer describe the various possible combinations separately. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of ​​the present application, they should also be regarded as the contents disclosed in the present application. It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0472] The method embodiment of the present application is described in detail above. The following describes in detail an apparatus embodiment of the present application in conjunction with Figures 11 to 13.

[0473] FIG11 is a schematic block diagram of a decoder 500 according to an embodiment of the present application.

[0474] As shown in FIG11 , the decoder 500 may include:

[0475] A parsing unit 510 is configured to parse a bitstream of a current sequence to obtain a first transform coefficient of a current block;

[0476] The transformation unit 520 is configured to:

[0477] determining a first intra prediction mode;

[0478] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0479] Performing a first transform on the first transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a second transform coefficient of the current block;

[0480] Performing a second transform on the second transform coefficient to obtain a residual block of the current block;

[0481] The reconstruction unit 530 is configured to determine a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.

[0482] In some embodiments, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode before upsampling; or, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode after upsampling.

[0483] In some embodiments, the transform unit 520 is specifically configured to:

[0484] The first intra prediction mode is determined based on a prediction mode used to predict the current block.

[0485] In some embodiments, the transform unit 520 is specifically configured to:

[0486] If the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0487] In some embodiments, the transform unit 520 is specifically configured to:

[0488] If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived from the TIMD mode.

[0489] In some embodiments, the transform unit 520 is specifically configured to:

[0490] If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0491] In some embodiments, the reconstruction unit 530 is further configured to:

[0492] determining a second intra prediction mode;

[0493] The second intra prediction mode includes any one of the following: a suboptimal MIP mode for predicting the current block, an intra prediction mode derived from the DIMD mode using the reconstructed samples within the first template area, and an intra prediction mode derived from the TIMD mode;

[0494] The current block is predicted based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block.

[0495] In some embodiments, the reconstruction unit 530 is specifically configured to:

[0496] Predicting the current block based on the optimal MIP mode to obtain a first predicted block;

[0497] Predicting the current block based on the second intra prediction mode to obtain a second prediction block;

[0498] Based on the weight of the optimal MIP mode and the weight of the second intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.

[0499] In some embodiments, the reconstruction unit 530 performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra prediction mode to obtain the prediction block of the current block, and is further configured to:

[0500] If the prediction mode used to predict the current block includes the optimal MIP mode and a suboptimal MIP mode used to predict the current block or an intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the second intra prediction mode based on a distortion cost of the optimal MIP mode and a distortion cost of the second intra prediction mode;

[0501] If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, it is determined that the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are both preset values.

[0502] In some embodiments, the transform unit 520 is specifically configured to:

[0503] Parsing the code stream of the current sequence to obtain a first identifier;

[0504] If the first identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, the second intra-frame prediction mode is determined.

[0505] In some embodiments, the transform unit 520 is specifically configured to:

[0506] If the first identifier is used to indicate that the optimal MIP mode and the second intra prediction mode are allowed to be used to predict the image block in the current sequence, parsing the code stream to obtain a second identifier;

[0507] If the second flag is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block, the second intra-frame prediction mode is determined.

[0508] In some embodiments, the reconstruction unit 530 is further configured to:

[0509] Determining the optimal MIP mode based on the distortion costs of multiple MIP modes;

[0510] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a second template area adjacent to the current block using the multiple MIP modes.

[0511] In some embodiments, the second template region is the same as or different from the first template region.

[0512] In some embodiments, the reconstruction unit 530 is specifically configured to:

[0513] Predicting samples within the second template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode;

[0514] The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.

[0515] In some embodiments, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the reconstruction unit 530 is further configured to:

[0516] Obtaining a MIP mode used by a neighboring block adjacent to the current block;

[0517] The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.

[0518] In some embodiments, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the reconstruction unit 530 is further configured to:

[0519] Filling a reference area adjacent to the second template area with reconstruction samples to obtain reference rows and reference columns of the second template area;

[0520] Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the second template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes;

[0521] Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the second template area.

[0522] In some embodiments, the reconstruction unit 530 is specifically configured to:

[0523] Downsampling the reference row and the reference column to obtain an input vector;

[0524] Taking the input vector as input, predicting samples in the second template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes;

[0525] Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.

[0526] In some embodiments, the reconstruction unit 530 is specifically configured to:

[0527] The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the second template area.

[0528] FIG12 is a schematic block diagram of an encoder 600 according to an embodiment of the present application.

[0529] As shown in FIG12 , the encoder 600 may include:

[0530] The residual unit 610 is used to obtain a residual block of the current block in the current sequence;

[0531] The transformation unit 620 is configured to:

[0532] Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;

[0533] determining a first intra prediction mode;

[0534] The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode;

[0535] Performing a fourth transform on the third transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a fourth transform coefficient of the current block;

[0536] The encoding unit 630 is configured to encode the fourth transform coefficient.

[0537] In some embodiments, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode before upsampling; or, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode after upsampling.

[0538] In some embodiments, the transform unit 620 is specifically configured to:

[0539] The first intra prediction mode is determined based on a prediction mode used to predict the current block.

[0540] In some embodiments, the transform unit 620 is specifically configured to:

[0541] If the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

[0542] In some embodiments, the transform unit 620 is specifically configured to:

[0543] If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived from the TIMD mode.

[0544] In some embodiments, the transform unit 620 is specifically configured to:

[0545] If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

[0546] In some embodiments, the residual unit 610 is specifically configured to:

[0547] determining a second intra prediction mode;

[0548] The second intra prediction mode includes any one of the following: a suboptimal MIP mode for predicting the current block, an intra prediction mode derived from the DIMD mode using the reconstructed samples within the first template area, and an intra prediction mode derived from the TIMD mode;

[0549] Predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block;

[0550] Based on the prediction block of the current block, a residual block of the current block is obtained.

[0551] In some embodiments, the residual unit 610 is specifically configured to:

[0552] Predicting the current block based on the optimal MIP mode to obtain a first predicted block;

[0553] Predicting the current block based on the second intra prediction mode to obtain a second prediction block;

[0554] Based on the weight of the optimal MIP mode and the weight of the second intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.

[0555] In some embodiments, the residual unit 610 performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra prediction mode to obtain the prediction block of the current block, and is further configured to:

[0556] If the prediction mode used to predict the current block includes the optimal MIP mode and a suboptimal MIP mode used to predict the current block or an intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the second intra prediction mode based on a distortion cost of the optimal MIP mode and a distortion cost of the second intra prediction mode;

[0557] If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, it is determined that the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are both preset values.

[0558] In some embodiments, the residual unit 610 is specifically configured to:

[0559] Get the first identifier;

[0560] If the first identifier is used to indicate that it is allowed to use the optimal MIP mode and the second intra-frame prediction mode to predict the image block in the current sequence, determining the second intra-frame prediction mode;

[0561] The encoding unit 630 is specifically configured to:

[0562] The fourth transform coefficient and the first identifier are encoded.

[0563] In some embodiments, the residual unit 610 is specifically configured to:

[0564] If the first identifier is used to indicate that it is permitted to use the optimal MIP mode and the second intra prediction mode to predict the image block in the current sequence, predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a first rate-distortion cost;

[0565] Predicting the current block based on at least one intra prediction mode to obtain at least one rate-distortion cost;

[0566] If the first rate-distortion cost is less than or equal to a minimum value of the at least one rate-distortion cost, determining a prediction block obtained by predicting the current block based on the optimal MIP mode and the second intra prediction mode as the prediction block of the current block;

[0567] The encoding unit 630 is specifically configured to:

[0568] encoding the fourth transform coefficient, the first identifier, and the second identifier;

[0569] Wherein, if the first rate-distortion cost is less than or equal to the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block; if the first rate-distortion cost is greater than the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are not allowed to be used to predict the current block.

[0570] In some embodiments, the residual unit 610 is further configured to:

[0571] Determining the optimal MIP mode based on the distortion costs of multiple MIP modes;

[0572] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a second template area adjacent to the current block using the multiple MIP modes.

[0573] In some embodiments, the second template region is the same as or different from the first template region.

[0574] In some embodiments, the residual unit 610 is specifically configured to:

[0575] Predicting samples within the second template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode;

[0576] The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.

[0577] In some embodiments, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the residual unit 610 is further configured to:

[0578] Obtaining a MIP mode used by a neighboring block adjacent to the current block;

[0579] The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.

[0580] In some embodiments, before determining the optimal MIP mode based on the distortion costs of multiple MIP modes, the residual unit 610 is further configured to:

[0581] Filling a reference area adjacent to the second template area with reconstruction samples to obtain reference rows and reference columns of the second template area;

[0582] Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the second template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes;

[0583] Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the second template area.

[0584] In some embodiments, the residual unit 610 is specifically configured to:

[0585] Downsampling the reference row and the reference column to obtain an input vector;

[0586] Taking the input vector as input, predicting samples in the second template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes;

[0587] Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.

[0588] In some embodiments, the residual unit 610 is specifically configured to:

[0589] The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the second template area.

[0590] It should be understood that the device embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, the decoder 500 shown in Figure 11 can correspond to the corresponding subject in the method 300 for executing the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the decoder 500 are respectively for implementing the corresponding processes in various methods such as method 300. Similarly, the encoder 600 shown in Figure 12 can correspond to the corresponding subject in the method 400 for executing the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the encoder 600 are respectively for implementing the corresponding processes in various methods such as method 400.

[0591] It should also be understood that the various units in the decoder 500 or encoder 600 involved in the embodiment of the present application can be separately or all merged into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the decoder 500 or encoder 600 can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the decoder 500 or encoder 600 involved in the embodiment of the present application can be constructed by running a computer program (including program code) that can execute each step involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), etc., to construct the decoder 500 or encoder 600 involved in the embodiment of the present application, and to implement the encoding method or decoding method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device via the computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application.

[0592] In other words, the units mentioned above can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software in a decoding processor. Optionally, the software can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in combination with its hardware.

[0593] FIG13 is a schematic structural diagram of an electronic device 700 provided in an embodiment of the present application.

[0594] As shown in Figure 13, the electronic device 700 includes at least a processor 710 and a computer-readable storage medium 720. The processor 710 and the computer-readable storage medium 720 may be connected via a bus or other means. The computer-readable storage medium 720 is used to store a computer program 721, which includes computer instructions. The processor 710 is used to execute the computer instructions stored in the computer-readable storage medium 720. The processor 710 is the computing core and control core of the electronic device 700. It is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.

[0595] As an example, the processor 710 may also be referred to as a central processing unit (CPU). The processor 710 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.

[0596] As an example, the computer-readable storage medium 720 may be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 710. Specifically, the computer-readable storage medium 720 includes, but is not limited to: volatile memory and / or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0597] In one implementation, the electronic device 700 may be an encoder or encoding framework involved in an embodiment of the present application; a first computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the first computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the encoding method provided in an embodiment of the present application; in other words, the first computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed. To avoid repetition, they are not repeated here.

[0598] In one implementation, the electronic device 700 may be a decoder or decoding framework involved in an embodiment of the present application; a second computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the second computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the decoding method provided in an embodiment of the present application; in other words, the second computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed, which will not be repeated here to avoid repetition.

[0599] According to another aspect of the present application, an embodiment of the present application further provides a coding and decoding system, including the encoder and decoder mentioned above.

[0600] According to another aspect of the present application, an embodiment of the present application further provides a computer-readable storage medium (Memory), which is a memory device in the electronic device 700 for storing programs and data. For example, a computer-readable storage medium 720. It can be understood that the computer-readable storage medium 720 here can include both the built-in storage medium in the electronic device 700 and, of course, the extended storage medium supported by the electronic device 700. The computer-readable storage medium provides a storage space, which stores the operating system of the electronic device 700. In addition, one or more computer instructions suitable for being loaded and executed by the processor 710 are also stored in the storage space. These computer instructions can be one or more computer programs 721 (including program codes).

[0601] According to another aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. For example, computer program 721 is provided. In this case, data processing device 700 may be a computer, processor 710 may read the computer instructions from computer-readable storage medium 720, and processor 710 may execute the computer instructions, causing the computer to perform the encoding method or decoding method provided in the various optional embodiments described above.

[0602] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the functions of the embodiment of the present application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.

[0603] Those skilled in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0604] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A decoding method, characterized in that: include: Parsing the code stream of the current sequence to obtain the first transform coefficient of the current block; determining a first intra prediction mode; The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode; Performing a first transform on the first transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a second transform coefficient of the current block; Performing a second transform on the second transform coefficient to obtain a residual block of the current block; A reconstructed block of the current block is determined based on the prediction block of the current block and the residual block of the current block.

2. The method according to claim 1, characterized in that The output vector of the optimal MIP mode is a vector output by the optimal MIP mode before upsampling; or, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode after upsampling.

3. The method according to claim 1 or 2, characterized in that The determining of the first intra-frame prediction mode includes: The first intra prediction mode is determined based on a prediction mode used to predict the current block.

4. The method according to claim 3, characterized in that The determining the first intra prediction mode based on the prediction mode used to predict the current block includes: If the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

5. The method according to claim 3, characterized in that The determining the first intra prediction mode based on the prediction mode used to predict the current block includes: If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived from the TIMD mode.

6. The method according to claim 3, characterized in that The determining the first intra prediction mode based on the prediction mode used to predict the current block includes: If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: determining a second intra prediction mode; The second intra prediction mode includes any one of the following: a suboptimal MIP mode for predicting the current block, an intra prediction mode derived from the DIMD mode using the reconstructed samples within the first template area, and an intra prediction mode derived from the TIMD mode; The current block is predicted based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block.

8. The method according to claim 7, characterized in that The predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block includes: Predicting the current block based on the optimal MIP mode to obtain a first predicted block; Predicting the current block based on the second intra prediction mode to obtain a second prediction block; Based on the weight of the optimal MIP mode and the weight of the second intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.

9. The method according to claim 8, characterized in that Before performing weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra prediction mode to obtain the prediction block of the current block, the method further includes: If the prediction mode used to predict the current block includes the optimal MIP mode and a suboptimal MIP mode used to predict the current block or an intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the second intra prediction mode based on a distortion cost of the optimal MIP mode and a distortion cost of the second intra prediction mode; If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, it is determined that the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are both preset values.

10. The method according to any one of claims 7 to 9, characterized in that The determining of the second intra-frame prediction mode includes: Parsing the code stream of the current sequence to obtain a first identifier; If the first identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, the second intra-frame prediction mode is determined.

11. The method according to claim 10, characterized in that The determining of the second intra-frame prediction mode includes: If the first identifier is used to indicate that the optimal MIP mode and the second intra prediction mode are allowed to be used to predict the image block in the current sequence, parsing the code stream to obtain a second identifier; If the second flag is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block, the second intra-frame prediction mode is determined.

12. The method according to any one of claims 1 to 11, characterized in that The method further comprises: Determining the optimal MIP mode based on the distortion costs of multiple MIP modes; The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a second template area adjacent to the current block using the multiple MIP modes.

13. The method according to claim 12, characterized in that The second template region is the same as or different from the first template region.

14. The method according to claim 12 or 13, characterized in that The determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: Predicting samples within the second template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode; The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.

15. The method according to any one of claims 12 to 14, characterized in that Before determining the optimal MIP mode based on the distortion costs of the multiple MIP modes, the method further includes: Obtaining a MIP mode used by a neighboring block adjacent to the current block; The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.

16. The method according to any one of claims 12 to 15, characterized in that Before determining the optimal MIP mode based on the distortion costs of the multiple MIP modes, the method further includes: Filling a reference area adjacent to the second template area with reconstruction samples to obtain reference rows and reference columns of the second template area; Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the second template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes; Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the second template area.

17. The method according to claim 16, characterized in that The step of taking the reference row and the reference column as input and using the multiple MIP modes to respectively predict samples within the second template area to obtain multiple prediction blocks corresponding to the multiple MIP modes includes: Downsampling the reference row and the reference column to obtain an input vector; Taking the input vector as input, predicting samples in the second template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.

18. The method according to any one of claims 12 to 17, characterized in that The determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the second template area.

19. A coding method, characterized in that: The method is applicable to an encoder, and the method comprises: Get the residual block of the current block in the current sequence; Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block; determining a first intra prediction mode; The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode; Performing a fourth transform on the third transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a fourth transform coefficient of the current block; The fourth transform coefficients are encoded.

20. The method according to claim 19, wherein The output vector of the optimal MIP mode is a vector output by the optimal MIP mode before upsampling; or, the output vector of the optimal MIP mode is a vector output by the optimal MIP mode after upsampling.

21. The method according to claim 19 or 20, characterized in that The determining of the first intra-frame prediction mode includes: The first intra prediction mode is determined based on a prediction mode used to predict the current block.

22. The method according to claim 21, characterized in that The determining the first intra prediction mode based on the prediction mode used to predict the current block includes: If the prediction mode used to predict the current block includes the optimal MIP mode and the suboptimal MIP mode for predicting the current block, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the output vector of the optimal MIP mode.

23. The method according to claim 21, characterized in that The determining the first intra prediction mode based on the prediction mode used to predict the current block includes: If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived from the TIMD mode.

24. The method according to claim 21, characterized in that The determining the first intra prediction mode based on the prediction mode used to predict the current block includes: If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, then the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the prediction block of the current block, or the first intra-frame prediction mode is determined to be the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area.

25. The method according to any one of claims 19 to 22, characterized in that The obtaining of the residual block of the current block in the current sequence includes: determining a second intra prediction mode; The second intra prediction mode includes any one of the following: a suboptimal MIP mode for predicting the current block, an intra prediction mode derived from the DIMD mode using the reconstructed samples within the first template area, and an intra prediction mode derived from the TIMD mode; Predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block; Based on the prediction block of the current block, a residual block of the current block is obtained.

26. The method according to claim 25, characterized in that The predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block includes: Predicting the current block based on the optimal MIP mode to obtain a first predicted block; Predicting the current block based on the second intra prediction mode to obtain a second prediction block; Based on the weight of the optimal MIP mode and the weight of the second intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.

27. The method according to claim 26, characterized in that Before performing weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the second intra prediction mode to obtain the prediction block of the current block, the method further includes: If the prediction mode used to predict the current block includes the optimal MIP mode and a suboptimal MIP mode used to predict the current block or an intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the second intra prediction mode based on a distortion cost of the optimal MIP mode and a distortion cost of the second intra prediction mode; If the prediction mode used to predict the current block includes the optimal MIP mode and the intra-frame prediction mode derived using the DIMD mode for the reconstructed samples within the first template area, it is determined that the weight of the optimal MIP mode and the weight of the second intra-frame prediction mode are both preset values.

28. The method according to any one of claims 25 to 27, characterized in that The determining of the second intra-frame prediction mode includes: Get the first identifier; If the first identifier is used to indicate that it is allowed to use the optimal MIP mode and the second intra-frame prediction mode to predict the image block in the current sequence, determining the second intra-frame prediction mode; The encoding of the fourth transform coefficient includes: The fourth transform coefficient and the first identifier are encoded.

29. The method according to claim 28, characterized in that The predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a prediction block of the current block includes: If the first identifier is used to indicate that it is permitted to use the optimal MIP mode and the second intra prediction mode to predict the image block in the current sequence, predicting the current block based on the optimal MIP mode and the second intra prediction mode to obtain a first rate-distortion cost; Predicting the current block based on at least one intra prediction mode to obtain at least one rate-distortion cost; If the first rate-distortion cost is less than or equal to a minimum value of the at least one rate-distortion cost, determining a prediction block obtained by predicting the current block based on the optimal MIP mode and the second intra prediction mode as the prediction block of the current block; The encoding of the fourth transform coefficient and the first identifier includes: encoding the fourth transform coefficient, the first identifier, and the second identifier; Wherein, if the first rate-distortion cost is less than or equal to the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are allowed to be used to predict the current block; if the first rate-distortion cost is greater than the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the second intra-frame prediction mode are not allowed to be used to predict the current block.

30. The method according to any one of claims 19 to 29, characterized in that The method further comprises: Determining the optimal MIP mode based on the distortion costs of multiple MIP modes; The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a second template area adjacent to the current block using the multiple MIP modes.

31. The method according to claim 30, wherein The second template region is the same as or different from the first template region.

32. The method according to claim 30 or 31, characterized in that The determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: Predicting samples within the second template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode; The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.

33. The method according to any one of claims 30 to 32, characterized in that Before determining the optimal MIP mode based on the distortion costs of the multiple MIP modes, the method further includes: Obtaining a MIP mode used by a neighboring block adjacent to the current block; The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.

34. The method according to any one of claims 30 to 33, characterized in that Before determining the optimal MIP mode based on the distortion costs of the multiple MIP modes, the method further includes: Filling a reference area adjacent to the second template area with reconstruction samples to obtain reference rows and reference columns of the second template area; Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the second template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes; Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the second template area.

35. The method according to claim 34, wherein The step of taking the reference row and the reference column as input and using the multiple MIP modes to respectively predict samples within the second template area to obtain multiple prediction blocks corresponding to the multiple MIP modes includes: Downsampling the reference row and the reference column to obtain an input vector; Taking the input vector as input, predicting samples in the second template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.

36. The method according to any one of claims 30 to 35, characterized in that The determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the second template area.

37. A decoder, characterized in that include: A parsing unit, configured to parse a code stream of a current sequence to obtain a first transform coefficient of a current block; Transformation unit, used to: determining a first intra prediction mode; The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode; Performing a first transform on the first transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a second transform coefficient of the current block; Performing a second transform on the second transform coefficient to obtain a residual block of the current block; A reconstruction unit is configured to determine a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.

38. An encoder, characterized in that include: The residual unit is used to obtain the residual block of the current block in the current sequence; Transformation unit, used to: Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block; determining a first intra prediction mode; The first intra prediction mode includes any one of the following: using an intra prediction mode derived from a decoder-side intra mode deriving DIMD mode for a prediction block of the current block, using an intra prediction mode derived from the DIMD mode for an output vector of an optimal matrix-based intra prediction MIP mode for predicting the current block, using an intra prediction mode derived from the DIMD mode for reconstructed samples within a first template region adjacent to the current block, and using an intra prediction mode derived from a template-based intra mode deriving TIMD mode; Performing a fourth transform on the third transform coefficient based on a transform set corresponding to the first intra prediction mode to obtain a fourth transform coefficient of the current block; A coding unit is used to encode the fourth transformation coefficient.

39. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein the computer program, when executed by the processor, implements the method according to any one of claims 1 to 18 or the method according to any one of claims 19 to 36.

40. A computer-readable storage medium, characterized in that Used to store a computer program, the computer program causing a computer to execute the method according to any one of claims 1 to 18 or the method according to any one of claims 19 to 36.

41. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 18 or the method according to any one of claims 19 to 36 is implemented.

42. A code stream, characterized in that The code stream is the code stream in the method according to any one of claims 1 to 18 or the code stream generated by the method according to any one of claims 19 to 36.

Images