Decoding method, encoding method, decoder, and encoder
Patent Information
- Application Number
- CN202280094571.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-08
- Publication Date
- 2024-11-19
AI Technical Summary
Existing digital video compression technologies still fall short in improving compression efficiency, especially when high video resolution is required, necessitating more efficient decompression technologies.
The optimal prediction mode is determined by the distortion cost of matrix-based intra-prediction MIP mode. The DIMD mode is derived by combining the decoder-side intra-prediction mode and the TIMD mode is derived by template-based intra-prediction mode. Video block prediction and reconstruction are performed by fusing the optimal MIP mode and the first intra-prediction mode.
It effectively reduces bit overhead at the coding unit level, improves compression efficiency and performance, balances prediction accuracy and diversity, and enhances the compression efficiency of video encoding and decoding.
Smart Images

Figure CN118985131A_ABST
Abstract
Description
Decoding method, encoding method, decoder and encoder Technical Field
[0001] The embodiments of the present application relate to the technical field of image and video coding and decoding, and more specifically, to a decoding method, an encoding method, a decoder, and an encoder. Background Art
[0002] Digital video compression technology primarily compresses large amounts of digital video data for easier transmission and storage. With the surge in Internet video usage and increasing demand for higher-quality video, while existing digital video compression standards are capable of implementing video decompression technology, there is still a need for better digital video decompression technologies to improve compression efficiency.
[0003] Summary of the Invention
[0004] The embodiments of the present application provide a decoding method, an encoding method, a decoder, and an encoder, which can improve compression efficiency.
[0005] In a first aspect, the present application provides a decoding method, comprising:
[0006] Parse the code stream to obtain the residual block of the current block in the current sequence;
[0007] determining an optimal MIP mode for predicting the current block based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0008] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0009] determining a first intra prediction mode;
[0010] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0011] Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0012] A reconstructed block of the current block is obtained based on the residual block of the current block and the prediction block of the current block.
[0013] In a second aspect, the present application provides an encoding method, comprising:
[0014] determining an optimal MIP mode for predicting a current block in a current sequence based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0015] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0016] determining a first intra prediction mode;
[0017] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0018] Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0019] Obtaining a residual block of the current block based on the prediction block of the current block and the original block of the current block;
[0020] The residual block of the current block is encoded to obtain a code stream of the current sequence.
[0021] In a third aspect, the present application provides a decoder, comprising:
[0022] A parsing unit, configured to parse a bitstream to obtain a residual block of a current block in a current sequence;
[0023] Prediction unit, used to:
[0024] determining an optimal MIP mode for predicting the current block based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0025] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0026] determining a first intra prediction mode;
[0027] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0028] Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0029] The reconstruction unit is configured to obtain a reconstructed block of the current block based on the residual block of the current block and the prediction block of the current block.
[0030] In a fourth aspect, the present application provides an encoder, comprising:
[0031] Prediction unit, used to:
[0032] determining an optimal MIP mode for predicting a current block in a current sequence based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0033] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0034] determining a first intra prediction mode;
[0035] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0036] Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0037] a residual unit, configured to obtain a residual block of the current block based on a prediction block of the current block and an original block of the current block;
[0038] The encoding unit is configured to encode the residual block of the current block to obtain a code stream of the current sequence.
[0039] In a fifth aspect, the present application provides a decoder, comprising:
[0040] a processor adapted to implement computer instructions; and,
[0041] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the decoding method in the above-mentioned first aspect or its various implementations.
[0042] In one implementation, there are one or more processors and one or more memories.
[0043] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0044] In a sixth aspect, the present application provides an encoder, comprising:
[0045] a processor adapted to implement computer instructions; and,
[0046] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the encoding method in the above-mentioned second aspect or its various implementations.
[0047] In one implementation, there are one or more processors and one or more memories.
[0048] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0049] In the seventh aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the decoding method involved in the first aspect or the encoding method involved in the second aspect.
[0050] In an eighth aspect, the present application provides a code stream, which is the code stream involved in the first aspect or the code stream involved in the second aspect.
[0051] Based on the above technical solution, the decoder predicts the current block based on the optimal MIP mode and the first intra-frame prediction mode, and designs the optimal MIP mode to be the optimal MIP mode for predicting the current block determined based on the distortion costs of multiple MIP modes, and designs the first intra-frame prediction mode to include at least one of the following: a suboptimal MIP mode for predicting the current block determined based on the distortion costs of the multiple MIP modes, an intra-frame prediction mode derived from the DIMD mode, and an intra-frame prediction mode derived from the TIMD mode; this is equivalent to avoiding the decoder from obtaining the MIP mode by parsing the code stream. Compared with the traditional MIP technology, it can effectively reduce the bit overhead at the coding unit level, thereby improving the decompression efficiency.
[0052] In addition, by fusing the optimal MIP mode and the first intra-frame prediction mode, the optimal prediction mode calculated based on the rate-distortion cost is avoided from being completely replaced by the optimal MIP mode, which can take into account both prediction accuracy and prediction diversity, thereby improving decompression performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] FIG1 is a schematic block diagram of a coding framework provided in an embodiment of the present application.
[0054] FIG2 is a schematic diagram of the MIP mode provided in an embodiment of the present application.
[0055] FIG3 is a schematic diagram of a prediction model derived based on DIDM provided in an embodiment of the present application.
[0056] FIG4 is a schematic diagram of deriving a prediction block based on DIMD according to an embodiment of the present application.
[0057] FIG5 is a schematic diagram of a template used by the TIMD provided in an embodiment of the present application.
[0058] FIG6 is a schematic block diagram of a decoding framework provided in an embodiment of the present application.
[0059] FIG7 is a schematic flowchart of a decoding method provided in an embodiment of the present application.
[0060] FIG8 is a schematic flowchart of the encoding method provided in an embodiment of the present application.
[0061] FIG9 is a schematic block diagram of a decoder provided in an embodiment of the present application.
[0062] FIG10 is a schematic block diagram of an encoder provided in an embodiment of the present application.
[0063] FIG11 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.
[0065] The solution provided in the embodiment of the present application can be applied to the field of digital video coding technology, for example, including but not limited to: image coding and decoding field, video coding and decoding field, hardware video coding and decoding field, dedicated circuit video coding and decoding field and real-time video coding and decoding field. In addition, the solution provided in the embodiment of the present application can be combined with the audio and video coding standard (Audio Video Coding Standard, AVS), the second generation AVS standard (AVS2) or the third generation AVS standard (AVS3). For example, including but not limited to: H.264 / Audio Video Coding (AVC) standard, H.265 / High Efficiency Video Coding (HEVC) standard and H.266 / Versatile Video Coding (VVC) standard. In addition, the solution provided in the embodiment of the present application can be used for lossy compression of images, and can also be used for lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.
[0066] Video coding standards all use a block-based hybrid coding framework. Each video frame is divided into square maximum coding units (LCUs) or coding tree units (CTUs) of the same size (e.g., 128x128, 64x64, etc.). Each LCU or CTU can be divided into rectangular coding units (CUs) according to a rule. Coding units may also be divided into prediction units (PUs) and transform units (TUs). The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-frame prediction and inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. Because adjacent pixels within a video frame are strongly correlated, intra-frame prediction is used in video coding to eliminate spatial redundancy between adjacent pixels. Intra-frame prediction only references information from the same frame to predict pixel information within the current block. Because adjacent frames in a video have strong similarities, inter-frame prediction is used in video coding and decoding to eliminate temporal redundancy between them, thereby improving coding efficiency. Inter-frame prediction refers to image information from different frames and uses motion estimation to search for the motion vector information that best matches the current block. Transformation converts the predicted image block to the frequency domain, redistributing the energy. Combined with quantization, it removes information that is insensitive to the human eye, eliminating visual redundancy. Entropy coding can eliminate character redundancy based on the current context model and the probabilistic information of the binary bitstream.
[0067] During digital video encoding, the encoder may first read a black-and-white or color image from the original video sequence and then encode the black-and-white or color image. A black-and-white image may include pixels of the luminance component, while a color image may include pixels of the chrominance component. Optionally, the color image may also include pixels of the luminance component. The color format of the original video sequence may be a luminance-chrominance (YCbCr, YUV) format or a red-green-blue (RGB) format. Specifically, after reading the black-and-white or color image, the encoder divides it into blocks and uses intra-frame prediction or inter-frame prediction on the current block to generate a prediction block for the current block. The prediction block is subtracted from the original block to obtain a residual block. The residual block is transformed and quantized to obtain a quantization coefficient matrix. The quantization coefficient matrix is entropy encoded and output to the bitstream. During digital video decoding, the decoder uses intra-frame prediction or inter-frame prediction on the current block to generate a prediction block for the current block. Furthermore, the decoder parses the bitstream to obtain a quantization coefficient matrix, dequantizes and inversely transforms the quantization coefficient matrix to obtain a residual block. The prediction block and the residual block are added together to obtain a reconstructed block. The reconstructed blocks can be used to compose a reconstructed image. The decoding end performs loop filtering on the reconstructed image based on an image or a block to obtain a decoded image.
[0068] The current block may be a current coding unit (CU) or a current prediction unit (PU), etc.
[0069] It should be noted that the encoder also requires similar operations as the decoder to obtain a decoded image. The decoded image can serve as a reference frame for subsequent frames for inter-frame prediction. The block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode information or parameter information determined by the encoder are output to the bitstream if necessary. The decoder determines the same block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode information or parameter information as the encoder by parsing and analyzing existing information, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical. The decoded image obtained by the encoder is also commonly referred to as a reconstructed image. During prediction, the current block can be divided into prediction units, and during transform, the current block can be divided into transform units. The division of prediction units and transform units can be the same or different. Of course, the above is only the basic process of a video codec within a block-based hybrid coding framework. With technological advancement, some modules of the framework or some steps of the process may be optimized. This application applies to the basic process of a video codec within this block-based hybrid coding framework.
[0070] To facilitate understanding, the coding framework provided in this application is briefly introduced first.
[0071] FIG1 is a schematic block diagram of a coding framework 100 provided in an embodiment of the present application.
[0072] As shown in FIG1 , the coding framework 100 may include an intra-frame prediction unit 180, an inter-frame prediction unit 170, a residual unit 110, a transform and quantization unit 120, an entropy coding unit 130, an inverse transform and inverse quantization unit 140, and a loop filter unit 150. Optionally, the coding framework 100 may further include a decoded image buffer unit 160. The coding framework 100 may also be referred to as a hybrid framework coding mode.
[0073] The intra-frame prediction unit 180 or the inter-frame prediction unit 170 may predict the image block to be encoded and output a prediction block. The residual unit 110 may calculate a residual block based on the prediction block and the image block to be encoded, i.e., the difference between the prediction block and the image block to be encoded. The transform and quantization unit 120 is configured to perform operations such as transform and quantization on the residual block to remove information that is insensitive to the human eye, thereby eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform and quantization unit 120 may be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform and quantization unit 120 may be referred to as a frequency residual block or a frequency-domain residual block. After receiving the transform and quantization coefficients output by the transform and quantization unit 120, the entropy coding unit 130 may output a bitstream based on the transform and quantization coefficients. For example, the entropy coding unit 130 may eliminate character redundancy based on a target context model and probability information of the binary bitstream. For example, the entropy coding unit 130 may be configured for context-based adaptive binary arithmetic entropy coding (CABAC). The entropy coding unit 130 may also be referred to as a header information coding unit. Optionally, in this application, the image block to be encoded may also be referred to as an original image block or a target image block, the prediction block may also be referred to as a prediction image block or an image prediction block, or may also be referred to as a prediction signal or prediction information, and the reconstruction block may also be referred to as a reconstructed image block or an image reconstruction block, or may also be referred to as a reconstruction signal or reconstruction information. In addition, for the encoding end, the image block to be encoded may also be referred to as a coding block or a coding image block, and for the decoding end, the image block to be encoded may also be referred to as a decoding block or a decoding image block. The image block to be encoded may be a CTU or a CU.
[0074] The encoding framework 100 calculates the residual between the prediction block and the image block to be encoded, generates a residual block, and transmits the residual block to the decoder through processes such as transformation and quantization. Correspondingly, the decoder receives and parses the bitstream, then performs inverse transformation and inverse quantization to obtain a residual block. The predicted block, obtained by the decoder, is then superimposed on the residual block to create a reconstructed block.
[0075] It should be noted that the inverse transform and inverse quantization unit 140, the loop filter unit 150, and the decoded image buffer unit 160 in the coding framework 100 can be used to form a decoder. This means that the intra-frame prediction unit 180 or the inter-frame prediction unit 170 can predict the image block to be coded based on the existing reconstructed block, thereby ensuring that the encoder and decoder have a consistent understanding of the reference frame. In other words, the encoder can replicate the decoder's processing loop and produce the same prediction as the decoder. Specifically, the quantized transform coefficients are inversely transformed and inversely quantized by the inverse transform and inverse quantization unit 140 to replicate the approximate residual block on the decoder side. This approximate residual block, after adding the prediction block, can pass through the loop filter unit 150 to smooth out block artifacts caused by block-based processing and quantization. The image block output by the loop filter unit 150 can be stored in the decoded image buffer unit 160 for use in subsequent image prediction.
[0076] It should be understood that FIG1 is merely an example of the present application and should not be construed as limiting the present application.
[0077] For example, the loop filtering unit 150 in the coding framework 100 may include a deblocking filter (DBF) and a sample adaptive offset filter (SAO). The function of DBF is to remove the blocking effect, and the function of SAO is to remove the ringing effect. In other embodiments of the present application, the coding framework 100 may adopt a neural network-based loop filtering algorithm to improve the compression efficiency of the video. In other words, the coding framework 100 may be a video coding hybrid framework based on a deep learning neural network. In one implementation, a convolutional neural network-based model may be used to calculate the result of pixel filtering on the basis of a deblocking filter and a sample adaptive offset filter. The network structure of the loop filtering unit 150 on the luminance component and the chrominance component may be the same or different. Considering that the luminance component contains more visual information, the luminance component may also be used to guide the filtering of the chrominance component to improve the reconstruction quality of the chrominance component.
[0078] The following describes the relevant contents of intra-frame prediction.
[0079] Intra-frame prediction only references information from the same frame, predicting pixel information within the image block to be encoded, thereby eliminating spatial redundancy. The frame used for intra-frame prediction can be an I-frame. For example, based on the left-to-right, top-to-bottom encoding order, the image block to be encoded can reference the upper left, upper, and left image blocks as reference information to predict the image block to be encoded. The image block to be encoded then serves as reference information for the next image block, allowing prediction of the entire image. If the input digital video is in a color format, such as YUV 4:2:0, each four pixels in each image frame of the digital video are composed of four Y components and two UV components. The encoding framework can encode the Y component (i.e., the luminance block) and the UV component (i.e., the chrominance block) separately. Similarly, the decoding end can also perform decoding based on the format.
[0080] The following is an introduction to other intra-frame prediction modes involved in this application:
[0081] (1) Matrix based Intra Prediction (MIP) mode.
[0082] The process involved in the MIP mode can be divided into three main steps, namely the downsampling process, the matrix multiplication process, and the upsampling process. Specifically, the downsampling process first downsamples spatially adjacent reconstructed samples, then uses the downsampled sample sequence as the input vector of the matrix multiplication process, that is, the output vector of the downsampling process is used as the input vector of the matrix multiplication process, multiplied by a pre-set matrix and added with a bias vector, and the calculated sample vector is output; finally, the output vector of the matrix multiplication process is used as the input vector of the upsampling process, and the final prediction block is obtained by upsampling.
[0083] FIG2 is a schematic diagram of the MIP mode provided in an embodiment of the present application.
[0084] As shown in Figure 2, the MIP mode obtains the upper adjacent down-sampled reconstructed sample vector bdry by averaging the reconstructed samples above the current coding unit during the down-sampling process. top , by averaging the left adjacent reconstructed samples, we get the left adjacent down-sampling reconstructed sample vector bdry left . Get bdry top and bdry left After that, it is used as the input vector bdry of the matrix multiplication process red Specifically, based on bdry red The top row vector bdry top red 、bdry left 、A k ·bdryred +b k Get the sample vector, where A k is a pre-set matrix, b k is a pre-set bias vector, and k is the index of the MIP mode. After obtaining the sample vector, it is upsampled by linear interpolation to obtain a prediction sample block that matches the number of samples in the actual coding unit.
[0085] In other words, to predict a block of width W and height H, MIP requires H reconstructed pixels in the column to the left of the current block and W reconstructed pixels in the row above the current block as input. MIP generates the prediction block using three steps: reference pixel averaging, matrix vector multiplication, and interpolation. The core of MIP is matrix multiplication, which can be thought of as a process of generating a prediction block from input pixels (reference pixels) using a matrix multiplication approach. MIP offers a variety of matrices, and different prediction methods are reflected in different matrices; using different matrices for the same input pixels will produce different results. The reference pixel averaging and interpolation processes are a compromise between performance and complexity. For larger blocks, reference pixel averaging can achieve a similar downsampling effect, allowing the input to fit into a smaller matrix, while interpolation achieves an upsampling effect. This eliminates the need to provide MIP matrices for every block size; instead, matrices of one or a few specific sizes are sufficient. With increasing demands for compression performance and improved hardware capabilities, even more complex MIPs may appear in next-generation standards.
[0086] The MIP mode can be simplified from a neural network. For example, the matrix it uses can be derived from training. Therefore, the MIP mode has strong generalization capabilities and prediction results that traditional prediction modes cannot achieve. The MIP mode can be a model derived from a neural network-based intra-frame prediction model through multiple hardware and software complexity simplifications. Based on a large number of training samples, multiple prediction modes represent a variety of models and parameters, which can effectively cover the texture conditions of natural sequences.
[0087] MIP is somewhat similar to the planar mode, but it is obviously more complex and more flexible than the planar mode.
[0088] It should be noted that the number of MIP modes may be different for coding units of different block sizes. For example, for a coding unit of size 4x4, the MIP mode has 16 prediction modes; for a coding unit of size 8x8, width equal to 4 or height equal to 4, the MIP mode has 8 prediction modes; for coding units of other sizes, the MIP mode has 6 prediction modes. At the same time, the MIP mode has a transposition function. For the prediction mode that meets the current size, the MIP mode can try to transpose the calculation on the encoder side. Therefore, the MIP mode not only requires a flag to indicate whether the current coding unit uses the MIP mode, but also, if the current coding unit uses the MIP mode, an additional transposition flag needs to be transmitted to the decoder.
[0089] The transposed flag of the MIP is binarized using a fixed length (FL) encoding method with a length of 1. The mode index of the MIP is binarized using a truncated binary (TB) encoding method.
[0090] (2) Decoder side Intra Mode Derivation (DIMD) prediction mode.
[0091] The main core point of the DIMD prediction mode is that the decoder uses the same method as the encoder to derive the intra-frame prediction mode, thereby avoiding transmitting the intra-frame prediction mode index of the current coding unit in the bit stream and achieving the purpose of saving bit overhead.
[0092] The specific process of the DIDM prediction model can be divided into the following two main steps:
[0093] Step 1: Derive the prediction model.
[0094] The encoder uses the Sobel operator to calculate the gradient histogram for each prediction mode. The operating area is the three rows of adjacent reconstructed samples above the current block, the three columns of adjacent reconstructed samples to the left, and the corresponding adjacent reconstructed samples to the upper left. By calculating the gradient histogram of the aforementioned L-shaped area, the prediction mode with the largest amplitude and the prediction mode with the second largest amplitude in the histogram can be obtained.
[0095] FIG3 is a schematic diagram of a prediction model derived based on DIDM provided in an embodiment of the present application.
[0096] As shown in FIG3(a), DIMD uses pixels in a template within a reconstructed region (reconstructed pixels to the left and above the current block) to derive a prediction mode. For example, the template may include three adjacent rows of reconstructed samples above the current block, three adjacent columns of reconstructed samples to the left, and corresponding adjacent reconstructed samples to the upper left of the current block. Based on this, multiple gradient values can be determined within the template according to a window (such as the window shown in FIG3(a) or FIG3(b)). Each gradient value is suitable for matching an intra-frame prediction mode (IPM) that is compatible with its gradient direction. Based on this, the encoder can use the prediction mode matched to the largest and second largest gradient values among the multiple gradient values as the derived prediction mode. For example, as shown in (b) of FIG3 , for a 4×4 block, all pixels for which gradient values need to be determined are analyzed and a corresponding gradient histogram is obtained. For example, as shown in (c) of FIG3 , for blocks of other sizes, all pixels for which gradient values need to be determined are analyzed and a corresponding gradient histogram is obtained. Finally, the prediction modes corresponding to the largest and second largest gradients in the gradient histogram are used as the derived prediction modes.
[0097] Of course, the gradient histogram in this application is only an example for determining the derived prediction mode. In specific implementation, it can be implemented in a variety of simple forms, and this application does not specifically limit this. In addition, this application does not limit the method of statistically analyzing the gradient histogram. For example, the gradient histogram can be statistically analyzed using the Sobel operator or other methods.
[0098] Step 2: Derive the prediction block.
[0099] FIG4 is a schematic diagram of deriving a prediction block based on DIMD according to an embodiment of the present application.
[0100] As shown in Figure 4, the encoder can weight the prediction values of the three intra-frame prediction modes (planar mode and two intra-frame prediction modes derived based on DIMD). The codec uses the same prediction block derivation method to obtain the prediction block of the current block. Assuming that the prediction mode corresponding to the maximum gradient value is prediction mode 1 and the prediction mode corresponding to the second largest gradient value is prediction mode 2, the encoder determines the following two conditions:
[0101] 1. The gradient of prediction mode 2 is not 0;
[0102] 2. Prediction mode 1 and prediction mode 2 are neither planar mode nor DC prediction mode.
[0103] If neither of the above two conditions holds, only prediction mode 1 is used to calculate the predicted sample values for the current block, applying the normal prediction process to prediction mode 1. Otherwise, if both conditions hold, a weighted average is used to derive the prediction block for the current block. Specifically, the planar mode accounts for 1 / 3 of the weight, and the remaining 2 / 3 is the combined weight of prediction modes 1 and 2. For example, the gradient amplitude of prediction mode 1 is divided by the sum of the gradient amplitudes of prediction mode 1 and prediction mode 2 to determine the weight of prediction mode 1, while the gradient amplitude of prediction mode 2 is divided by the sum of the gradient amplitudes of prediction mode 1 and prediction mode 2 to determine the weight of prediction mode 2. Finally, the prediction blocks obtained based on the three prediction modes—prediction blocks 1, 2, and 3 obtained based on planar mode, prediction mode 1, and prediction mode 2, respectively—are weighted averaged to obtain the prediction block for the current coding unit. The decoder uses the same steps to obtain the prediction block.
[0104] In other words, the specific weight calculation in step 2 above is as follows:
[0105] Weight(PLANAR)=1 / 3;
[0106] Weight(mode1)=2 / 3*(amp1 / (amp1+amp2));
[0107] Weight(mode2)=1-Weight(PLANAR)-Weight(mode1);
[0108] Wherein, mode1 and mode2 represent prediction mode 1 and prediction mode 2 respectively, amp1 and amp2 represent the gradient amplitude value of prediction mode 1 and the gradient amplitude value of prediction mode 2 respectively. The DIMD prediction mode requires a flag to be transmitted to the decoder, which is used to indicate whether the current coding unit uses the DIDM prediction mode.
[0109] Of course, the above-mentioned weighted averaging method is only an example of the present application and should not be understood as a limitation of the present application.
[0110] In summary, DIMD uses gradient analysis of reconstructed pixels to select the intra prediction mode, and can weight two intra prediction modes plus the planar mode based on the analysis results. The advantage of DIMD is that if DIMD mode is selected for the current block, the bitstream no longer needs to indicate the specific intra prediction mode used. Instead, the decoder automatically derives the selected intra prediction mode through the above process, which reduces overhead to a certain extent.
[0111] (3) Template based Intra Mode Derivation (TIMD) prediction mode.
[0112] For the TIMD prediction mode, the codec also uses the method of deriving the prediction mode by operation to save the overhead of the transmission mode index. The TIMD prediction mode can be understood as two main parts. First, the cost information of each prediction mode is calculated based on the template. The prediction mode corresponding to the minimum cost and the second minimum cost will be selected. The prediction mode corresponding to the minimum cost is recorded as prediction mode 1, and the prediction mode corresponding to the second minimum cost is recorded as prediction mode 2. If the ratio of the value of the second minimum cost (costMode2) to the value of the minimum cost (costMode1) meets the preset conditions, such as costMode2<2*costMode1, the prediction blocks corresponding to prediction mode 1 and prediction mode 2 can be weighted and fused according to the corresponding weights of prediction mode 1 and prediction mode 2 to obtain the final prediction block.
[0113] Exemplarily, the weights corresponding to prediction mode 1 and prediction mode 2 are determined according to the following method:
[0114] weight1=costMode2 / (costMode1+costMode2);
[0115] weight2=1-weight1;
[0116] Where weight1 is the weight of the prediction block corresponding to prediction mode 1, and weight2 is the weight of the prediction block corresponding to prediction mode 2. However, if the ratio of the value of the second smallest cost, costMode2, to the value of the smallest cost, costMode1, does not meet the preset conditions, weighted fusion between the prediction blocks is not performed, and the prediction block corresponding to prediction mode 1 is the TIMD prediction block.
[0117] It should be noted that when using the TIMD prediction mode for intra prediction of the current block, if the reconstructed sample template of the current block does not contain available adjacent reconstructed samples, the TIMD prediction mode selects the planar mode for intra prediction of the current block, that is, no unweighted fusion is performed. Similar to the DIMD prediction mode, the TIMD prediction mode requires a flag to be transmitted to the decoder to indicate whether the current coding unit uses the TIMD prediction mode.
[0118] The encoder or decoder calculates the cost information of each prediction mode mainly as follows: based on the reconstructed samples adjacent to the upper or left side of the template area, intra-frame prediction is performed on the samples within the template area. The prediction process is the same as the original intra-frame prediction mode. For example, when using the DC mode to perform intra-frame prediction on the samples within the template area, the mean of the entire coding unit is calculated. For example, when using the angle prediction mode to perform intra-frame prediction on the samples within the template area, the corresponding interpolation filter is selected according to the mode and the predicted samples are interpolated according to the rules. At this time, the distortion between the predicted samples and the reconstructed samples in the template area can be calculated based on the predicted samples and the reconstructed samples in the area, which is the cost information of the current prediction mode.
[0119] FIG5 is a schematic diagram of a template used by the TIMD provided in an embodiment of the present application.
[0120] As shown in Figure 5, if the current block is a coding unit with a width equal to M and a height equal to N, the codec can predict the samples in the template area of the current block based on the reconstructed samples in the reference template (Reference of template) of the current block selected in the coding unit with a width equal to 2(M+L1)+1 and a height equal to 2(N+L2)+1. Of course, if the template area of the current block does not contain available adjacent reconstructed samples, the TIMD prediction mode selects the plane mode to perform intra-frame prediction on the current block. For example, the available adjacent reconstructed samples can be the samples adjacent to the left and top sides of the current CU in Figure 5, that is, there are no available reconstructed samples in the diagonal filling area. In other words, if there are no available reconstructed samples in the diagonal filling area, the TIMD prediction mode selects the plane mode to perform intra-frame prediction on the current block.
[0121] It should be noted that, except for edge cases, when encoding and decoding the current block, reconstructed values can theoretically be obtained for the left and top sides of the current block. This means that the template of the current block contains available adjacent reconstructed samples. In specific implementations, the decoder can use a specific intra prediction mode to predict on the template and compare the predicted value with the reconstructed value to obtain the cost of the intra prediction mode on the template. Examples include SAD, SATD, and SSE. Because the template and the current block are adjacent, the reconstructed samples within the template are correlated with the pixels within the current block. Therefore, the performance of a prediction mode on the template can be used to estimate the performance of that prediction mode on the current block. TIMD predicts several candidate intra prediction modes on the template, obtains their costs on the template, and replaces the one or two intra prediction modes with the lowest cost as the intra prediction value for the current block. If the cost difference between two intra prediction modes on the template is not significant, taking a weighted average of the prediction values of the two intra prediction modes can improve compression performance. Optionally, the weights of the prediction values of the two prediction modes are related to the aforementioned costs, for example, the weights are inversely proportional to the costs.
[0122] In summary, TIMD uses the prediction performance of intra-frame prediction modes on a template to select the appropriate intra-frame prediction mode, and can weight the two intra-frame prediction modes based on their cost on the template. The advantage of TIMD is that if TIMD mode is selected for the current block, the bitstream no longer needs to indicate the specific intra-frame prediction mode used. Instead, the decoder automatically derives the selected intra-frame prediction mode through the above process, which reduces overhead to a certain extent.
[0123] Through the above brief introduction to several intra-frame prediction modes, it is not difficult to find that the technical principles of the DIMD prediction mode are similar to those of the TIMD prediction mode. Both use the decoder to perform the same operations as the encoder to infer the prediction mode of the current coding unit. This prediction mode can save the transmission of the prediction mode index when the complexity is acceptable, thereby saving overhead and improving compression efficiency. However, due to the limitations of the reference information and the fact that there is not much improvement in the prediction quality, the DIMD prediction mode and the TIMD prediction mode are better in areas with consistent texture characteristics over a large area. If the texture changes slightly or the template area cannot be covered, the prediction effect of this prediction mode is poor.
[0124] In addition, regardless of the DIMD prediction mode or the TIMD prediction mode, the prediction blocks obtained based on multiple traditional prediction modes are fused or weighted. The fusion of prediction blocks can generate effects that cannot be achieved by a single prediction mode. Although the DIMD prediction mode introduces the planar mode as an additional weighted prediction mode to increase the spatial correlation between adjacent reconstructed samples and prediction samples, thereby improving the prediction effect of intra-frame prediction, due to the relatively simple prediction principle of the planar mode, for some prediction blocks with obvious difference between the upper right corner and the lower left corner, using the planar mode as an additional weighted prediction mode may have a counterproductive effect.
[0125] FIG6 is a schematic block diagram of a decoding framework 200 provided in an embodiment of the present application.
[0126] As shown in FIG6 , the decoding framework 200 may include an entropy decoding unit 210 , an inverse transform and dequantization unit 220 , a residual unit 230 , an intra-frame prediction unit 240 , an inter-frame prediction unit 250 , a loop filtering unit 260 , and a decoded image buffer unit 270 .
[0127] After receiving and parsing the bitstream, the entropy decoding unit 210 obtains a prediction block and a frequency domain residual block. The frequency domain residual block is subjected to inverse transformation and inverse quantization by the inverse transform and inverse quantization unit 220 to obtain a time domain residual block. The residual unit 230 superimposes the prediction block predicted by the intra-frame prediction unit 240 or the inter-frame prediction unit 250 onto the time domain residual block that has been inverse transformed and inverse quantized by the inverse transform and inverse quantization unit 220 to obtain a reconstructed block. For example, the intra-frame prediction unit 240 or the inter-frame prediction unit 250 can obtain the prediction block by decoding header information of the bitstream.
[0128] FIG7 is a schematic flow chart of a decoding method 300 provided in an embodiment of the present application. It should be understood that the decoding method 300 can be executed by a decoder. For example, it can be applied to the decoding framework 200 shown in FIG6 . For ease of description, the following description uses a decoder as an example.
[0129] As shown in FIG7 , the decoding method 300 may include some or all of the following:
[0130] S310, parsing the bitstream to obtain a residual block of a current block in a current sequence;
[0131] S320, determining an optimal MIP mode for predicting the current block based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0132] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0133] S330, determining a first intra prediction mode;
[0134] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0135] S340, predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0136] S350 : Obtain a reconstructed block of the current block based on the residual block of the current block and the prediction block of the current block.
[0137] Exemplarily, the present application may also refer to S320 to S340 (i.e., the process in which the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes and predicts the current block based on the optimal MIP mode and the first intra-frame prediction mode) as Template Matching MIP (TMMIP) technology, a MIP prediction mode derivation method based on template matching, or TMMIP fusion enhancement technology; that is, after obtaining the residual block of the current block, the decoder can enhance the performance of the prediction process of the current block based on the derived optimal MIP mode and the first intra-frame prediction mode. In other words, the TMMIP technology can utilize the optimal MIP prediction mode and at least one of the following: a suboptimal MIP prediction mode, an intra-frame prediction mode derived from a TIMD mode, and an intra-frame prediction mode derived from a DIMD mode to enhance the performance of the prediction process of the current block.
[0138] In this embodiment, the decoder predicts the current block based on the optimal MIP mode and the first intra-frame prediction mode, and designs the optimal MIP mode to be the optimal MIP mode for predicting the current block determined based on the distortion costs of multiple MIP modes, and designs the first intra-frame prediction mode to include at least one of the following: a suboptimal MIP mode for predicting the current block determined based on the distortion costs of the multiple MIP modes, an intra-frame prediction mode derived from the DIMD mode, and an intra-frame prediction mode derived from the TIMD mode; this is equivalent to avoiding the decoder from obtaining the MIP mode by parsing the bitstream. Compared with traditional MIP technology, it can effectively reduce the bit overhead at the coding unit level, thereby improving decompression efficiency.
[0139] In other words, traditional MIP technology has a higher bit overhead than other intra-frame prediction tools. It requires not only a flag to indicate whether MIP technology is used, but also a flag to indicate whether MIP transposition is used. Finally, and this is the largest overhead, it requires the use of truncated binary code to represent the MIP prediction mode. MIP is a technology simplified based on neural network technology and is significantly different from traditional interpolation filter prediction technology. For some special textures, although MIP prediction is more effective than traditional intra-frame prediction mode, its large flag overhead is a drawback of MIP technology. Taking a 4x4 coding unit as an example, there are 16 prediction samples, but the bit overhead includes 1 MIP use flag, 1 MIP transposition flag, and 5 or 6 bits of truncated binary flag. In view of this, the present application utilizes a method in which the decoder autonomously determines the optimal MIP mode for predicting the current block and determines the intra-frame prediction mode for the current block based on the optimal MIP mode. This can save up to 5 or 6 bits of overhead, effectively reducing the bit overhead at the coding unit level and thereby improving decompression efficiency.
[0140] Furthermore, saving up to 5 or 6 bits of overhead per block requires that the template matching-based prediction mode derivation algorithm be very accurate. If the MIP prediction mode calculated by the fast template matching algorithm differs from the MIP prediction mode calculated by the encoder using rate-distortion optimization and the original sequence, it may not be the optimal choice. Therefore, the performance of the template matching-based MIP prediction mode selection depends on the accuracy of the match, that is, the accuracy rate. However, the accuracy of both the traditional intra-frame prediction mode template-based derivation algorithm and the inter-frame template matching-based derivation algorithm is unsatisfactory. Although performance has been significantly improved, the large amount of additional block-level bit overhead caused by the template matching algorithm has made it difficult for subsequent technologies to achieve higher performance by relying solely on the template matching algorithm. In view of this, the present application fuses the optimal MIP mode and the first intra-frame prediction mode, that is, performs a fusion prediction for the current block based on the optimal MIP mode and the first intra-frame prediction mode. This avoids completely replacing the optimal prediction mode calculated based on the rate-distortion cost with the optimal MIP mode, thus balancing prediction accuracy and prediction diversity, and thus improving decompression performance.
[0141] In addition, it is worth noting that the distortion cost involved in the decoder in this application is different from the rate-distortion cost (RDcost) involved in the encoder. The rate-distortion cost is the distortion cost used by the encoder to determine a certain intra-frame prediction technology among multiple intra-frame prediction technologies. The rate-distortion cost can be the cost value obtained by comparing the distorted image and the original image. Since the decoder cannot obtain the original image, the distortion cost involved in the decoder can be the distortion cost between the reconstructed sample and the predicted sample, such as the sum of absolute transformed differences (SATD) between the reconstructed sample and the predicted sample or other costs that can be used to calculate the difference between the reconstructed sample and the predicted sample.
[0142] The beneficial effects of the solution provided by the present application are described below with reference to the test results in Tables 1 and 2. Table 1 shows the test results of the solution of the present application when the first intra-frame prediction mode is designed as a suboptimal MIP mode for predicting the current block determined based on the distortion costs of the multiple MIP modes, and Table 2 shows the test results of the solution of the present application when the first intra-frame prediction mode is designed as an intra-frame prediction mode derived from the TIMD mode.
[0143] Table 1
[0144]
[0145] Table 2
[0146]
[0147]
[0148] As shown in Table 1 and Table 2, the test results show that both the MIP fusion scheme and the TIMD fusion enhancement scheme can provide excellent performance. The most obvious performance is in category A1, where the two technologies have an average performance improvement of 0.15 and 0.12 respectively. Category A1 is a 4K test sequence, which was a category that was in urgent need of performance improvement at the time. This result effectively proves the technology's ability to improve coding efficiency. A negative delta bit rate (BD-rate) indicates a performance improvement based on the solution provided by this application compared to the test results of ECM2.0. It is worth noting that the TIMD prediction mode integrated in ECM2.0 is more complex than that of ECM1.0 and only has a 0.4% performance gain. This application can achieve significant performance gains, especially for 4K sequences, without increasing decoder complexity. Furthermore, due to server load, even if encoding and decoding times fluctuate slightly, theoretically, decoding time will remain essentially unchanged.
[0149] In some embodiments, the S320 may include:
[0150] Parsing the code stream of the current sequence to obtain a first identifier;
[0151] If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0152] Exemplarily, if the value of the first identifier is a first numerical value, it is used to indicate that the image blocks in the current sequence are allowed to be predicted using the optimal MIP mode and the first intra-frame prediction mode; if the value of the first identifier is a second numerical value, it is used to indicate that the image blocks in the current sequence are not allowed to be predicted using the optimal MIP mode and the first intra-frame prediction mode. In one implementation, the first numerical value is 1 and the second numerical value is 0. In another implementation, the first numerical value is 0 and the second numerical value is 1. Of course, the first numerical value and the second numerical value may also be other numerical values, which is not limited in this application.
[0153] Exemplarily, if the first identifier is true, it is used to indicate that the image blocks in the current sequence are allowed to be predicted using the optimal MIP mode and the first intra-frame prediction mode; if the first identifier is false, it is used to indicate that the image blocks in the current sequence are not allowed to be predicted using the optimal MIP mode and the first intra-frame prediction mode.
[0154] Exemplarily, the decoder parses the block-level identifier, and if the current block adopts the intra-frame prediction mode, parses or obtains the first identifier, and if the first identifier is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0155] Exemplarily, the first flag is recorded as sps_timd_enable_flag. At this time, the decoder parses or obtains sps_timd_enable_flag. If the sps_timd_enable_flag is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0156] Exemplarily, the first identifier is a sequence-level identifier.
[0157] It should be noted that the first identifier is used to identify that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the image blocks in the current sequence, and may also be replaced by a description with a similar or identical meaning. For example, in other alternative embodiments, the first identifier is used to identify that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the image blocks in the current sequence, and may also be replaced by any one of the following: the first identifier is used to identify that the intra-frame prediction mode of the image blocks in the current sequence is allowed to be determined using the TMMIP technology, the first identifier is used to identify that the intra-frame prediction of the image blocks in the current sequence is allowed to be performed using the TMMIP technology, the first identifier is used to identify that the image blocks in the current sequence are allowed to use the TMMIP technology, and the first identifier is used to identify that the MIP mode determined based on the multiple MIP modes is allowed to be used to predict the image blocks in the current sequence.
[0158] In addition, in other alternative embodiments, when TMMIP technology is combined with other technologies, the permission flag of the other technology can also be used to indirectly indicate whether the current sequence allows the use of TMMIP technology. For example, using TIMD technology as an example, if the first flag is used to indicate that the current sequence allows the use of TIMD technology, it means that the current sequence also allows the use of TMMIP technology; or, if the first flag is used to indicate that the current sequence allows the use of TIMD technology, it means that the current sequence allows the use of both TIMD technology and TMMIP technology, thereby further saving bit overhead.
[0159] In some embodiments, if the first identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, the code stream is parsed to obtain the second identifier; if the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0160] Exemplarily, the decoder parses the block-level identifier. If the current block adopts the intra-frame prediction mode, the decoder parses or obtains the first identifier. If the first identifier is true, the decoder parses or obtains the second identifier. If the second identifier is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0161] Exemplarily, if the third value of the second identifier is set, it is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block; if the fourth value of the second identifier is set, it is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are not allowed to be used to predict the current block. In one implementation, the third value is 1 and the fourth value is 0. In another implementation, the third value is 0 and the fourth value is 1. Of course, the third value and the fourth value may also be other values, and this application does not specifically limit this.
[0162] Exemplarily, if the second identifier is true, it is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block; if the second identifier is false, it is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are not allowed to be used to predict the current block.
[0163] Exemplarily, the first identifier is recorded as sps_timd_enable_flag, and the second identifier is recorded as cu_timd_enable_flag. At this time, the decoder parses or obtains sps_timd_enable_flag. If the sps_timd_enable_flag is true, the decoder can parse or obtain cu_timd_enable_flag. If the cu_timd_enable_flag is true, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0164] Exemplarily, the second identifier is a block-level identifier or a coding unit-level identifier.
[0165] It should be noted that the second identifier is used to identify that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block, and may also be replaced by a description with a similar or identical meaning. For example, in other alternative embodiments, the second identifier is used to identify that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block, and may also be replaced by any one of the following: the second identifier is used to identify that the intra-frame prediction mode of the current block is allowed to be determined using the TMMIP technology, the second identifier is used to identify that the intra-frame prediction of the current block is allowed to be performed using the TMMIP technology, the second identifier is used to identify that the image block in the current block is allowed to use the TMMIP technology, and the second identifier is used to identify that the MIP mode determined based on the multiple MIP modes is allowed to be used to predict the current block.
[0166] Furthermore, in alternative embodiments, when TMMIP is combined with other technologies, the permission flag of the other technologies can be used to indirectly indicate whether the current block is allowed to use TMMIP. For example, using TIMD as an example, if the second flag is used to indicate that the current block is allowed to use TIMD, then it indicates that the current block is also allowed to use TMMIP. Alternatively, if the second flag is used to indicate that the current block is allowed to use TIMD, then it indicates that the current block is allowed to use both TIMD and TMMIP, thereby further saving bit overhead.
[0167] In addition, when the decoding end parses the second identifier, it can parse the second identifier before parsing the residual block of the current block, or it can parse the second identifier after parsing the residual block of the current block. This application does not make specific restrictions on this.
[0168] In some embodiments, the method 300 may further include:
[0169] determining an arrangement order of the multiple MIP modes based on the distortion costs of the MIP modes;
[0170] Determining, based on the arrangement order of the multiple MIP modes, a coding mode used by the optimal MIP mode;
[0171] The code stream of the current sequence is decoded based on the encoding mode used by the optimal MIP mode to obtain an index of the optimal MIP mode.
[0172] Exemplarily, before the decoder determines the optimal MIP mode for predicting the current block based on the distortion costs of multiple MIP modes, it needs to calculate the distortion cost of each MIP mode in the multiple MIP modes, and sort the multiple MIP modes according to the distortion cost of each MIP mode. The MIP mode with the lowest cost is the optimal prediction result.
[0173] It should be noted that, for traditional MIP technology, the index of the MIP mode is usually binarized by truncating the binary. This encoding method is closer to equal probability encoding, that is, it divides all prediction modes into two segments, one segment is represented by N codewords, and the other is represented by N+1 codewords. In view of this, in this application, before the decoder determines the optimal MIP mode for predicting the current block based on the distortion costs of multiple MIP modes, it can first calculate the distortion cost of each MIP mode in the multiple MIP modes and sort the multiple MIP modes according to the distortion cost of each MIP mode. Finally, the decoder can choose to use a more flexible variable-length coding method based on the sorting of the multiple MIP modes. Compared with the equal probability coding method, the flexible setting of the MIP mode encoding method is conducive to saving the bit overhead of the index of the MIP mode.
[0174] In some embodiments, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
[0175] Illustratively, N can be any value greater than or equal to 1.
[0176] Exemplarily, the arrangement order is the order obtained by the decoder arranging the multiple MIP modes in order of distortion cost from small to large, and the codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable-length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
[0177] In this embodiment, since the smaller the distortion cost of the MIP mode, the greater the probability that the encoder uses it to perform intra-frame prediction on the current block, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is designed to be smaller than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the encoding method used by the first n MIP modes is designed to be a variable-length encoding method and the encoding method used by the MIP modes after the nth MIP mode is designed to be a truncated binary encoding method; this is equivalent to the MIP mode used by the encoder with a high probability using a shorter codeword length or a variable-length encoding method, which can save the bit overhead of the MIP mode index and improve the decompression performance.
[0178] In some embodiments, before S330, the method 300 may further include:
[0179] If the first intra-frame prediction mode is the suboptimal MIP mode, the decoder determines whether to adopt the suboptimal MIP mode to predict the current block based on the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode; if it is determined not to adopt the suboptimal MIP mode, the decoder may directly predict the current block based on the optimal MIP mode; if it is determined to adopt the suboptimal MIP mode, the decoder may predict the current block based on the optimal MIP mode and the suboptimal MIP mode to obtain a prediction block of the current block.
[0180] Exemplarily, when the first intra-frame prediction mode is a suboptimal MIP mode, if the ratio between the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode is less than or equal to a preset ratio, the decoder may directly predict the current block based on the optimal MIP mode to obtain a predicted block for the current block. Alternatively, when the first intra-frame prediction mode is a suboptimal MIP mode, if the ratio between the distortion cost of the suboptimal MIP mode and the distortion cost of the optimal MIP mode is greater than or equal to a preset ratio, the decoder may directly predict the current block based on the optimal MIP mode to obtain a predicted block for the current block. For example, if the distortion cost of the suboptimal MIP mode is greater than or equal to a certain multiple (e.g., twice) of the distortion cost of the optimal MIP mode, it can be interpreted that the suboptimal MIP mode already has a large distortion and is not suitable for the current block, that is, the fusion enhancement technology can be omitted and only the optimal MIP mode can be used to predict the current block.
[0181] In this embodiment, the decoder determines whether to adopt the suboptimal MIP mode to predict the current block based on the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode. This is equivalent to the decoder determining whether to adopt the suboptimal MIP mode to enhance the performance of the optimal MIP mode based on the distortion cost of the optimal MIP mode and the distortion cost of the suboptimal MIP mode. This avoids carrying an identifier for determining whether to adopt the suboptimal MIP mode to enhance the performance of the optimal MIP mode in the bitstream, saves bit overhead, and can further enhance decompression performance.
[0182] In some embodiments, the S340 may include:
[0183] The decoder predicts the current block based on the optimal MIP mode to obtain a first prediction block; and predicts the current block based on the first intra-frame prediction mode to obtain a second prediction block; then, the decoder performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the first intra-frame prediction mode to obtain a prediction block for the current block.
[0184] Exemplarily, the decoder may directly perform intra-frame prediction on the current block based on the optimal MIP mode to obtain the first prediction block. In addition, the decoder may directly obtain the optimal prediction mode and the suboptimal prediction mode based on the TIMD mode, perform prediction on the current block, and obtain the second prediction block. For example, if neither the optimal prediction mode nor the suboptimal prediction mode is a DC mode (also known as a mean mode) or a plane mode (also known as a flat mode), and the distortion cost of the suboptimal prediction mode is less than twice the distortion cost of the optimal prediction mode, a prediction block fusion operation is required; that is, the decoder may first perform intra-frame prediction on the current block according to the optimal prediction mode to obtain the optimal prediction block; secondly, perform intra-frame prediction on the current block according to the suboptimal prediction mode to obtain the suboptimal prediction block; then use the ratio between the distortion cost of the optimal prediction mode and the distortion cost of the suboptimal prediction mode to calculate the weight value of the optimal prediction block and the weight value of the suboptimal prediction block; finally, the optimal prediction block and the suboptimal prediction block are weightedly fused to obtain the second prediction block. For another example, if the optimal prediction mode or the suboptimal prediction mode is a planar mode or a DC mode, or if the distortion cost of the optimal prediction mode is greater than twice the distortion cost of the optimal prediction mode, then no prediction block fusion operation is required. That is, only the optimal prediction block obtained based on the optimal prediction mode can be directly used as the second prediction block. After the decoder obtains the first prediction block and the second prediction block, it performs weighted processing on the first prediction block and the second prediction block to obtain the prediction block of the current block.
[0185] In some embodiments, the S340 may include:
[0186] If the first intra-frame prediction mode includes the suboptimal MIP mode or the intra-frame prediction mode derived from the TIMD mode, the decoder determines the weight of the optimal MIP mode and the weight of the first intra-frame prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the first intra-frame prediction mode; if the first intra-frame prediction mode includes the intra-frame prediction mode derived from the DIMD mode, the decoder determines that the weight of the optimal MIP mode and the weight of the first intra-frame prediction mode are both preset values.
[0187] In some embodiments, the S320 may include:
[0188] The decoder predicts samples within the template area based on the third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes in each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode; the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes in each state of the third identifier.
[0189] As mentioned above, traditional MIP technology has a higher bit overhead than other intra-frame prediction tools. It not only requires a flag to indicate whether MIP technology is used, but also a flag to indicate whether MIP is transposed. Finally, and this is the largest overhead, it requires the use of truncated binary code to represent the MIP prediction mode. MIP is a technology simplified based on neural network technology and is significantly different from traditional interpolation filter prediction technology. For some special textures, although MIP prediction is better than traditional intra-frame prediction mode, its large flag overhead is a drawback of MIP technology. Taking a 4x4 coding unit as an example, there are 16 prediction samples, but its bit overhead includes 1 MIP use flag, 1 MIP transposition flag, and 5 or 6 bits of truncated binary flags. In view of this, when determining the optimal MIP mode, this application takes into account the transposition function of the MIP mode by traversing each state of the third flag, which can save the overhead of 1 MIP transposition flag, thereby improving decompression efficiency.
[0190] Exemplarily, the decoder traverses each state of the third identifier and the multiple MIP modes, determines the distortion cost of the multiple MIP modes in each state of the third identifier, and determines the optimal MIP mode based on the distortion cost of the multiple MIP modes in each state of the third identifier; or the decoder traverses each state of the third identifier and the multiple MIP modes, determines the distortion cost of the multiple MIP modes in each state of the third identifier, and determines the optimal MIP mode based on the distortion cost of the multiple MIP modes in each state of the third identifier. In other words, the decoding end can first traverse the multiple MIP modes, or first traverse the state of the third identifier.
[0191] Exemplarily, if the value of the third identifier is the fifth numerical value, it is used to identify the input vector and output vector of the transposed MIP mode; if the value of the third identifier is the sixth numerical value, it is used to identify the input vector and output vector of the non-transposed MIP mode. In this case, each state of the third identifier can also be replaced by each value of the third identifier. In one implementation, the fifth numerical value is 1 and the sixth numerical value is 0. In another implementation, the fifth numerical value is 0 and the sixth numerical value is 1. Of course, the fifth numerical value and the sixth numerical value can also be other numerical values, and this application is not limited to this.
[0192] Exemplarily, if the third flag is true, it is used to identify the input vector and output vector of the transposed MIP mode; if the third flag is false, it is used to identify the input vector and output vector of the non-transposed MIP mode. In this case, whether the third flag is true or false is a state of the third flag.
[0193] Exemplarily, the third identifier is a sequence-level identifier, a block-level identifier, or a coding unit-level identifier.
[0194] Exemplarily, the third identifier may also be referred to as transposition information, a transposition identifier, or a MIP transposition identifier.
[0195] It should be noted that the third flag is used to identify whether the input vector and output vector of the MIP mode are transposed, and can also be replaced by a description with a similar or identical meaning. For example, in other alternative embodiments, the third flag is used to identify whether the input and output of the MIP mode need to be transposed, the third flag is used to identify whether the input vector and output vector of the MIP mode are transposed vectors, and the third flag is used to indicate whether transposition is performed.
[0196] In some embodiments, the S320 may include:
[0197] If the size of the current block is a preset size, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0198] Exemplarily, the preset size may include a size having a preset width and a preset height. That is, if the width of the current block is the preset width and the height is the preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0199] Exemplarily, the preset size can be implemented by pre-storing corresponding codes, tables, or other methods that can be used to indicate relevant information in a device (e.g., including a decoder and an encoder). This application does not limit the specific implementation method. For example, the preset size can refer to a size defined in a protocol. Alternatively, the "protocol" can refer to a standard protocol in the field of codec technology, such as a VCC or ECM protocol.
[0200] Of course, in other alternative embodiments, the decoder may also determine whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes based on the preset size in other ways, and this application does not make specific limitations on this.
[0201] For example, the decoder may determine whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes based solely on the width or height of the current block. In one implementation, if the width of the current block is a preset width or the height is a preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. For another example, the decoder may determine whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes by comparing the size of the current block with the preset size. In one implementation, if the size of the current block is greater than or less than the preset size, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. In another implementation, if the width of the current block is greater than or less than the preset width, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. In another implementation, if the height of the current block is greater than or less than the preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0202] In some embodiments, the S320 may include:
[0203] If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0204] Exemplarily, if the image frame containing the current block is an I-frame, the width of the current block is a preset width, and the height of the current block is a preset height, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes. In other words, only when the image frame containing the current block is an I-frame does the decoder determine, based on the size of the current block, whether to determine the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0205] In some embodiments, the S320 may include:
[0206] If the image frame where the current block is located is a B frame, the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes.
[0207] For example, if the image frame containing the current block is a B-frame, the decoder may determine the optimal MIP mode directly based on the distortion costs of the multiple MIP modes. In other words, when the image frame containing the current block is a B-frame, regardless of the size of the current block, the decoder may determine the optimal MIP mode directly based on the distortion costs of the multiple MIP modes.
[0208] In some embodiments, before S320, the method 300 may further include:
[0209] The decoder obtains the MIP mode used by the neighboring blocks adjacent to the current block; and the decoder determines the MIP mode used by the neighboring blocks as the multiple MIP modes.
[0210] Exemplarily, the adjacent block may be an image block adjacent to at least one of the upper side, left side, lower left, upper right, and upper left of the current block. For example, the decoder may determine the image blocks obtained in the order of the upper side, left side, lower left, upper right, and upper left of the current block as the adjacent block. Optionally, the multiple MIP modes may be used to construct an available MIP mode or a list of available MIP modes that the decoder determines to be used for predicting the current block, so that the decoder can determine the optimal MIP mode from the available MIP modes or the list of available MIP modes by predicting samples within the template area.
[0211] In some embodiments, before S320, the method 300 may further include:
[0212] The decoder performs reconstructed sample filling on a reference area adjacent to the template area to obtain reference rows and reference columns of the template area; the decoder uses the reference rows and the reference columns as input and uses the multiple MIP modes to predict samples in the template area respectively to obtain multiple prediction blocks corresponding to the multiple MIP modes; the decoder determines the distortion costs of the multiple MIP modes based on the multiple prediction blocks and the reconstructed blocks in the template area.
[0213] Exemplarily, the decoder fills in the reference reconstructed samples required for template prediction,
[0214] Exemplarily, the width of the region adjacent to the upper side of the template region in the reference region is equal to the width of the template region, and the height of the region adjacent to the left side of the template region is equal to the width of the template region. If the width of the region adjacent to the upper side of the template region in the reference region is greater than the width of the template region, the decoder may downsample or perform dimensionality reduction processing on the region adjacent to the upper side of the template region in the reference region to obtain the reference row. If the height of the region adjacent to the left side of the template region in the reference region is greater than the width of the template region, the decoder may downsample or perform dimensionality reduction processing on the region adjacent to the left side of the template region in the reference region to obtain the reference column.
[0215] Exemplarily, the template area may be the template area used in the TIMD mode mentioned above, and the reference area may be the reference template used in the TIMD mode. For example, in conjunction with FIG5 , if the current block is a coding unit with a width equal to M and a height equal to N, the decoder fills the reference area composed of coding units with a width equal to 2(M+L1)+1 and a height equal to 2(N+L2)+1 with reconstructed samples, and downsamples or reduces the dimension of the filled reference area to obtain the reference rows and the reference columns, and then constructs the input vector of the MIP mode based on the reference rows and the reference columns.
[0216] Exemplarily, after obtaining the reference row and the reference column, the decoder uses the reference row and the reference column as input, and uses the multiple MIP modes to predict the samples in the template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes; that is, the decoder predicts the samples in the template area of the current block based on the reconstructed samples in the reference template of the current block by traversing the multiple MIP modes. Taking the current traversal MIP mode as an example, the decoder takes the reference row, the reference column, the index of the current traversal MIP mode, and the third identifier mentioned above as input to obtain the prediction block corresponding to the current traversal MIP mode; wherein the reference row and the reference column are used to construct the input vector of the current traversal MIP mode; the index of the current traversal MIP mode is used to determine the matrix and / or bias vector of the current traversal MIP mode; the third identifier is used to identify whether to transpose the input vector and output vector of the MIP mode; for example, if the third identifier is used to identify the input vector and output vector of the MIP mode that are not transposed, the reference column is spliced after the reference row to form the input vector of the current traversal MIP mode; if the third identifier is used to identify the input vector and output vector of the transposed MIP mode, the reference row is spliced after the reference column to form the input vector of the current traversal MIP mode. Accordingly, if the third identifier is used to identify the input vector and output vector of the transposed MIP mode, the decoder transposes the output of the currently traversed MIP mode to obtain a prediction block for the template region. After the decoder traverses the multiple MIP modes to obtain multiple prediction blocks corresponding to the multiple MIP modes, the decoder can select the MIP mode with the lowest cost based on the distortion costs between the multiple prediction blocks and the reconstructed samples in the template region according to the principle of minimum distortion cost, and determine it as the optimal MIP mode among the template matching-based MIP modes for the current block.
[0217] In some embodiments, when the decoder uses the multiple MIP modes to predict the samples in the template area respectively, it first downsamples the reference rows and the reference columns to obtain an input vector; then, using the input vector as input, it predicts the samples in the template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; finally, it upsamples the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.
[0218] Exemplarily, the reference row and the reference column meet the input conditions of the multiple MIP modes. If the reference row and the reference column do not meet the input conditions of the multiple MIP modes, the reference row and / or the reference column may be first processed into input samples that meet the input conditions of the multiple MIP modes, and then the input vectors of the multiple MIP modes are determined based on the input samples that meet the input conditions of the multiple MIP modes. For example, taking the input condition as a specified number of input samples as an example, if the reference row and the reference column do not meet the number of input samples of the MIP mode, the decoder may perform Haar downsampling or other methods on the reference row and / or the reference column to reduce the dimension to the specified number of input samples, and determine the input vectors of the multiple MIP modes based on the specified number of input samples after the dimension reduction.
[0219] In some embodiments, the S320 may include:
[0220] The decoder determines the optimal MIP mode based on the sum of absolute transform differences (SATD) of the multiple MIP modes in the template area.
[0221] In this embodiment, when the decoder determines the optimal MIP mode based on the distortion costs of the multiple MIP modes in the template area, the distortion costs of the multiple MIP modes are designed as the SATDs of the multiple MIP modes. Compared with directly calculating the rate-distortion costs of the multiple MIP modes, not only can the optimal MIP mode be determined based on the distortion costs of the multiple MIP modes in the template area, but the calculation complexity of the distortion costs of the multiple MIP modes can also be simplified, thereby improving the decompression performance of the decoder.
[0222] In summary, the solution proposed in this application proposes the concept of fusion enhancement based on the optimal MIP mode. Specifically, the decoder not only needs to determine the optimal MIP mode for predicting the current block, but also needs to fuse it with other prediction blocks to achieve different prediction effects. This not only saves bit overhead but also creates a new prediction technology. The fusion process is actually because the optimal MIP mode cannot completely replace the optimal prediction mode calculated by the encoder based on the rate-distortion cost. Therefore, the fusion method is used to balance prediction accuracy and prediction diversity.
[0223] For example, the main ideas of the decoder's MIP mode derivation method based on template matching can be divided into the following parts:
[0224] First, fill the reference area (e.g., the reference template shown in FIG5 ) with reconstructed samples, i.e., the reference reconstructed samples required for prediction of samples within the template area (e.g., the template shown in FIG5 ). Optionally, the width and height of the reference area do not need to exceed the width and height of the template area. If the reference area is filled with samples that exceed the width and height of the template area, downsampling or other dimensionality reduction methods are required to meet the MIP input dimensionality requirements.
[0225] Then, the decoder uses the reference reconstructed samples in the reference area, the indexes of the multiple MIP modes, and the MIP transposition flag as input to predict the samples in the template area to obtain the prediction blocks corresponding to the multiple MIP modes. Optionally, the reference reconstructed samples in the reference area need to meet the MIP input conditions, such as Haar downsampling and other input samples that are reduced to a specified number of dimensions. The indexes of the multiple MIP modes are used to determine the matrix index of the MIP technology, and then obtain the MIP prediction matrix coefficients. The MIP transposition flag is used to identify whether the input and output need to be transposed.
[0226] Then, for the prediction blocks corresponding to the multiple MIP modes, all combinations of MIP modes and whether or not the MIP is transposed can be traversed to obtain the prediction samples of the template area under each MIP mode and each state of the MIP transposition flag, and the distortion between the prediction samples and the reconstructed samples in the template area is calculated and the cost information is recorded; finally, according to the principle of minimum distortion, the MIP mode with the minimum cost and its corresponding MIP transposition information are selected, which is the optimal MIP mode under the MIP prediction derivation mode based on template matching for the current block.
[0227] Finally, the decoder predicts the current block using the first intra-frame prediction mode of the optimal MIP prediction mode to obtain a first prediction block and a second prediction block, and performs weighted calculation on the first prediction block and the second prediction block according to the weighted weight of the first intra-frame prediction mode of the optimal MIP prediction mode to obtain a prediction block of the current block.
[0228] It's worth noting that some of the calculations involved in this application can be replaced by lookup tables and shifting. Although the lookup table approach may produce some errors compared to direct division, it facilitates hardware implementation and reduces encoding and decoding costs. For example, calculations related to distortion cost calculations or determining the optimal MIP mode can be used.
[0229] The decoding method according to the embodiment of the present application is described in detail above from the perspective of a decoder. The encoding method according to the embodiment of the present application will be described below from the perspective of an encoder in conjunction with FIG. 6 .
[0230] FIG8 is a schematic flow chart of an encoding method 400 provided in an embodiment of the present application. It should be understood that the encoding method 400 can be performed by an encoder. For example, it can be applied to the encoding framework 100 shown in FIG1 . For ease of description, the following description uses an encoder as an example.
[0231] As shown in FIG8 , the encoding method 400 may include:
[0232] S410, determining an optimal MIP mode for predicting a current block in a current sequence based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0233] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0234] S420, determining a first intra prediction mode;
[0235] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0236] S430, predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0237] S440, obtaining a residual block of the current block based on the prediction block of the current block and the original block of the current block;
[0238] S450: Encode the residual block of the current block to obtain a code stream of the current sequence.
[0239] In some embodiments, the S410 may include:
[0240] Get the first identifier;
[0241] If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, determining the optimal MIP mode based on the distortion costs of the multiple MIP modes;
[0242] The S450 may include:
[0243] The residual block of the current block and the first identifier are encoded to obtain the code stream.
[0244] In some embodiments, the S430 may include:
[0245] If the first identifier is used to indicate that it is permitted to use the optimal MIP mode and the first intra prediction mode to predict the image block in the current sequence, predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain the first rate-distortion cost;
[0246] Predicting the current block based on at least one intra prediction mode to obtain at least one rate-distortion cost;
[0247] If the first rate-distortion cost is less than or equal to a minimum value of the at least one rate-distortion cost, determining a prediction block obtained by predicting the current block based on the optimal MIP mode and the first intra prediction mode as the prediction block of the current block;
[0248] The S450 may include:
[0249] Encoding the residual block of the current block, the first identifier, and the second identifier to obtain the bitstream;
[0250] Wherein, if the first rate-distortion cost is less than or equal to the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block; if the first rate-distortion cost is greater than the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are not allowed to be used to predict the current block.
[0251] In some embodiments, the S450 may include:
[0252] determining an arrangement order of the multiple MIP modes based on the distortion costs of the MIP modes;
[0253] Determining, based on the arrangement order of the multiple MIP modes, a coding mode used by the optimal MIP mode;
[0254] The residual block of the current block is encoded and the index of the optimal MIP mode is encoded based on the encoding method used by the optimal MIP mode to obtain the code stream.
[0255] In some embodiments, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
[0256] In some embodiments, the S430 may include:
[0257] Predicting the current block based on the optimal MIP mode to obtain a first predicted block;
[0258] Predicting the current block based on the first intra prediction mode to obtain a second prediction block;
[0259] Based on the weight of the optimal MIP mode and the weight of the first intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.
[0260] In some embodiments, the S430 may include:
[0261] If the first intra prediction mode includes the suboptimal MIP mode or the intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the first intra prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the first intra prediction mode;
[0262] If the first intra prediction mode includes an intra prediction mode derived from the DIMD mode, it is determined that the weight of the optimal MIP mode and the weight of the first intra prediction mode are both preset values.
[0263] In some embodiments, the S410 may include:
[0264] Predicting samples within the template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode;
[0265] The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.
[0266] In some embodiments, the S410 may include:
[0267] If the size of the current block is a preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0268] In some embodiments, the S410 may include:
[0269] If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0270] In some embodiments, the S410 may include:
[0271] If the image frame where the current block is located is a B frame, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0272] In some embodiments, before S410, the method 400 may further include:
[0273] Obtaining a MIP mode used by a neighboring block adjacent to the current block;
[0274] The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.
[0275] In some embodiments, before S410, the method 400 may further include:
[0276] Filling the adjacent reference area outside the template area with reconstruction samples to obtain reference rows and reference columns of the template area;
[0277] Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes;
[0278] Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the template area.
[0279] In some embodiments, when using the multiple MIP modes to predict the samples in the template area respectively, the reference rows and the reference columns are first downsampled to obtain an input vector; then, using the input vector as input, the samples in the template area are predicted by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; finally, the output vectors of the multiple MIP modes are upsampled to obtain prediction blocks corresponding to the multiple MIP modes.
[0280] In some embodiments, the S410 may include:
[0281] The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the template area.
[0282] It should be understood that the encoding method can be understood as the inverse process of the decoding method. Therefore, the specific scheme of the encoding method 400 can refer to the relevant content of the decoding method 300. For the sake of ease of description, this application will not go into details.
[0283] The solution of this application is described below with reference to specific embodiments.
[0284] Example 1:
[0285] In this embodiment, the first intra prediction mode mentioned above is the suboptimal MIP mode, that is, the encoder or decoder may perform intra prediction on the current block based on the optimal MIP mode and the suboptimal MIP mode to obtain a prediction block of the current block.
[0286] The encoder iterates through the prediction modes. If the current block is in intra mode, the encoder obtains the sequence-level permission flag, which indicates whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current encoder allows the use of TMMIP technology.
[0287] Exemplarily, the encoder process can be implemented as follows:
[0288] Step 1:
[0289] If sps_tmmip_enable_flag is true, the encoder tries the TMMIP technology, that is, executes step 2; if sps_tmmip_enable_flag is false, the encoder does not try the TMMIP technology, that is, skips step 2 and directly executes step 3.
[0290] Step 2:
[0291] First, the encoder fills adjacent rows and columns outside the template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the encoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, they are filled with all available reconstructed samples in turn; if all reconstructed samples are unavailable, they are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the encoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.
[0292] Secondly, the encoder takes the reconstructed samples outside the filled template area as input and uses the allowed MIP mode to predict the samples within the template area.
[0293] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.
[0294] Exemplarily, the specific prediction calculation process includes: the encoder first performs Haar downsampling on the reconstructed samples, for example, the encoder determines the downsampling step size based on the block size. Then, the encoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced after the reconstructed samples after the upper downsampling, and the resulting vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced after the reconstructed samples after the left downsampling, and the resulting vector is used as input. Then, the encoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the encoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the template area.
[0295] Next, the encoder calculates the distortion cost based on the prediction block of the template area obtained by traversing each MIP mode and the reconstructed samples in the template area, and records the distortion cost value under each prediction mode and transposition information. After traversing all allowed prediction modes and transposition information, the optimal MIP mode and its corresponding transposition information, as well as the suboptimal MIP mode and its corresponding transposition information are selected according to the principle of minimum cost. The encoder determines whether fusion enhancement is required based on the relationship between the cost value of the optimal MIP mode and the cost value of the suboptimal MIP mode. If the cost value of the suboptimal MIP mode is less than twice the cost value of the optimal MIP mode, the optimal MIP prediction block and the suboptimal MIP prediction block need to be fused and enhanced. If the cost value of the suboptimal prediction mode is greater than or equal to twice the cost value of the optimal MIP mode, fusion enhancement is not required.
[0296] Finally, if fusion enhancement is required, the encoder obtains the prediction block corresponding to the optimal MIP mode and the prediction block corresponding to the suboptimal MIP mode based on the optimal MIP mode, the suboptimal MIP mode, the transposed information of the optimal MIP mode, and the transposed information of the suboptimal MIP mode. Specifically, the encoder first optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them as an input vector based on the transposed information. The encoder then reads the matrix coefficients of the current mode using the MIP mode as an index and obtains an output vector by calculating the input vector and the matrix coefficients. The encoder then transposes the output based on the transposed information and upsamples the output vector based on the size of the current block and the number of samples in the output vector to obtain the optimal MIP prediction block and the suboptimal MIP prediction block of the same size as the current block. The encoder then performs a weighted average of the optimal MIP prediction block and the suboptimal MIP prediction block based on the calculated weight values of the optimal MIP mode and the suboptimal MIP mode to obtain a new prediction block as the final prediction block for the current block. If fusion enhancement is not required, the encoder calculates the optimal MIP prediction block based on the optimal MIP mode and its transposed information. The calculation process is the same as described above. Finally, the encoder uses the optimal MIP prediction block as the prediction block for the current block. In addition, the encoder obtains the rate-distortion cost of the current block, which is recorded as cost1.
[0297] Step 3:
[0298] The encoder continues to traverse other intra prediction techniques and calculates the corresponding rate-distortion costs, which are recorded as cost2…costN.
[0299] Step 4:
[0300] If cost1 is the minimum among all rate-distortion costs, then TMMIP is used for the current block, and the encoder sets the TMMIP usage flag for the current block to true and writes it into the bitstream. If cost1 is not the minimum rate-distortion cost, then another intra-frame prediction technique is used for the current block, and the encoder sets the TMMIP usage flag for the current block to false and writes it into the bitstream. It should be understood that the flags or indices of other intra-frame prediction techniques are transmitted according to their definitions and are not elaborated on here.
[0301] Step 5:
[0302] The encoder determines a residual block for the current block based on the predicted block and the original block of the current block, and performs operations such as transformation and quantization, entropy coding, and loop filtering on the residual block of the current block. It should be understood that the specific process can be found in the relevant content above, and to avoid repetition, it will not be repeated here.
[0303] The following describes the relevant solutions of the decoder in this embodiment.
[0304] The decoder parses the block-level type flag. If it is intra mode, it parses or obtains the sequence-level permission flag, which is used to indicate whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current decoder allows the use of TMMIP technology.
[0305] Exemplarily, the decoder process can be implemented as follows:
[0306] Step 1:
[0307] If sps_tmmip_enable_flag is true, the decoder parses the TMMIP usage flag for the current block. Otherwise, the current decoding process does not need to decode the block-level TMMIP usage flag, and the block-level TMMIP usage flag defaults to false. If the TMMIP usage flag for the current block is true, proceed to step 2; otherwise, proceed to step 3.
[0308] Step 2:
[0309] First, the decoder fills adjacent rows and columns outside the template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the decoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, they are filled with all available reconstructed samples in turn; if all reconstructed samples are unavailable, they are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the decoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.
[0310] Secondly, the decoder takes the reconstructed samples outside the filled template area as input and uses the allowed MIP mode to predict the samples within the template area.
[0311] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.
[0312] Exemplarily, the specific prediction calculation process includes: the decoder first performs Haar downsampling on the reconstructed samples, for example, the decoder determines the downsampling step size based on the block size. Then, the decoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the decoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the decoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the template area.
[0313] Next, the decoder calculates the distortion cost based on the prediction block of the template area obtained by traversing each MIP mode and the reconstructed samples in the template area, and records the distortion cost value under each prediction mode and transposition information. After traversing all allowed prediction modes and transposition information, the optimal MIP mode and its corresponding transposition information, as well as the suboptimal MIP mode and its corresponding transposition information are selected according to the principle of minimum cost. The decoder determines whether fusion enhancement is required based on the relationship between the cost value of the optimal MIP mode and the cost value of the suboptimal MIP mode. If the cost value of the suboptimal MIP mode is less than twice the cost value of the optimal MIP mode, the optimal MIP prediction block and the suboptimal MIP prediction block need to be fused and enhanced. If the cost value of the suboptimal prediction mode is greater than or equal to twice the cost value of the optimal MIP mode, fusion enhancement is not required.
[0314] Finally, if fusion enhancement is required, the decoder obtains the prediction block corresponding to the optimal MIP mode and the prediction block corresponding to the suboptimal MIP mode based on the optimal MIP mode, the suboptimal MIP mode, the transposed information of the optimal MIP mode, and the transposed information of the suboptimal MIP mode. Specifically, the decoder first optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them as an input vector based on the transposed information. The decoder then reads the matrix coefficients of the current mode using the MIP mode as an index. The output vector is then calculated by combining the input vector and the matrix coefficients. The decoder can perform an output transposition based on the transposed information and upsample the output vector based on the size of the current block and the number of samples in the output vector to obtain the optimal MIP prediction block and the suboptimal MIP prediction block of the same size as the current block. The decoder then performs a weighted average of the optimal MIP prediction block and the suboptimal MIP prediction block based on the calculated weight values of the optimal MIP mode and the suboptimal MIP mode to obtain a new prediction block as the final prediction block for the current block. If fusion enhancement is not required, the decoder can calculate the optimal MIP prediction block based on the optimal MIP mode and its transposed information. The calculation process is the same as described above. Finally, the decoder uses the optimal MIP prediction block as the prediction block of the current block.
[0315] Step 3:
[0316] The decoder continues to parse information such as usage identification bits or indexes of other intra-frame prediction technologies, and obtains the final prediction block of the current block based on the parsed information.
[0317] Step 4:
[0318] The decoder parses the bitstream and obtains the frequency domain residual block of the current block (also called frequency domain residual information), and dequantizes and inverse transforms the frequency domain residual block of the current block to obtain the residual block of the current block (also called time domain residual block or time domain residual information); then the decoder superimposes the prediction block of the current block with the residual block of the current block to obtain a reconstructed sample block.
[0319] Step 5:
[0320] If all reconstructed sample blocks in the current image are processed through loop filtering and other technologies, the final reconstructed image is obtained.
[0321] Optionally, the reconstructed image can be output as a video or used as a reference for subsequent decoding.
[0322] In this embodiment, the size of the template region used by the encoder or decoder in TMMIP technology can be predefined based on the size of the current block. For example, the upper region of the template region adjacent to the current block has a width equal to the width of the current block and a height equal to the height of two rows of samples; the left region of the template region adjacent to the left side of the current block has a height equal to the height of the current block and a width equal to the width of two rows of samples. Of course, in other alternative embodiments, template regions of other sizes can also be implemented, and this application does not specifically limit this.
[0323] Example 2:
[0324] In this embodiment, the first intra-frame prediction mode mentioned above is the intra-frame prediction mode derived from the TIMD mode, that is, the encoder or decoder can perform intra-frame prediction on the current block based on the optimal MIP mode and the intra-frame prediction mode derived from the TIMD mode to obtain a prediction block of the current block.
[0325] That is to say, the MIP mode derivation fusion enhancement technology based on template matching can not only fuse two derived MIP prediction blocks, but also fuse with prediction blocks generated by other derivation technologies based on template matching. This application fuses TMMIP technology with TIMD technology to obtain a method for fusing derived traditional prediction blocks with matrix-based prediction blocks. TIMD uses the idea of template matching at the codec end to derive the optimal traditional intra-frame prediction mode, and the technology can also offset and expand the prediction mode to obtain an updated intra-frame prediction mode. The TMMIP technology also uses the idea of template matching at the codec end to derive the optimal MIP mode. By fusing these two optimal prediction modes, it is possible to take into account the directionality of the traditional prediction block and the unique texture characteristics of the MIP prediction, generate a new prediction block, and improve coding efficiency.
[0326] The encoder iterates through the prediction modes. If the current block is in intra mode, the encoder obtains the sequence-level permission flag, which indicates whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current encoder allows the use of TMMIP technology.
[0327] Exemplarily, the encoder process can be implemented as follows:
[0328] Step 1:
[0329] If sps_tmmip_enable_flag is true, the encoder tries the TMMIP technology, that is, executes step 2; if sps_tmmip_enable_flag is false, the encoder does not try the TMMIP technology, that is, skips step 2 and directly executes step 3.
[0330] Step 2:
[0331] First, the encoder fills adjacent rows and columns outside the template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the encoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, they are filled with all available reconstructed samples in turn; if all reconstructed samples are unavailable, they are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the encoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.
[0332] Secondly, the encoder takes the reconstructed samples outside the filled template area as input and uses the allowed MIP mode to predict the samples within the template area.
[0333] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.
[0334] Exemplarily, the specific prediction calculation process includes: the encoder first performs Haar downsampling on the reconstructed samples, for example, the encoder determines the downsampling step size based on the block size. Then, the encoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced after the reconstructed samples after the upper downsampling, and the resulting vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced after the reconstructed samples after the left downsampling, and the resulting vector is used as input. Then, the encoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the encoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the template area.
[0335] In addition, the encoder also needs to try the template matching calculation process of TIMD, obtain different interpolation filters according to different prediction mode indexes, and interpolate the reference samples to obtain the predicted samples within the template.
[0336] Next, the encoder calculates the distortion cost based on the predicted samples of the template area obtained by traversing each MIP mode and the reconstructed samples within the template area, and records the distortion cost value under each prediction mode and transposition information. Based on the distortion cost value under each prediction mode and transposition information, the encoder selects the optimal MIP mode and its corresponding transposition information according to the principle of minimum cost. In addition, the encoder also needs to traverse all intra-frame prediction modes allowed by TIMD, calculate the predicted samples within the template, and calculate the distortion cost with the reconstructed samples within the template. According to the principle of minimum cost, the encoder records the optimal prediction mode, suboptimal prediction mode, distortion cost value of the optimal prediction mode, and distortion cost value of the suboptimal prediction mode derived by TIMD technology.
[0337] Finally, based on the obtained optimal MIP mode and transposition information, the encoder optionally samples the reconstructed samples above and to the left of the current block and concatenates them according to the transposition information as the input vector. The encoder then reads the matrix coefficients of the current mode using the MIP mode as an index and calculates the output vector by combining the input vector and the matrix coefficients. The encoder can perform an output transposition based on the transposition information and upsample the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block.
[0338] For the optimal and suboptimal prediction modes derived from TIMD technology, if neither the optimal nor the suboptimal prediction mode is a mean (DC) or flat (PLANAR) mode, and the distortion cost of the suboptimal prediction mode is less than twice the distortion cost of the optimal prediction mode, the encoder needs to perform a prediction block fusion operation. First, the encoder obtains interpolation filter coefficients based on the optimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the optimal prediction block. Second, the encoder obtains interpolation filter coefficients based on the suboptimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the suboptimal prediction block. Next, the encoder calculates the weights of the optimal and suboptimal prediction blocks using the ratio between the cost values of the optimal and suboptimal prediction modes. Finally, the encoder performs a weighted fusion of the optimal and suboptimal prediction blocks to obtain the prediction block for the current block as the output. If the optimal prediction mode or the suboptimal prediction mode is the mean mode (DC) or the flat mode (PLANAR), or the cost of the suboptimal prediction mode is greater than twice the cost of the optimal prediction mode, the encoder does not need to perform a prediction block fusion operation, and only uses the optimal prediction mode to interpolate and filter the upper and left adjacent reconstructed samples to obtain the optimal prediction block as the optimal TIMD prediction block of the current block.
[0339] Finally, the encoder takes a weighted average of the optimal MIP prediction block and the optimal TIMD prediction block based on the calculated weight of the optimal MIP mode and the weight of the prediction mode derived from the TIMD technique. This new prediction block becomes the prediction block for the current block. The encoder also calculates the rate-distortion cost for the current block, which is recorded as cost1.
[0340] It should be noted that since the template areas of TIMD technology and TMMIP technology can be set the same, that is, the distortion cost areas of the calculated template areas are the same, the template area cost information of the two technologies can be equivalent and at the same comparison level. At this time, whether to perform fusion enhancement can also be determined based on the cost information. This application does not make specific restrictions on this.
[0341] Step 3:
[0342] The encoder continues to traverse other intra prediction techniques and calculates the corresponding rate-distortion costs, which are recorded as cost2…costN.
[0343] Step 4:
[0344] If cost1 is the minimum among all rate-distortion costs, then TMMIP is used for the current block, and the encoder sets the TMMIP usage flag for the current block to true and writes it into the bitstream. If cost1 is not the minimum rate-distortion cost, then another intra-frame prediction technique is used for the current block, and the encoder sets the TMMIP usage flag for the current block to false and writes it into the bitstream. It should be understood that the flags or indices of other intra-frame prediction techniques are transmitted according to their definitions and are not elaborated on here.
[0345] Step 5:
[0346] The encoder determines a residual block for the current block based on the predicted block and the original block of the current block, and performs operations such as transformation and quantization, entropy coding, and loop filtering on the residual block of the current block. It should be understood that the specific process can be found in the relevant content above, and to avoid repetition, it will not be repeated here.
[0347] The following describes the relevant solutions of the decoder in this embodiment.
[0348] The decoder parses the block-level type flag. If it is intra mode, it parses or obtains the sequence-level permission flag, which is used to indicate whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current decoder allows the use of TMMIP technology.
[0349] Exemplarily, the decoder process can be implemented as follows:
[0350] Step 1:
[0351] If sps_tmmip_enable_flag is true, the decoder parses the TMMIP usage flag for the current block. Otherwise, the current decoding process does not need to decode the block-level TMMIP usage flag, and the block-level TMMIP usage flag defaults to false. If the TMMIP usage flag for the current block is true, proceed to step 2; otherwise, proceed to step 3.
[0352] Step 2:
[0353] First, the decoder fills adjacent rows and columns outside the template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the decoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, they are filled with all available reconstructed samples in turn; if all reconstructed samples are unavailable, they are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the decoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.
[0354] Secondly, the decoder takes the reconstructed samples outside the filled template area as input and uses the allowed MIP mode to predict the samples within the template area.
[0355] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.
[0356] Exemplarily, the specific prediction calculation process includes: the decoder first performs Haar downsampling on the reconstructed samples, for example, the decoder determines the downsampling step size based on the block size. Then, the decoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the decoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the decoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the template area.
[0357] In addition, the decoder also needs to try the template matching calculation process of TIMD, obtain different interpolation filters according to different prediction mode indexes, and interpolate the reference samples to obtain the predicted samples within the template.
[0358] Next, the decoder calculates the distortion cost based on the predicted samples of the template area obtained by traversing each MIP mode and the reconstructed samples within the template area, and records the distortion cost value under each prediction mode and transposition information. Based on the distortion cost value under each prediction mode and transposition information, the decoder selects the optimal MIP mode and its corresponding transposition information according to the principle of minimum cost. In addition, the decoder also needs to traverse all intra-frame prediction modes allowed by TIMD, calculate the predicted samples within the template, and calculate the distortion cost with the reconstructed samples within the template. According to the principle of minimum cost, the decoder records the optimal prediction mode, suboptimal prediction mode, distortion cost value of the optimal prediction mode, and distortion cost value of the suboptimal prediction mode derived by TIMD technology.
[0359] Finally, based on the obtained optimal MIP mode and transposition information, the decoder optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them according to the transposition information as the input vector. Using the MIP mode as an index, the decoder reads the matrix coefficients of the current mode and calculates the output vector by combining the input vector and the matrix coefficients. The decoder can perform an output transposition based on the transposition information and upsample the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block.
[0360] For the optimal and suboptimal prediction modes derived from TIMD technology, if neither the optimal nor the suboptimal prediction mode is a mean (DC) or flat (PLANAR) mode, and the distortion cost of the suboptimal prediction mode is less than twice the distortion cost of the optimal prediction mode, the decoder needs to perform prediction block fusion. First, the decoder obtains interpolation filter coefficients based on the optimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the optimal prediction block. Second, the decoder obtains interpolation filter coefficients based on the suboptimal prediction mode and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples at all positions within the current block, which is recorded as the suboptimal prediction block. Next, the decoder calculates the weights of the optimal and suboptimal prediction blocks using the ratio between the cost values of the optimal and suboptimal prediction modes. Finally, the decoder performs a weighted fusion of the optimal and suboptimal prediction blocks to obtain the prediction block for the current block as output. If the optimal prediction mode or the suboptimal prediction mode is the mean mode (DC) or the flat mode (PLANAR), or the cost of the suboptimal prediction mode is greater than twice the cost of the optimal prediction mode, the decoder does not need to perform the prediction block fusion operation, and only uses the optimal prediction mode to interpolate and filter the upper and left adjacent reconstructed samples to obtain the optimal prediction block as the optimal TIMD prediction block of the current block.
[0361] Finally, the decoder performs a weighted average of the optimal MIP prediction block and the optimal TIMD prediction block based on the calculated weight value of the optimal MIP mode and the weight value of the prediction mode derived by TIMD technology to obtain a new prediction block, which is the prediction block of the current block.
[0362] Step 3:
[0363] The decoder continues to parse information such as usage identification bits or indexes of other intra-frame prediction technologies, and obtains the final prediction block of the current block based on the parsed information.
[0364] Step 4:
[0365] The decoder parses the bitstream and obtains the frequency domain residual block of the current block (also called frequency domain residual information), and dequantizes and inverse transforms the frequency domain residual block of the current block to obtain the residual block of the current block (also called time domain residual block or time domain residual information); then the decoder superimposes the prediction block of the current block with the residual block of the current block to obtain a reconstructed sample block.
[0366] Step 5:
[0367] If all reconstructed sample blocks in the current image are processed through loop filtering and other technologies, the final reconstructed image is obtained.
[0368] Optionally, the reconstructed image can be output as a video or used as a reference for subsequent decoding.
[0369] In this embodiment, the weight calculation process for weighted fusion of TIMD prediction blocks can be found in the description of TIMD technology above and will not be repeated here to avoid repetition. Furthermore, the encoder or decoder can determine whether to perform fusion enhancement based on the optimal prediction mode derived from TIMD. For example, if the optimal prediction mode derived from TIMD is the DC mode or the PLANAR mode, the encoder or decoder may not use fusion enhancement, i.e., only the prediction block generated by the optimal MIP mode derived from the TMMIP technology can be used as the prediction block for the current block. Furthermore, the size of the template region used by the encoder or decoder in the TMMIP technology can be predefined based on the size of the current block. For example, the definition of the template region in the TMMIP technology can be consistent with or different from the definition of the template region in the TIMD technology. For example, if the width of the current block is less than or equal to 8, the height of the upper region of the template region adjacent to the current block is two rows of samples; otherwise, the height is four rows of samples. Similarly, if the height of the current block is less than or equal to 8, the width of the left region of the template region adjacent to the left side of the current block is two columns of samples; otherwise, the width is four columns of samples.
[0370] Example 3:
[0371] In this embodiment, the first intra-frame prediction mode mentioned above is the intra-frame prediction mode derived from the DIMD mode, that is, the encoder or decoder can perform intra-frame prediction on the current block based on the optimal MIP mode and the intra-frame prediction mode derived from the DIMD mode to obtain a prediction block of the current block.
[0372] Similar to Example 2, the TMMIP technology can also be integrated and enhanced with the DIMD technology.
[0373] It should be noted that although both DIMD and TIMD derive prediction modes from traditional intra-frame prediction modes, the resulting prediction modes are not necessarily identical due to different derivation methods. Furthermore, the fusion enhancement approach for TMMIP and DIMD differs from that for TMMIP and TIMD. For example, since the template areas for TMMIP and TIMD are generally the same size, and the cost information is essentially the sum of absolute transformed differences (SATD), also known as the distortion cost based on the Hadamard transform, TMMIP and TIMD can directly calculate fusion weights based on this cost information. However, the template area for DIMD is generally different from that for TMMIP (or DIMD), and the criterion for DIMD's prediction mode derivation is based on gradient magnitude, which is not directly equivalent to the SATD cost. Therefore, the weights cannot be simply calculated based on the fusion approach for TMMIP and TIMD.
[0374] The encoder iterates through the prediction modes. If the current block is in intra mode, the encoder obtains the sequence-level permission flag, which indicates whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current encoder allows the use of TMMIP technology.
[0375] Exemplarily, the encoder process can be implemented as follows:
[0376] Step 1:
[0377] If sps_tmmip_enable_flag is true, the encoder tries the TMMIP technology, that is, executes step 2; if sps_tmmip_enable_flag is false, the encoder does not try the TMMIP technology, that is, skips step 2 and directly executes step 3.
[0378] Step 2:
[0379] First, the encoder fills adjacent rows and columns outside the template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the encoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, they are filled with all available reconstructed samples in turn; if all reconstructed samples are unavailable, they are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the encoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.
[0380] Secondly, the encoder takes the reconstructed samples outside the filled template area as input and uses the allowed MIP mode to predict the samples within the template area.
[0381] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.
[0382] Exemplarily, the specific prediction calculation process includes: the encoder first performs Haar downsampling on the reconstructed samples, for example, the encoder determines the downsampling step size based on the block size. Then, the encoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced after the reconstructed samples after the upper downsampling, and the resulting vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced after the reconstructed samples after the left downsampling, and the resulting vector is used as input. Then, the encoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the encoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the template area.
[0383] In addition, the encoder uses DIMD technology to derive the optimal intra-frame prediction mode, which is the optimal DIMD mode. DIMD technology calculates the gradient value of the reconstructed sample within the template area based on the Sobel operator, and converts the gradient value according to the angle value of different prediction modes to obtain the amplitude value of the corresponding prediction mode.
[0384] Next, the encoder iterates over the template prediction blocks obtained for each MIP mode, calculates the distortion cost with the reconstructed samples within the template, and records the optimal MIP mode and transposition information based on the principle of minimum cost. Furthermore, the encoder iterates over all allowed intra-frame prediction modes, calculates the amplitude values for each intra-frame prediction mode, and records the optimal DIMD prediction mode based on the principle of maximum amplitude.
[0385] Finally, based on the obtained optimal MIP mode and transposition information, the encoder optionally samples the reconstructed samples above and to the left of the current block and concatenates them according to the transposition information as the input vector. Using the MIP mode as an index, it reads the matrix coefficients for the current mode. The encoder then calculates the output vector by combining the input vector and the matrix coefficients. The encoder transposes the output according to the transposition information and upsamples the output vector based on the current block size and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block. Furthermore, for the optimal DIMD prediction mode, the encoder obtains the corresponding interpolation filter coefficients and performs interpolation filtering on the reconstructed samples above and to the left to obtain prediction samples for all positions within the current block, denoted as the optimal DIMD prediction block. The encoder then performs a weighted average of each prediction sample using the optimal MIP prediction block and the optimal DIMD prediction block according to pre-set weights to obtain a new prediction block, which serves as the prediction block for the current block. Furthermore, the encoder obtains the rate-distortion cost for the current block, denoted as cost1.
[0386] Step 3:
[0387] The encoder continues to traverse other intra prediction techniques and calculates the corresponding rate-distortion costs, which are recorded as cost2…costN.
[0388] Step 4:
[0389] If cost1 is the minimum among all rate-distortion costs, then TMMIP is used for the current block, and the encoder sets the TMMIP usage flag for the current block to true and writes it into the bitstream. If cost1 is not the minimum rate-distortion cost, then another intra-frame prediction technique is used for the current block, and the encoder sets the TMMIP usage flag for the current block to false and writes it into the bitstream. It should be understood that the flags or indices of other intra-frame prediction techniques are transmitted according to their definitions and are not elaborated on here.
[0390] Step 5:
[0391] The encoder determines a residual block for the current block based on the predicted block and the original block of the current block, and performs operations such as transformation and quantization, entropy coding, and loop filtering on the residual block of the current block. It should be understood that the specific process can be found in the relevant content above, and to avoid repetition, it will not be repeated here.
[0392] The following describes the relevant solutions of the decoder in this embodiment.
[0393] The decoder parses the block-level type flag. If it is intra mode, it parses or obtains the sequence-level permission flag, which is used to indicate whether the current sequence is allowed to use the template matching-based MIP mode derivation technology, which can be in the form of sps_tmmip_enable_flag. If all the tmmip permission flags are true, it means that the current decoder allows the use of TMMIP technology.
[0394] Exemplarily, the decoder process can be implemented as follows:
[0395] Step 1:
[0396] If sps_tmmip_enable_flag is true, the decoder parses the TMMIP usage flag for the current block. Otherwise, the current decoding process does not need to decode the block-level TMMIP usage flag, and the block-level TMMIP usage flag defaults to false. If the TMMIP usage flag for the current block is true, proceed to step 2; otherwise, proceed to step 3.
[0397] Step 2:
[0398] First, the decoder fills adjacent rows and columns outside the template area with reconstructed samples. The filling process is the same as the filling method used in the original intra-frame prediction process. For example, the decoder can traverse and fill from the lower left corner to the upper right corner. If all reconstructed samples are available, they are filled with all available reconstructed samples in turn; if all reconstructed samples are unavailable, they are filled with the mean; if some reconstructed samples are available, the available reconstructed samples are filled first. For the remaining unavailable reconstructed samples, the decoder can traverse in the order from the lower left corner to the upper right corner until the first available reconstructed sample appears, and then fill the previously unavailable positions with the first available reconstructed sample.
[0399] Secondly, the decoder takes the reconstructed samples outside the filled template area as input and uses the allowed MIP mode to predict the samples within the template area.
[0400] For example, for a 4x4 block, 16 MIP modes are allowed. For a block with a width or height of 4, or an 8x8 block, 8 MIP modes are allowed. For blocks of other sizes, 6 MIP modes are allowed. Furthermore, the MIP transposition function can be used for blocks of any size. The TMMIP prediction mode is the same as the MIP technique.
[0401] Exemplarily, the specific prediction calculation process includes: the decoder first performs Haar downsampling on the reconstructed samples, for example, the decoder determines the downsampling step size based on the block size. Then, the decoder adjusts the splicing order of the reconstructed samples after the upper downsampling and the reconstructed samples after the left downsampling according to the information of whether to transpose or not; if transposition is not required, the reconstructed samples after the left downsampling are spliced after the reconstructed samples after the upper downsampling, and the obtained vector is used as input; if transposition is required, the reconstructed samples after the upper downsampling are spliced after the reconstructed samples after the left downsampling, and the obtained vector is used as input. Then, the decoder obtains the MIP matrix coefficients based on the traversed prediction mode as an index, and calculates the output vector with the input. Finally, the decoder upsamples the output vector based on the number of output vectors and the current template size. If upsampling is not required, the vector is filled in the horizontal direction in sequence as the template prediction block output. If upsampling is required, the horizontal direction is first upsampled and then the vertical direction is downsampled. After upsampling to the same size as the template, it is output as the prediction block of the template area.
[0402] In addition, the decoder uses DIMD technology to derive the optimal intra-frame prediction mode, which is the optimal DIMD mode. DIMD technology calculates the gradient value of the reconstructed sample within the template area based on the Sobel operator, and converts the gradient value according to the angle value of different prediction modes to obtain the amplitude value of the corresponding prediction mode.
[0403] Next, the decoder iterates through the template prediction blocks obtained for each MIP mode, calculates the distortion cost with the reconstructed samples within the template, and records the optimal MIP mode and transposition information based on the principle of minimum cost. Furthermore, the decoder iterates through all allowed intra-frame prediction modes, calculates the amplitude values for each intra-frame prediction mode, and records the optimal DIMD prediction mode based on the principle of maximum amplitude.
[0404] Finally, based on the obtained optimal MIP mode and transposition information, the decoder optionally samples the reconstructed samples adjacent to the upper and left sides of the current block and concatenates them according to the transposition information as the input vector. Using the MIP mode as an index, the decoder reads the matrix coefficients for the current mode. The decoder then calculates the input vector and the matrix coefficients to obtain the output vector. The decoder transposes the output according to the transposition information and upsamples the output vector based on the size of the current block and the number of samples in the output vector, obtaining an output of the same size as the current block as the optimal MIP prediction block for the current block. Furthermore, for the optimal DIMD prediction mode, the decoder obtains the corresponding interpolation filter coefficients and performs interpolation filtering on the adjacent reconstructed samples above and to the left to obtain prediction samples for all positions within the current block, which are recorded as the optimal DIMD prediction block. The decoder then performs a weighted average of each prediction sample using the optimal MIP prediction block and the optimal DIMD prediction block according to pre-set weights to obtain a new prediction block, which is the prediction block for the current block.
[0405] Step 3:
[0406] The decoder continues to parse information such as usage identification bits or indexes of other intra-frame prediction technologies, and obtains the final prediction block of the current block based on the parsed information.
[0407] Step 4:
[0408] The decoder parses the bitstream and obtains the frequency domain residual block of the current block (also called frequency domain residual information), and dequantizes and inverse transforms the frequency domain residual block of the current block to obtain the residual block of the current block (also called time domain residual block or time domain residual information); then the decoder superimposes the prediction block of the current block with the residual block of the current block to obtain a reconstructed sample block.
[0409] Step 5:
[0410] If all reconstructed sample blocks in the current image are processed through loop filtering and other technologies, the final reconstructed image is obtained.
[0411] Optionally, the reconstructed image can be output as a video or used as a reference for subsequent decoding.
[0412] In this embodiment, the calculation process of the optimal DIMD prediction block can be found in the description of the DIMD technology above and will not be repeated here to avoid repetition. Furthermore, the fusion weights of the optimal MIP prediction block and the optimal DIMD prediction block can be pre-set values, such as the optimal MIP prediction block accounting for 5 / 9 and the optimal DIMD prediction block accounting for 4 / 9. Of course, in other alternative embodiments, the fusion weights of the optimal MIP prediction block and the optimal DIMD prediction block can also be other values, and this application does not impose specific limitations on this.
[0413] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will no longer describe the various possible combinations separately. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of the present application, they should also be regarded as the contents disclosed in the present application. It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0414] The method embodiment of the present application is described in detail above. The device embodiment of the present application is described in detail below with reference to FIG. 9 to FIG. 11 .
[0415] FIG9 is a schematic block diagram of a decoder 500 according to an embodiment of the present application.
[0416] As shown in FIG9 , the decoder 500 may include:
[0417] A parsing unit 510 is configured to parse a bitstream to obtain a residual block of a current block in a current sequence;
[0418] The prediction unit 520 is configured to:
[0419] determining an optimal MIP mode for predicting the current block based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0420] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0421] determining a first intra prediction mode;
[0422] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0423] Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0424] The reconstruction unit 530 is configured to obtain a reconstructed block of the current block based on the residual block of the current block and the prediction block of the current block.
[0425] In some embodiments, the prediction unit 520 is specifically configured to:
[0426] Parsing the code stream of the current sequence to obtain a first identifier;
[0427] If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0428] In some embodiments, the prediction unit 520 is specifically configured to:
[0429] If the first identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, parsing the code stream to obtain a second identifier;
[0430] If the second flag is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the current block, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0431] In some embodiments, the parsing unit 510 is further configured to:
[0432] determining an arrangement order of the multiple MIP modes based on the distortion costs of the MIP modes;
[0433] Determining, based on the arrangement order of the multiple MIP modes, a coding mode used by the optimal MIP mode;
[0434] The code stream of the current sequence is decoded based on the encoding mode used by the optimal MIP mode to obtain an index of the optimal MIP mode.
[0435] In some embodiments, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
[0436] In some embodiments, the prediction unit 520 is specifically configured to:
[0437] Predicting the current block based on the optimal MIP mode to obtain a first predicted block;
[0438] Predicting the current block based on the first intra prediction mode to obtain a second prediction block;
[0439] Based on the weight of the optimal MIP mode and the weight of the first intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.
[0440] In some embodiments, the prediction unit 520 performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the first intra prediction mode to obtain the prediction block of the current block, and is further configured to:
[0441] If the first intra prediction mode includes the suboptimal MIP mode or the intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the first intra prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the first intra prediction mode;
[0442] If the first intra prediction mode includes an intra prediction mode derived from the DIMD mode, it is determined that the weight of the optimal MIP mode and the weight of the first intra prediction mode are both preset values.
[0443] In some embodiments, the prediction unit 520 is specifically configured to:
[0444] Predicting samples within the template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode;
[0445] The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.
[0446] In some embodiments, the prediction unit 520 is specifically configured to:
[0447] If the size of the current block is a preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0448] In some embodiments, the prediction unit 520 is specifically configured to:
[0449] If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0450] In some embodiments, the prediction unit 520 is specifically configured to:
[0451] If the image frame where the current block is located is a B frame, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0452] In some embodiments, before determining the optimal MIP mode for predicting the current block based on the distortion costs of a plurality of matrix-based intra prediction MIP modes, the prediction unit 520 is further configured to:
[0453] Obtaining a MIP mode used by a neighboring block adjacent to the current block;
[0454] The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.
[0455] In some embodiments, before determining the optimal MIP mode for predicting the current block based on the distortion costs of a plurality of matrix-based intra prediction MIP modes, the prediction unit 520 is further configured to:
[0456] Filling the adjacent reference area outside the template area with reconstruction samples to obtain reference rows and reference columns of the template area;
[0457] Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes;
[0458] Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the template area.
[0459] In some embodiments, the prediction unit 520 is specifically configured to:
[0460] Downsampling the reference row and the reference column to obtain an input vector;
[0461] Taking the input vector as input, predicting samples in the template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes;
[0462] Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.
[0463] In some embodiments, the prediction unit 520 is specifically configured to:
[0464] The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the template area.
[0465] FIG10 is a schematic block diagram of an encoder 600 according to an embodiment of the present application.
[0466] As shown in FIG10 , the encoder 600 may include:
[0467] The prediction unit 610 is configured to:
[0468] determining an optimal MIP mode for predicting a current block in a current sequence based on distortion costs of a plurality of matrix-based intra prediction MIP modes;
[0469] The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes;
[0470] determining a first intra prediction mode;
[0471] The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode;
[0472] Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block;
[0473] A residual unit 620, configured to obtain a residual block of the current block based on a prediction block of the current block and an original block of the current block;
[0474] The encoding unit 630 is configured to encode the residual block of the current block to obtain a code stream of the current sequence.
[0475] In some embodiments, the prediction unit 610 is specifically configured to:
[0476] Get the first identifier;
[0477] If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, determining the optimal MIP mode based on the distortion costs of the multiple MIP modes;
[0478] The encoding unit 630 is specifically configured to:
[0479] The residual block of the current block and the first identifier are encoded to obtain the code stream.
[0480] The predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block includes:
[0481] In some embodiments, the prediction unit 610 is specifically configured to:
[0482] If the first identifier is used to indicate that it is permitted to use the optimal MIP mode and the first intra prediction mode to predict the image block in the current sequence, predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain the first rate-distortion cost;
[0483] Predicting the current block based on at least one intra prediction mode to obtain at least one rate-distortion cost;
[0484] If the first rate-distortion cost is less than or equal to a minimum value of the at least one rate-distortion cost, determining a prediction block obtained by predicting the current block based on the optimal MIP mode and the first intra prediction mode as the prediction block of the current block;
[0485] The encoding unit 630 is specifically configured to:
[0486] Encoding the residual block of the current block, the first identifier, and the second identifier to obtain the bitstream;
[0487] Wherein, if the first rate-distortion cost is less than or equal to the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block; if the first rate-distortion cost is greater than the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are not allowed to be used to predict the current block.
[0488] In some embodiments, the encoding unit 630 is specifically configured to:
[0489] determining an arrangement order of the multiple MIP modes based on the distortion costs of the MIP modes;
[0490] Determining, based on the arrangement order of the multiple MIP modes, a coding mode used by the optimal MIP mode;
[0491] The residual block of the current block is encoded and the index of the optimal MIP mode is encoded based on the encoding method used by the optimal MIP mode to obtain the code stream.
[0492] In some embodiments, the codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
[0493] In some embodiments, the prediction unit 610 is specifically configured to:
[0494] Predicting the current block based on the optimal MIP mode to obtain a first predicted block;
[0495] Predicting the current block based on the first intra prediction mode to obtain a second prediction block;
[0496] Based on the weight of the optimal MIP mode and the weight of the first intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.
[0497] In some embodiments, the prediction unit 610 performs weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the first intra prediction mode to obtain the prediction block of the current block, and is further configured to:
[0498] If the first intra prediction mode includes the suboptimal MIP mode or the intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the first intra prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the first intra prediction mode;
[0499] If the first intra prediction mode includes an intra prediction mode derived from the DIMD mode, it is determined that the weight of the optimal MIP mode and the weight of the first intra prediction mode are both preset values.
[0500] In some embodiments, the prediction unit 610 is specifically configured to:
[0501] Predicting samples within the template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode;
[0502] The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.
[0503] In some embodiments, the prediction unit 610 is specifically configured to:
[0504] If the size of the current block is a preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0505] In some embodiments, the prediction unit 610 is specifically configured to:
[0506] If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0507] In some embodiments, the prediction unit 610 is specifically configured to:
[0508] If the image frame where the current block is located is a B frame, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
[0509] In some embodiments, before determining the optimal MIP mode for predicting the current block in the current sequence based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, the prediction unit 610 is further configured to:
[0510] Obtaining a MIP mode used by a neighboring block adjacent to the current block;
[0511] The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.
[0512] In some embodiments, before determining the optimal MIP mode for predicting the current block in the current sequence based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, the prediction unit 610 is further configured to:
[0513] Filling the adjacent reference area outside the template area with reconstruction samples to obtain reference rows and reference columns of the template area;
[0514] Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes;
[0515] Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the template area.
[0516] In some embodiments, the prediction unit 610 is specifically configured to:
[0517] Downsampling the reference row and the reference column to obtain an input vector;
[0518] Taking the input vector as input, predicting samples in the template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes;
[0519] Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.
[0520] In some embodiments, the prediction unit 610 is specifically configured to:
[0521] The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the template area.
[0522] It should be understood that the device embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, the decoder 500 shown in Figure 9 can correspond to the corresponding subject in the method 300 for executing the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the decoder 500 are respectively for implementing the corresponding processes in the various methods such as the method 300. Similarly, the encoder 600 shown in Figure 10 can correspond to the corresponding subject in the method 400 for executing the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the encoder 600 are respectively for implementing the corresponding processes in the various methods such as the method 400.
[0523] It should also be understood that the various units in the decoder 500 or encoder 600 involved in the embodiment of the present application can be separately or all merged into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the decoder 500 or encoder 600 can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the decoder 500 or encoder 600 involved in the embodiment of the present application can be constructed by running a computer program (including program code) that can execute each step involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), etc., to construct the decoder 500 or encoder 600 involved in the embodiment of the present application, and to implement the encoding method or decoding method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device via the computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application.
[0524] In other words, the units mentioned above can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software in a decoding processor. Optionally, the software can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in combination with its hardware.
[0525] FIG11 is a schematic structural diagram of an electronic device 700 provided in an embodiment of the present application.
[0526] As shown in Figure 11, the electronic device 700 includes at least a processor 710 and a computer-readable storage medium 720. The processor 710 and the computer-readable storage medium 720 may be connected via a bus or other means. The computer-readable storage medium 720 is used to store a computer program 721, which includes computer instructions. The processor 710 is used to execute the computer instructions stored in the computer-readable storage medium 720. The processor 710 is the computing core and control core of the electronic device 700. It is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.
[0527] As an example, the processor 710 may also be referred to as a central processing unit (CPU). The processor 710 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.
[0528] As an example, the computer-readable storage medium 720 may be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 710. Specifically, the computer-readable storage medium 720 includes, but is not limited to: volatile memory and / or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).
[0529] In one implementation, the electronic device 700 may be an encoder or encoding framework involved in an embodiment of the present application; a first computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the first computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the encoding method provided in an embodiment of the present application; in other words, the first computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed. To avoid repetition, they are not repeated here.
[0530] In one implementation, the electronic device 700 may be a decoder or decoding framework involved in an embodiment of the present application; a second computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the second computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the decoding method provided in an embodiment of the present application; in other words, the second computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed, which will not be repeated here to avoid repetition.
[0531] According to another aspect of the present application, an embodiment of the present application further provides a coding and decoding system, including the encoder and decoder mentioned above.
[0532] According to another aspect of the present application, an embodiment of the present application further provides a computer-readable storage medium (Memory), which is a memory device in the electronic device 700 for storing programs and data. For example, a computer-readable storage medium 720. It can be understood that the computer-readable storage medium 720 here can include both the built-in storage medium in the electronic device 700 and, of course, the extended storage medium supported by the electronic device 700. The computer-readable storage medium provides a storage space, which stores the operating system of the electronic device 700. In addition, one or more computer instructions suitable for being loaded and executed by the processor 710 are also stored in the storage space. These computer instructions can be one or more computer programs 721 (including program codes).
[0533] According to another aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. For example, computer program 721 is provided. In this case, data processing device 700 may be a computer, processor 710 may read the computer instructions from computer-readable storage medium 720, and processor 710 may execute the computer instructions, causing the computer to perform the encoding method or decoding method provided in the various optional embodiments described above.
[0534] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the functions of the embodiment of the present application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.
[0535] Those skilled in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0536] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.
Claims
1. A decoding method, characterized in that: The method is applicable to a decoder, and comprises: Parse the code stream to obtain the residual block of the current block in the current sequence; determining an optimal MIP mode for predicting the current block based on distortion costs of a plurality of matrix-based intra prediction MIP modes; The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes; determining a first intra prediction mode; The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode; Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block; A reconstructed block of the current block is obtained based on the residual block of the current block and the prediction block of the current block.
2. The method according to claim 1, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting the current block, comprises: Parsing the code stream of the current sequence to obtain a first identifier; If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
3. The method according to claim 2, characterized in that If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: If the first identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the image block in the current sequence, parsing the code stream to obtain a second identifier; If the second flag is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the current block, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
4. The method according to claim 1, wherein The method further comprises: determining an arrangement order of the multiple MIP modes based on the distortion costs of the MIP modes; Determining, based on the arrangement order of the multiple MIP modes, a coding mode used by the optimal MIP mode; The code stream of the current sequence is decoded based on the encoding mode used by the optimal MIP mode to obtain an index of the optimal MIP mode.
5. The method according to claim 4, characterized in that The codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable-length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
6. The method according to any one of claims 1 to 5, characterized in that The predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block includes: Predicting the current block based on the optimal MIP mode to obtain a first predicted block; Predicting the current block based on the first intra prediction mode to obtain a second prediction block; Based on the weight of the optimal MIP mode and the weight of the first intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.
7. The method according to claim 6, characterized in that Before performing weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the first intra prediction mode to obtain the prediction block of the current block, the method further includes: If the first intra prediction mode includes the suboptimal MIP mode or the intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the first intra prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the first intra prediction mode; If the first intra prediction mode includes an intra prediction mode derived from the DIMD mode, it is determined that the weight of the optimal MIP mode and the weight of the first intra prediction mode are both preset values.
8. The method according to any one of claims 1 to 7, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting the current block, comprises: Predicting samples within the template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode; The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.
9. The method according to any one of claims 1 to 8, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting the current block, comprises: If the size of the current block is a preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
10. The method according to claim 9, characterized in that If the size of the current block is a preset size, determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
11. The method according to any one of claims 1 to 8, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting the current block, comprises: If the image frame where the current block is located is a B frame, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
12. The method according to any one of claims 1 to 11, characterized in that Before determining the optimal MIP mode for predicting the current block based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, the method further comprises: Obtaining a MIP mode used by a neighboring block adjacent to the current block; The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.
13. The method according to any one of claims 1 to 12, characterized in that Before determining the optimal MIP mode for predicting the current block based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, the method further comprises: Filling the adjacent reference area outside the template area with reconstruction samples to obtain reference rows and reference columns of the template area; Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes; Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the template area.
14. The method according to claim 13, wherein: The step of taking the reference row and the reference column as input and using the multiple MIP modes to respectively predict samples within the template area to obtain multiple prediction blocks corresponding to the multiple MIP modes includes: Downsampling the reference row and the reference column to obtain an input vector; Taking the input vector as input, predicting samples in the template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.
15. The method according to any one of claims 1 to 14, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting the current block, comprises: The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the template area.
16. A coding method, characterized in that: The method is applicable to an encoder, and the method comprises: determining an optimal MIP mode for predicting a current block in a current sequence based on distortion costs of a plurality of matrix-based intra prediction MIP modes; The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes; determining a first intra prediction mode; The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode; Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block; Obtaining a residual block of the current block based on the prediction block of the current block and the original block of the current block; The residual block of the current block is encoded to obtain a code stream of the current sequence.
17. The method according to claim 16, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting a current block in a current sequence comprises: Get the first identifier; If the first identifier is used to indicate that the optimal MIP mode and the first intra prediction mode are allowed to be used to predict the image block in the current sequence, determining the optimal MIP mode based on the distortion costs of the multiple MIP modes; The encoding of the residual block of the current block to obtain the code stream of the current sequence includes: The residual block of the current block and the first identifier are encoded to obtain the code stream.
18. The method according to claim 17, characterized in that The predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block includes: If the first identifier is used to indicate that it is permitted to use the optimal MIP mode and the first intra prediction mode to predict the image block in the current sequence, predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain the first rate-distortion cost; Predicting the current block based on at least one intra prediction mode to obtain at least one rate-distortion cost; If the first rate-distortion cost is less than or equal to a minimum value of the at least one rate-distortion cost, determining a prediction block obtained by predicting the current block based on the optimal MIP mode and the first intra prediction mode as the prediction block of the current block; The step of encoding the residual block of the current block and the first identifier to obtain the bitstream includes: Encoding the residual block of the current block, the first identifier, and the second identifier to obtain the bitstream; Wherein, if the first rate-distortion cost is less than or equal to the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are allowed to be used to predict the current block; if the first rate-distortion cost is greater than the minimum value of the at least one rate-distortion cost, the second identifier is used to indicate that the optimal MIP mode and the first intra-frame prediction mode are not allowed to be used to predict the current block.
19. The method according to claim 16, wherein The encoding of the residual block of the current block to obtain the code stream of the current sequence includes: determining an arrangement order of the multiple MIP modes based on the distortion costs of the MIP modes; Determining, based on the arrangement order of the multiple MIP modes, a coding mode used by the optimal MIP mode; The residual block of the current block is encoded and the index of the optimal MIP mode is encoded based on the encoding method used by the optimal MIP mode to obtain the code stream.
20. The method according to claim 19, wherein The codeword length of the encoding method used by the first n MIP modes in the arrangement order is less than the codeword length of the encoding method used by the MIP modes after the nth MIP mode in the arrangement order; and / or, the first n MIP modes use a variable-length encoding method and the MIP modes after the nth MIP mode use a truncated binary encoding method.
21. The method according to any one of claims 16 to 20, characterized in that The predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block includes: Predicting the current block based on the optimal MIP mode to obtain a first predicted block; Predicting the current block based on the first intra prediction mode to obtain a second prediction block; Based on the weight of the optimal MIP mode and the weight of the first intra prediction mode, weighted processing is performed on the first prediction block and the second prediction block to obtain a prediction block of the current block.
22. The method according to claim 21, characterized in that Before performing weighted processing on the first prediction block and the second prediction block based on the weight of the optimal MIP mode and the weight of the first intra prediction mode to obtain the prediction block of the current block, the method further includes: If the first intra prediction mode includes the suboptimal MIP mode or the intra prediction mode derived from the TIMD mode, determining a weight of the optimal MIP mode and a weight of the first intra prediction mode based on the distortion cost of the optimal MIP mode and the distortion cost of the first intra prediction mode; If the first intra prediction mode includes an intra prediction mode derived from the DIMD mode, it is determined that the weight of the optimal MIP mode and the weight of the first intra prediction mode are both preset values.
23. The method according to any one of claims 16 to 22, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting a current block in a current sequence comprises: Predicting samples within the template area based on a third identifier and the multiple MIP modes to obtain distortion costs of the multiple MIP modes under each state of the third identifier; the third identifier is used to identify the input vector and output vector of whether to transpose the MIP mode; The optimal MIP mode is determined based on the distortion costs of the multiple MIP modes in each state of the third identifier.
24. The method according to any one of claims 16 to 23, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting a current block in a current sequence comprises: If the size of the current block is a preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
25. The method according to claim 24, characterized in that If the size of the current block is a preset size, determining the optimal MIP mode based on the distortion costs of the multiple MIP modes includes: If the image frame where the current block is located is an I frame and the size of the current block is the preset size, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
26. The method according to any one of claims 16 to 23, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting a current block in a current sequence comprises: If the image frame where the current block is located is a B frame, the optimal MIP mode is determined based on the distortion costs of the multiple MIP modes.
27. The method according to any one of claims 16 to 26, characterized in that Before determining the optimal MIP mode for predicting a current block in a current sequence based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, the method further comprises: Obtaining a MIP mode used by a neighboring block adjacent to the current block; The MIP modes used by the adjacent blocks are determined as the multiple MIP modes.
28. The method according to any one of claims 16 to 27, characterized in that Before determining the optimal MIP mode for predicting a current block in a current sequence based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, the method further comprises: Filling the adjacent reference area outside the template area with reconstruction samples to obtain reference rows and reference columns of the template area; Taking the reference row and the reference column as input, using the multiple MIP modes to predict samples in the template area respectively, to obtain multiple prediction blocks corresponding to the multiple MIP modes; Distortion costs of the multiple MIP modes are determined based on the multiple prediction blocks and the reconstructed blocks within the template area.
29. The method according to claim 28, characterized in that The step of taking the reference row and the reference column as input and using the multiple MIP modes to respectively predict samples within the template area to obtain multiple prediction blocks corresponding to the multiple MIP modes includes: Downsampling the reference row and the reference column to obtain an input vector; Taking the input vector as input, predicting samples in the template area by traversing the multiple MIP modes to obtain output vectors of the multiple MIP modes; Up-sampling is performed on the output vectors of the multiple MIP modes to obtain prediction blocks corresponding to the multiple MIP modes.
30. The method according to any one of claims 16 to 29, characterized in that The determining, based on the distortion costs of the plurality of matrix-based intra prediction MIP modes, an optimal MIP mode for predicting a current block in a current sequence comprises: The optimal MIP mode is determined based on the sum of absolute transformation differences (SATD) of the multiple MIP modes in the template area.
31. A decoder, characterized in that include: A parsing unit, configured to parse a bitstream to obtain a residual block of a current block in a current sequence; Prediction unit, used to: determining an optimal MIP mode for predicting the current block based on distortion costs of a plurality of matrix-based intra prediction MIP modes; The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes; determining a first intra prediction mode; The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode; Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block; The reconstruction unit is configured to obtain a reconstructed block of the current block based on the residual block of the current block and the prediction block of the current block.
32. An encoder, characterized in that include: Prediction unit, used to: determining an optimal MIP mode for predicting a current block in a current sequence based on distortion costs of a plurality of matrix-based intra prediction MIP modes; The distortion costs of the multiple MIP modes include distortion costs obtained by predicting samples in a template area adjacent to the current block using the multiple MIP modes; determining a first intra prediction mode; The first intra prediction mode includes at least one of the following: a suboptimal MIP mode for predicting the current block determined based on distortion costs of the multiple MIP modes, an intra prediction mode derived from a decoder-side intra mode-derived DIMD mode, and an intra prediction mode derived from a template-based intra mode-derived TIMD mode; Predicting the current block based on the optimal MIP mode and the first intra prediction mode to obtain a prediction block of the current block; a residual unit, configured to obtain a residual block of the current block based on a prediction block of the current block and an original block of the current block; The encoding unit is configured to encode the residual block of the current block to obtain a code stream of the current sequence.
33. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 15 or the method according to any one of claims 16 to 30 is implemented.
34. A computer-readable storage medium, characterized in that Used to store a computer program, the computer program causing a computer to execute the method according to any one of claims 1 to 15 or the method according to any one of claims 16 to 30.
35. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 15 or the method according to any one of claims 16 to 30 is implemented.
36. A code stream, characterized in that The code stream is the code stream in the method according to any one of claims 1 to 15 or the code stream generated by the method according to any one of claims 16 to 30.
Citation Information
Cited By
Encoding method, decoding method, encoder, decoder, storage medium, and bitstream
WO2026152266A1
