Decoding method, encoding method, decoder, and encoder
Patent Information
- Application Number
- CN202280094735.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-12
- Publication Date
- 2024-11-22
AI Technical Summary
Existing digital video compression technology has shortcomings in improving compression efficiency. Especially under the demand for high video definition, existing decompression technology is difficult to meet higher compression efficiency requirements.
A decoding method and encoding method are proposed. By introducing prediction mode derivation mode and transformation technology, the decompression efficiency of video blocks is improved. The specific steps include obtaining transformation coefficients, performing multiple transformations and encoding, and using prediction mode and transformation technology to optimize the video. block reconstruction process.
Significantly improves the decompression efficiency of video blocks, can more effectively compress and decompress video data under high video definition requirements, and improves coding efficiency.
Smart Images

Figure CN119013989A_ABST
Abstract
Description
Decoding method, encoding method, decoder and encoder Technical Field
[0001] The embodiments of the present application relate to the technical field of image and video coding and decoding, and more specifically, to a decoding method, an encoding method, a decoder, and an encoder. Background Art
[0002] Digital video compression technology primarily compresses large amounts of digital video data for easier transmission and storage. With the surge in Internet video usage and increasing demand for higher-quality video, while existing digital video compression standards are capable of implementing video decompression technology, there is still a need for better digital video decompression technologies to improve compression efficiency.
[0003] Summary of the Invention
[0004] The embodiments of the present application provide a decoding method, an encoding method, a decoder, and an encoder, which can improve compression efficiency.
[0005] In a first aspect, the present application provides a decoding method, comprising:
[0006] Decoding the code stream to obtain the first transform coefficient of the current block;
[0007] Performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block;
[0008] Performing a second transform on the second transform coefficient to obtain a residual block of the current block;
[0009] Predicting the current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block;
[0010] A reconstructed block of the current block is obtained based on the prediction block of the current block and the residual block of the current block.
[0011] In a second aspect, the present application provides an encoding method, comprising:
[0012] Predicting a current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block;
[0013] Obtaining a residual block of the current block based on the prediction block of the current block;
[0014] Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;
[0015] Performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block;
[0016] The fourth transform coefficients are encoded.
[0017] In a third aspect, the present application provides a decoder, comprising:
[0018] A decoding unit, configured to decode a code stream to obtain a first transform coefficient of a current block;
[0019] Transformation unit, used to:
[0020] Performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block;
[0021] Performing a second transform on the second transform coefficient to obtain a residual block of the current block;
[0022] A prediction unit, configured to predict the current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode, to obtain a prediction block of the current block;
[0023] The reconstruction unit is configured to obtain a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.
[0024] In a fourth aspect, the present application provides an encoder, comprising:
[0025] A prediction unit, configured to predict a current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode, to obtain a prediction block of the current block;
[0026] A residual unit, configured to obtain a residual block of the current block based on a prediction block of the current block;
[0027] Transformation unit, used to:
[0028] Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;
[0029] Performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block;
[0030] A coding unit is used to encode the fourth transformation coefficient.
[0031] In a fifth aspect, the present application provides a decoder, comprising:
[0032] a processor adapted to implement computer instructions; and,
[0033] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the decoding method in the above-mentioned first aspect or its various implementations.
[0034] In one implementation, there are one or more processors and one or more memories.
[0035] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0036] In a sixth aspect, the present application provides an encoder, comprising:
[0037] a processor adapted to implement computer instructions; and,
[0038] Computer-readable storage medium, the computer-readable storage medium stores computer instructions, and the computer instructions are suitable for being loaded by a processor and executing the encoding method in the above-mentioned second aspect or its various implementations.
[0039] In one implementation, there are one or more processors and one or more memories.
[0040] In one implementation, the computer-readable storage medium may be integrated with the processor, or the computer-readable storage medium may be provided separately from the processor.
[0041] In the seventh aspect, the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are read and executed by a processor of a computer device, the computer device executes the decoding method involved in the first aspect or the encoding method involved in the second aspect.
[0042] In an eighth aspect, the present application provides a code stream, which is the code stream involved in the first aspect or the code stream involved in the second aspect.
[0043] Based on the above technical solutions, the present application introduces a first transformation on the basis of the prediction mode derivation mode and the second transformation, which can improve the decompression efficiency of the current block. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] FIG1 is a schematic block diagram of a coding framework provided in an embodiment of the present application.
[0045] FIG2 is a schematic diagram of specific directions of 65 angle prediction modes provided in an embodiment of the present application.
[0046] FIG3 is an example of reference pixels in a wide-angle prediction mode provided by an embodiment of the present application.
[0047] FIG4 is a schematic diagram of the MIP mode provided in an embodiment of the present application.
[0048] FIG5 is a schematic diagram of a DIMD-based prediction model provided in an embodiment of the present application.
[0049] FIG6 is a schematic diagram of deriving a prediction block based on DIMD according to an embodiment of the present application.
[0050] FIG7 is a schematic diagram of a template used by the TIMD provided in an embodiment of the present application.
[0051] FIG8 is an example of a weight map corresponding to 64 weight derivation modes of the GPM on a square block provided by an embodiment of the present application.
[0052] FIG9 is an example of a dividing line of a weight derivation mode provided in an embodiment of the present application.
[0053] FIG10 is an example of a weight map corresponding to 56 weight derivation modes of the AWP on a square block provided by an embodiment of the present application.
[0054] FIG11 is a schematic diagram of a GPM or AWP provided in an embodiment of the present application.
[0055] FIG12 is an example of a DCT2 type base image provided in an embodiment of the present application.
[0056] FIG13 is an example of LFNST provided by an embodiment of the present application.
[0057] FIG14 is an example of a transformation matrix group of LFNST provided in an embodiment of the present application.
[0058] FIG15 is a schematic block diagram of a decoding framework provided in an embodiment of the present application.
[0059] FIG16 is a schematic flowchart of the decoding method provided in an embodiment of the present application.
[0060] FIG17 is a schematic flowchart of the encoding method provided in an embodiment of the present application.
[0061] Figure 18 is a schematic block diagram of a decoder provided in an embodiment of the present application.
[0062] FIG19 is a schematic block diagram of an encoder provided in an embodiment of the present application.
[0063] Figure 20 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.
[0065] The solution provided in the embodiment of the present application can be applied to the field of digital video coding technology, for example, including but not limited to: image coding and decoding field, video coding and decoding field, hardware video coding and decoding field, dedicated circuit video coding and decoding field and real-time video coding and decoding field. In addition, the solution provided in the embodiment of the present application can be combined with the audio and video coding standard (Audio Video Coding Standard, AVS), the second generation AVS standard (AVS2) or the third generation AVS standard (AVS3). For example, including but not limited to: H.264 / Audio Video Coding (AVC) standard, H.265 / High Efficiency Video Coding (HEVC) standard and H.266 / Versatile Video Coding (VVC) standard. In addition, the solution provided in the embodiment of the present application can be used for lossy compression of images, and can also be used for lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.
[0066] Video codec standards all use a block-based hybrid coding framework. Each video frame is divided into square largest coding units (LCUs) or coding tree units (CTUs) of the same size (e.g., 128x128, 64x64, etc.). Each LCU or CTU can be divided into rectangular coding units (CUs) according to a rule. Coding units may also be divided into prediction units (PUs) and transform units (TUs). The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-frame prediction and inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. Because adjacent pixels within a video frame are strongly correlated, intra-frame prediction is used in video codecs to eliminate spatial redundancy between adjacent pixels. Intra-frame prediction only references information from the same frame to predict pixel information within the current block. Because adjacent frames in a video have strong similarities, inter-frame prediction is used in video coding and decoding to eliminate temporal redundancy between them, thereby improving coding efficiency. Inter-frame prediction refers to image information from different frames and uses motion estimation to search for the motion vector information that best matches the current block. Transformation converts the predicted image block to the frequency domain, redistributing the energy. Combined with quantization, it removes information that is insensitive to the human eye, eliminating visual redundancy. Entropy coding can eliminate character redundancy based on the current context model and the probabilistic information of the binary bitstream.
[0067] During digital video encoding, the encoder may first read a black-and-white or color image from the original video sequence and then encode the black-and-white or color image. A black-and-white image may include pixels of the luminance component, while a color image may include pixels of the chrominance component. Optionally, the color image may also include pixels of the luminance component. The color format of the original video sequence may be a luminance-chrominance (YCbCr, YUV) format or a red-green-blue (RGB) format. Specifically, after reading the black-and-white or color image, the encoder divides it into blocks and uses intra-frame prediction or inter-frame prediction on the current block to generate a prediction block for the current block. The prediction block is subtracted from the original block of the current block to obtain a residual block. The residual block is transformed and quantized to obtain a quantization coefficient matrix. The quantization coefficient matrix is entropy encoded and output to the bitstream. During digital video decoding, the decoder uses intra-frame prediction or inter-frame prediction on the current block to generate a prediction block for the current block. Furthermore, the decoder decodes the bitstream to obtain a quantization coefficient matrix, dequantizes and inversely transforms the quantization coefficient matrix to obtain a residual block. The prediction block and the residual block are added together to obtain a reconstructed block. The reconstructed blocks can be used to compose a reconstructed image. The decoding end performs loop filtering on the reconstructed image based on an image or a block to obtain a decoded image.
[0068] The current block may be a current coding unit (CU) or a current prediction unit (PU), etc.
[0069] It should be noted that the encoder also requires similar operations as the decoder to obtain a decoded image. The decoded image can serve as a reference frame for subsequent frames for inter-frame prediction. The block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode information or parameter information determined by the encoder are output to the bitstream if necessary. The decoder determines the same block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode information or parameter information as the encoder by parsing and analyzing existing information, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical. The decoded image obtained by the encoder is also commonly referred to as a reconstructed image. During prediction, the current block can be divided into prediction units, and during transform, the current block can be divided into transform units. The division of prediction units and transform units can be the same or different. Of course, the above is only the basic process of a video codec within a block-based hybrid coding framework. With technological advancement, some modules of the framework or some steps of the process may be optimized. This application applies to the basic process of a video codec within this block-based hybrid coding framework.
[0070] To facilitate understanding, the coding framework provided in this application is briefly introduced first.
[0071] FIG1 is a schematic block diagram of a coding framework 100 provided in an embodiment of the present application.
[0072] As shown in FIG1 , the coding framework 100 may include an intra-frame prediction unit 180, an inter-frame prediction unit 170, a residual unit 110, a transform and quantization unit 120, an entropy coding unit 130, an inverse transform and inverse quantization unit 140, and a loop filter unit 150. Optionally, the coding framework 100 may further include a decoded image buffer unit 160. The coding framework 100 may also be referred to as a hybrid framework coding mode.
[0073] The intra-frame prediction unit 180 or the inter-frame prediction unit 170 may predict the image block to be encoded and output a prediction block. The residual unit 110 may calculate a residual block based on the prediction block and the image block to be encoded, i.e., the difference between the prediction block and the image block to be encoded. The transform and quantization unit 120 is configured to perform operations such as transform and quantization on the residual block to remove information that is insensitive to the human eye, thereby eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform and quantization unit 120 may be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform and quantization unit 120 may be referred to as a frequency residual block or a frequency-domain residual block. After receiving the transform and quantization coefficients output by the transform and quantization unit 120, the entropy coding unit 130 may output a bitstream based on the transform and quantization coefficients. For example, the entropy coding unit 130 may eliminate character redundancy based on a target context model and probability information of the binary bitstream. For example, the entropy coding unit 130 may be configured for context-based adaptive binary arithmetic entropy coding (CABAC). The entropy coding unit 130 may also be referred to as a header information coding unit. Optionally, in this application, the image block to be encoded may also be referred to as an original image block or a target image block, the prediction block may also be referred to as a prediction image block or an image prediction block, or may also be referred to as a prediction signal or prediction information, and the reconstruction block may also be referred to as a reconstructed image block or an image reconstruction block, or may also be referred to as a reconstruction signal or reconstruction information. In addition, for the encoding end, the image block to be encoded may also be referred to as a coding block or a coding image block, and for the decoding end, the image block to be encoded may also be referred to as a decoding block or a decoding image block. The image block to be encoded may be a CTU or a CU.
[0074] The coding framework 100 calculates the residual between the prediction block and the image block to be coded, generates a residual block, and transmits the residual block to the decoder through processes such as transformation and quantization. Correspondingly, the decoder receives and decodes the bitstream, then performs inverse transformation and inverse quantization to obtain a residual block. The predicted block, obtained by the decoder, is then superimposed on the residual block to create a reconstructed block.
[0075] It should be noted that the inverse transform and inverse quantization unit 140, the loop filter unit 150, and the decoded image buffer unit 160 in the coding framework 100 can be used to form a decoder. This means that the intra-frame prediction unit 180 or the inter-frame prediction unit 170 can predict the image block to be coded based on the existing reconstructed block, thereby ensuring that the encoder and decoder have a consistent understanding of the reference frame. In other words, the encoder can replicate the decoder's processing loop and produce the same prediction as the decoder. Specifically, the quantized transform coefficients are inversely transformed and inversely quantized by the inverse transform and inverse quantization unit 140 to replicate the approximate residual block on the decoder side. This approximate residual block, after adding the prediction block, can pass through the loop filter unit 150 to smooth out block artifacts caused by block-based processing and quantization. The image block output by the loop filter unit 150 can be stored in the decoded image buffer unit 160 for use in subsequent image prediction.
[0076] It should be understood that FIG1 is merely an example of the present application and should not be construed as limiting the present application.
[0077] For example, the loop filtering unit 150 in the coding framework 100 may include a deblocking filter (DBF) and a sample adaptive offset filter (SAO). The function of DBF is to remove the blocking effect, and the function of SAO is to remove the ringing effect. In other embodiments of the present application, the coding framework 100 may adopt a neural network-based loop filtering algorithm to improve the compression efficiency of the video. In other words, the coding framework 100 may be a video coding hybrid framework based on a deep learning neural network. In one implementation, a convolutional neural network-based model may be used to calculate the result of pixel filtering on the basis of a deblocking filter and a sample adaptive offset filter. The network structure of the loop filtering unit 150 on the luminance component and the chrominance component may be the same or different. Considering that the luminance component contains more visual information, the luminance component may also be used to guide the filtering of the chrominance component to improve the reconstruction quality of the chrominance component.
[0078] The following describes the intra-frame prediction and inter-frame prediction.
[0079] For inter-frame prediction, inter-frame prediction can refer to image information of different frames and use motion estimation to search for motion vector information that best matches the image block to be encoded to eliminate temporal redundancy; the frames used for inter-frame prediction can be P frames and / or B frames, P frames refer to forward prediction frames, and B frames refer to bidirectional prediction frames.
[0080] Intra-frame prediction only references information from the same frame, predicting pixel information within the image block to be encoded, thereby eliminating spatial redundancy. The frame used for intra-frame prediction can be an I-frame. For example, based on the left-to-right, top-to-bottom encoding order, the image block to be encoded can reference the upper left, upper, and left image blocks as reference information to predict the image block to be encoded. The image block to be encoded then serves as reference information for the next image block, allowing prediction of the entire image. If the input digital video is in a color format, such as YUV 4:2:0, each four pixels in each image frame of the digital video are composed of four Y components and two UV components. The encoding framework can encode the Y component (i.e., the luminance block) and the UV component (i.e., the chrominance block) separately. Similarly, the decoding end can also perform decoding based on the format.
[0081] Intra-frame prediction uses both angular and non-angular prediction modes to predict the image block to be coded. This generates a prediction block. Based on the rate-distortion information calculated between the prediction block and the image block to be coded, the optimal prediction mode for the image block to be coded is selected and transmitted to the decoder via the bitstream. The decoder parses the prediction mode, predicts the prediction block for the target decoding block, and then overlays it with the time-domain residual block obtained via the bitstream to reconstruct the block.
[0082] Throughout the development of digital video codec standards, non-angular prediction modes have remained relatively stable, including the mean and planar modes. Angular prediction modes have continued to increase with the evolution of digital video codec standards. For example, the H-series of international digital video coding standards, H.264 / AVC, has only eight angular prediction modes and one non-angular prediction mode; H.265 / HEVC has expanded this to 33 angular prediction modes and two non-angular prediction modes. In H.266 / VVC, intra-frame prediction modes have been further expanded, with a total of 67 traditional prediction modes and a non-traditional prediction mode, matrix-weighted intra-frame prediction (MIP), for luma blocks. These 67 traditional prediction modes include planar and DC modes, and 65 angular prediction modes. Planar modes are typically used for blocks with gradual texture changes, while DC modes, as the name suggests, are typically used for flat areas. Angular prediction modes are typically used for blocks with pronounced angular texture.
[0083] It should be noted that, in this application, the current block used for intra-frame prediction can be a square block or a rectangular block.
[0084] Furthermore, since the intra-frame prediction blocks are all square, the probability of using each angle prediction mode is equal. When the length and width of the current block are not equal, the probability of using the reference pixel on the top of the horizontal block (width is greater than height) is greater than the probability of using the reference pixel on the left, and the probability of using the reference pixel on the top of the vertical block (height is greater than width) is less than the probability of using the reference pixel on the left. Based on this, the present application introduces a wide-angle prediction mode. When predicting a rectangular block, the traditional angle prediction mode is converted to a wide-angle prediction mode. When the wide-angle prediction mode is used to predict the rectangular block, the prediction angle range of the current block is greater than the prediction angle range when the traditional angle prediction mode is used to predict the rectangular block. Optionally, when using the wide-angle prediction mode, the index of the traditional angle prediction mode can still be used to send a signal. Accordingly, the decoding end can convert the traditional angle prediction mode into the wide-angle prediction mode after receiving the signal. As a result, the total number of intra-frame prediction modes and the intra-frame mode encoding method remain unchanged, and the intra-frame mode encoding method remains unchanged.
[0085] FIG2 is a schematic diagram of specific directions of 65 angle prediction modes provided in an embodiment of the present application.
[0086] As shown in Figure 2, index 0 is used to identify the planar mode, index 1 is used to identify the DC mode, and indices -14 to 80 can be used to identify different angular prediction modes. Specifically, indices 2 to 66 are used to identify the traditional angular prediction mode, and indices -1 to -14 and 67 to 80 are used to identify the wide-angle prediction mode. In other words, the traditional intra-frame prediction mode identified by indices 2 to 66 can be used to predict square blocks, and the wide-angle prediction mode identified by indices -1 to -14 and 67 to 80 can be used to predict rectangular blocks.
[0087] It should be understood that the prediction mode identified by index x in this application may also be referred to as prediction mode x. For example, the intra-frame prediction mode identified by index 2 may also be referred to as intra-frame prediction mode 2.
[0088] FIG3 is an example of reference pixels in a wide-angle prediction mode provided by an embodiment of the present application.
[0089] As shown in Figure 3, in the wide-angle prediction mode, for a CU of size W×H, the number of reference pixels above it is 2W+1, and the number of reference pixels to its left is 2H+1. Specifically, as shown in (a) in Figure 3, when W>H (for example, an 8×4 CU), there is a situation where the lower right corner of the CU near intra-frame prediction mode 2 (larger than intra-frame prediction mode 2) cannot index to the reference pixel, while there is still a part near intra-frame prediction mode 66 (larger than intra-frame prediction mode 66) that can index to the reference pixel. Therefore, it is necessary to replace some horizontal angle modes near intra-frame prediction mode 2 (larger than intra-frame prediction mode 2) with some vertical angle modes near intra-frame prediction mode 66 (larger than intra-frame prediction mode 66) to expand the prediction angle range. Similarly, as shown in (b) in Figure 3, when W<H (for example, a 4×8 CU), since some points near intra-frame prediction mode 66 (smaller than intra-frame prediction mode 66) cannot index to reference pixels, while some points near intra-frame prediction mode 2 (smaller than intra-frame prediction mode 2) can still index to reference pixels), it is necessary to replace some vertical angle modes near intra-frame prediction mode 66 (smaller than intra-frame prediction mode 66) with some horizontal angle modes near intra-frame prediction mode 2 (smaller than intra-frame prediction mode 2) to expand the prediction angle range.
[0090] In some cases, the intra-frame prediction mode to be performed can be determined or selected based on the size of the current block; for example, the wide-angle prediction mode can be determined or selected based on the size of the current block to perform intra-frame prediction on the current block; for example, when the current block is a rectangular block (with different sizes of width and height), the wide-angle prediction mode can be used to perform intra-frame prediction on the current block. Among them, the aspect ratio of the current block can be used to determine the angular prediction mode replaced by the wide-angle prediction mode and the angular prediction mode after the replacement. For example, when predicting the current block, any intra-frame prediction mode with an angle that does not exceed the diagonal angle of the current block (from the lower left corner to the upper right corner of the current block) can be selected as the angular prediction mode after the replacement.
[0091] The following is an introduction to other intra-frame prediction modes involved in this application:
[0092] (1) Matrix based Intra Prediction (MIP) mode.
[0093] The MIP mode can also be called the Matrix Weighted Intra Prediction mode. The process involved in the MIP mode can be divided into three main steps, namely the downsampling process, the matrix multiplication process, and the upsampling process. Specifically, the downsampling process first downsamples spatially adjacent reconstructed samples, and then the downsampled sample sequence is used as the input vector of the matrix multiplication process, that is, the output vector of the downsampling process is used as the input vector of the matrix multiplication process, multiplied with a pre-set matrix and added with a bias vector, and the calculated sample vector is output; the output vector of the matrix multiplication process is used as the input vector of the upsampling process, and the final prediction block is obtained by upsampling.
[0094] FIG4 is a schematic diagram of the MIP mode provided in an embodiment of the present application.
[0095] As shown in Figure 4, the MIP mode obtains the upper adjacent down-sampled reconstructed sample vector bdry by averaging the reconstructed samples above the current coding unit during the down-sampling process. top , by averaging the left adjacent reconstructed samples, we get the left adjacent down-sampling reconstructed sample vector bdry left . Get bdry top and bdry left After that, it is used as the input vector bdry of the matrix multiplication process red Specifically, based on bdry red The top row vector bdry top red 、bdry left 、A k ·bdry red +b k Get the sample vector, where A k is a pre-set matrix, b k is a pre-set bias vector, and k is the index of the MIP mode. After obtaining the sample vector, it is upsampled by linear interpolation to obtain a prediction sample block that matches the number of samples in the actual coding unit.
[0096] In other words, to predict a block of width W and height H, MIP requires H reconstructed pixels in the column to the left of the current block and W reconstructed pixels in the row above the current block as input. MIP generates the prediction block using three steps: reference pixel averaging, matrix vector multiplication, and interpolation. The core of MIP is matrix multiplication, which can be thought of as a process of generating a prediction block from input pixels (reference pixels) using a matrix multiplication approach. MIP offers a variety of matrices, and different prediction methods are reflected in different matrices; using different matrices for the same input pixels will produce different results. The reference pixel averaging and interpolation processes are a compromise between performance and complexity. For larger blocks, reference pixel averaging can achieve a similar downsampling effect, allowing the input to fit into a smaller matrix, while interpolation achieves an upsampling effect. This eliminates the need to provide MIP matrices for every block size; instead, matrices of one or a few specific sizes are sufficient. With increasing demands for compression performance and improved hardware capabilities, even more complex MIPs may appear in next-generation standards.
[0097] The MIP mode can be simplified from a neural network. For example, the matrix it uses can be derived from training. Therefore, the MIP mode has strong generalization capabilities and prediction results that traditional prediction modes cannot achieve. The MIP mode can be a model derived from a neural network-based intra-frame prediction model through multiple hardware and software complexity simplifications. Based on a large number of training samples, multiple prediction modes represent a variety of models and parameters, which can effectively cover the texture conditions of natural sequences.
[0098] MIP is somewhat similar to the planar mode, but it is obviously more complex and more flexible than the planar mode.
[0099] It should be noted that the number of MIP modes may be different for coding units of different block sizes. For example, for a coding unit of size 4x4, the MIP mode has 16 prediction modes; for a coding unit of size 8x8, width equal to 4 or height equal to 4, the MIP mode has 8 prediction modes; for coding units of other sizes, the MIP mode has 6 prediction modes. At the same time, the MIP mode has a transposition function. For the prediction mode that meets the current size, the MIP mode can try to transpose the calculation on the encoder side. Therefore, the MIP mode not only requires a flag to indicate whether the current coding unit uses the MIP mode, but also, if the current coding unit uses the MIP mode, an additional transposition flag needs to be transmitted to the decoder.
[0100] (2) Decoder side Intra Mode Derivation (DIMD) mode.
[0101] The main core point of the DIMD mode is that the decoder uses the same method as the encoder to derive the intra-frame prediction mode, thereby avoiding transmitting the intra-frame prediction mode index of the current coding unit in the bit stream and saving bit overhead.
[0102] The specific process of DIMD mode can be divided into the following two main steps:
[0103] Step 1: Derive the prediction model.
[0104] FIG5 is a schematic diagram of a DIMD-based prediction model provided in an embodiment of the present application.
[0105] As shown in FIG5(a), DIMD uses pixels in a template within a reconstructed region (reconstructed pixels to the left and above the current block) to derive a prediction mode. For example, the template may include three adjacent rows of reconstructed samples above the current block, three adjacent columns of reconstructed samples to the left, and corresponding adjacent reconstructed samples to the upper left of the current block. Based on this, multiple gradient values can be determined within the template according to a window (such as the window shown in FIG5(a) or FIG5(b)). Each gradient value is suitable for matching an intra prediction mode (IPM) corresponding to its gradient direction. Based on this, the encoder can use the prediction mode matched to the largest and second largest gradient values among the multiple gradient values as the derived prediction mode. For example, as shown in (b) of FIG5 , for a 4×4 block, all pixels for which gradient values need to be determined are analyzed and a corresponding gradient histogram is obtained. For example, as shown in (c) of FIG5 , for blocks of other sizes, all pixels for which gradient values need to be determined are analyzed and a corresponding gradient histogram is obtained. Finally, the prediction modes corresponding to the largest and second largest gradients in the gradient histogram are used as the derived prediction modes.
[0106] Of course, the gradient histogram in this application is only an example for determining the derived prediction mode. In specific implementation, it can be implemented in a variety of simple forms, and this application does not specifically limit this. In addition, this application does not limit the method of statistically analyzing the gradient histogram. For example, the gradient histogram can be statistically analyzed using the Sobel operator or other methods.
[0107] Step 2: Derive the prediction block.
[0108] FIG6 is a schematic diagram of deriving a prediction block based on DIMD according to an embodiment of the present application.
[0109] As shown in Figure 6, the encoder can weight the prediction values of the three intra-frame prediction modes (planar mode and two intra-frame prediction modes derived based on DIMD). The codec uses the same prediction block derivation method to obtain the prediction block of the current block. Assuming that the prediction mode corresponding to the maximum gradient value is prediction mode 1 and the prediction mode corresponding to the second largest gradient value is prediction mode 2, the encoder determines the following two conditions:
[0110] 1. The gradient of prediction mode 2 is not 0;
[0111] 2. Prediction mode 1 and prediction mode 2 are neither planar mode nor DC prediction mode.
[0112] If neither of the above two conditions holds, only prediction mode 1 is used to calculate the predicted sample values for the current block, applying the normal prediction process to prediction mode 1. Otherwise, if both conditions hold, a weighted average is used to derive the prediction block for the current block. Specifically, the planar mode accounts for 1 / 3 of the weight, and the remaining 2 / 3 is the combined weight of prediction modes 1 and 2. For example, the weight for prediction mode 1 is calculated by dividing the gradient amplitude of prediction mode 1 by the sum of the gradient amplitudes of prediction mode 1 and prediction mode 2, and the weight for prediction mode 2 is calculated by dividing the gradient amplitude of prediction mode 2 by the sum of the gradient amplitudes of prediction mode 1 and prediction mode 2. The prediction blocks obtained based on the three prediction modes—prediction blocks 1, 2, and 3 obtained based on planar mode, prediction mode 1, and prediction mode 2, respectively—are weighted averaged to obtain the prediction block for the current coding unit. The decoder uses the same steps to obtain the prediction block.
[0113] In other words, the specific weight calculation in step 2 above is as follows:
[0114] Weight(PLANAR)=1 / 3;
[0115] Weight(mode1)=2 / 3*(amp1 / (amp1+amp2));
[0116] Weight(mode2)=1-Weight(PLANAR)-Weight(mode1);
[0117] Wherein, mode1 and mode2 represent prediction mode 1 and prediction mode 2 respectively, amp1 and amp2 represent the gradient amplitude value of prediction mode 1 and the gradient amplitude value of prediction mode 2 respectively. The DIMD mode requires a flag to be transmitted to the decoder, which is used to indicate whether the current coding unit uses the DIMD mode.
[0118] Of course, the above-mentioned weighted averaging method is only an example of the present application and should not be understood as a limitation of the present application.
[0119] In summary, DIMD uses gradient analysis of reconstructed pixels to select the intra prediction mode, and can weight two intra prediction modes plus the planar mode based on the analysis results. The advantage of DIMD is that if DIMD mode is selected for the current block, the bitstream no longer needs to indicate the specific intra prediction mode used. Instead, the decoder automatically derives the selected intra prediction mode through the above process, which reduces overhead to a certain extent.
[0120] (3) Template based Intra Mode Derivation (TIMD) mode.
[0121] The technical principle of the TIMD mode is similar to that of the aforementioned DIMD mode. Both utilize the same codec operation to derive prediction modes to save transmission mode index overhead. The TIMD mode can be understood as two main parts. First, the cost information of each prediction mode is calculated based on the template. The prediction modes corresponding to the minimum cost and the second minimum cost are selected. The prediction mode corresponding to the minimum cost is recorded as prediction mode 1, and the prediction mode corresponding to the second minimum cost is recorded as prediction mode 2. If the ratio of the value of the second minimum cost (costMode2) to the value of the minimum cost (costMode1) meets the preset conditions, such as costMode2<2*costMode1, the prediction blocks corresponding to prediction mode 1 and prediction mode 2 can be weightedly fused according to the corresponding weights of prediction mode 1 and prediction mode 2 to obtain the final prediction block.
[0122] Exemplarily, the weights corresponding to prediction mode 1 and prediction mode 2 are determined according to the following method:
[0123] weight1=costMode2 / (costMode1+costMode2);
[0124] weight2=1-weight1;
[0125] Where weight1 is the weight of the prediction block corresponding to prediction mode 1, and weight2 is the weight of the prediction block corresponding to prediction mode 2. However, if the ratio of the value of the second smallest cost, costMode2, to the value of the smallest cost, costMode1, does not meet the preset conditions, weighted fusion between the prediction blocks is not performed, and the prediction block corresponding to prediction mode 1 is the TIMD prediction block.
[0126] It should be noted that when using TIMD mode for intra prediction of the current block, if the reconstructed sample template of the current block does not contain available adjacent reconstructed samples, TIMD mode selects planar mode for intra prediction of the current block, that is, no unweighted fusion is performed. Similar to DIMD mode, TIMD mode requires a flag to be transmitted to the decoder to indicate whether the current coding unit uses TIMD mode.
[0127] FIG7 is a schematic diagram of a template used by the TIMD provided in an embodiment of the present application.
[0128] As shown in Figure 7, if the current block is a coding unit with a width equal to M and a height equal to N, the codec can calculate the template of the current block based on the reference template (Reference of template) of the current block selected in the coding unit with a width equal to 2(M+L1)+1 and a height equal to 2(N+L2)+1. At this time, if the template of the current block does not contain available adjacent reconstructed samples, the TIMD mode selects the plane mode to perform intra-frame prediction on the current block. For example, the available adjacent reconstructed samples can be the samples adjacent to the left and top sides of the current CU in Figure 7, that is, there are no available reconstructed samples in the diagonal filling area. In other words, if there are no available reconstructed samples in the diagonal filling area, the TIMD mode selects the plane mode to perform intra-frame prediction on the current block.
[0129] Except for edge cases, when encoding and decoding the current block, reconstructed values can theoretically be obtained for the left and top sides of the current block. This means that the template for the current block contains available adjacent reconstructed samples. In specific implementations, the decoder can use a specific intra prediction mode to predict on the template and compare the predicted value with the reconstructed value to obtain the cost of the intra prediction mode on the template. Examples include SAD, SATD, and SSE. Because the template and the current block are adjacent, the reconstructed samples within the template are correlated with the pixels within the current block. Therefore, the performance of a prediction mode on the template can be used to estimate the performance of that prediction mode on the current block. TIMD predicts several candidate intra prediction modes on the template, obtains their costs on the template, and replaces the lowest cost intra prediction mode or modes as the intra prediction value for the current block. If the cost difference between two intra prediction modes on the template is not significant, taking a weighted average of the prediction values of the two intra prediction modes can improve compression performance. Optionally, the weights of the prediction values of the two prediction modes are related to the aforementioned costs, for example, the weights are inversely proportional to the costs.
[0130] In summary, TIMD uses the prediction performance of intra-frame prediction modes on a template to select the appropriate intra-frame prediction mode, and can weight the two intra-frame prediction modes based on their cost on the template. The advantage of TIMD is that if TIMD mode is selected for the current block, the bitstream no longer needs to indicate the specific intra-frame prediction mode used. Instead, the decoder automatically derives the selected intra-frame prediction mode through the above process, which reduces overhead to a certain extent.
[0131] Through the above brief introduction to several intra-frame prediction modes, it is not difficult to find that the technical principles of the DIMD mode are similar to those of the TIMD mode. Both use the decoder to perform the same operations as the encoder to infer the prediction mode of the current coding unit. This prediction mode can save the transmission of the prediction mode index when the complexity is acceptable, thus saving overhead and improving compression efficiency. However, due to the limitations of the reference information and the fact that there is not much improvement in the prediction quality, the DIMD mode and the TIMD mode are more effective in areas with consistent texture characteristics over a large area. If the texture changes slightly or the template area cannot be covered, the prediction effect of this prediction mode is poor.
[0132] In addition, regardless of the DIMD mode or the TIMD mode, the prediction blocks obtained based on multiple traditional prediction modes are fused or weighted. The fusion of prediction blocks can generate effects that cannot be achieved by a single prediction mode. Although the DIMD mode introduces the planar mode as an additional weighted prediction mode to increase the spatial correlation between adjacent reconstructed samples and prediction samples, thereby improving the prediction effect of intra-frame prediction, due to the relatively simple prediction principle of the planar mode, for some prediction blocks with obvious difference between the upper right corner and the lower left corner, using the planar mode as an additional weighted prediction mode may have a counterproductive effect.
[0133] (4) Geometric partitioning mode (GPM) and angular weighted prediction (AWP).
[0134] In video coding standards, traditional unidirectional prediction only finds one reference block of the same size as the current block, while traditional bidirectional prediction uses two reference blocks of the same size as the current block, and the pixel value of each point in the predicted block is the average of the corresponding positions in the two reference blocks, that is, all points in each reference block account for 50%. Furthermore, bidirectional weighted prediction allows the proportions of the two reference blocks to be different, such as all points in the first reference block account for 75% and all points in the second reference block account for 25%, but all points in the same reference block have the same proportion. In addition, for some optimization methods, such as decoder-side motion vector refinement (DMVR) and bi-directional optical flow (BIO or BDOF), it will cause some changes to the reference pixels or predicted pixels.
[0135] GPM or AWP also uses two reference blocks of the same size as the current block. However, some pixel locations use 100% of the pixel values from the first reference block, while others use 100% of the pixel values from the second reference block. In the boundary area, or transition zone, pixel values from both reference blocks are used in a proportional manner. The weights in the boundary area also transition gradually. The specific distribution of these weights is determined by the weight derivation mode of the GPM or AWP. The weight of each pixel position is determined based on the weight derivation mode of the GPM or AWP.
[0136] Of course, in some cases, such as when the block size is very small, it may be impossible to guarantee in some GPM or AWP modes that some pixel positions 100% use the pixel values of the corresponding positions of the first reference block, and some pixel positions 100% use the pixel values of the corresponding positions of the second reference block; at this time, it can also be considered that GPM or AWP uses two reference blocks of different sizes from the current block, that is, each takes the required part as the reference block; that is, the part with a non-zero weight is used as the reference block, and the part with a weight of 0 is eliminated. This application does not limit its specific implementation.
[0137] FIG8 is an example of a weight map corresponding to 64 weight derivation modes of the GPM on a square block provided by an embodiment of the present application.
[0138] As shown in Figure 8, GPM has weight maps corresponding to 64 weight derivation modes for square blocks. For each weight map corresponding to a weight derivation mode, black indicates a weight of 0% for the position corresponding to the first reference block, white indicates a weight of 100%, and gray areas, depending on the depth of the color, indicate a weight greater than 0% but less than 100% for the position corresponding to the first reference block. The weight for the position corresponding to the second reference block is 100% minus the weight for the position corresponding to the first reference block.
[0139] FIG9 is an example of a dividing line of a weight derivation mode provided in an embodiment of the present application.
[0140] As shown in FIG9 , the dividing line of the weight derivation mode can be a line composed of points with the same weights for the two prediction modes corresponding to the GPM. In other words, in the weight matrix of the GPM, the dividing line is a line composed of points with the same weights for the two prediction modes corresponding to the GPM. In other words, the dividing line can be a line composed of points with the same weights for the two prediction modes corresponding to the GPM in the area where the weights change in the weight matrix of the GPM. In other words, the dividing line is a line composed of points with median weights, and the dividing line is a line composed of points corresponding to the median weights, wherein the point with the median weight may be located in the middle of the entire pixel or may not be located in the middle of the entire pixel; taking weights from 0 to 8 as an example, the median weight can be 4.
[0141] FIG10 is an example of a weight map corresponding to 56 weight derivation modes of the AWP on a square block provided by an embodiment of the present application.
[0142] Figure 10 shows the weight maps corresponding to the 56 weight derivation modes of the AWP on a square block. For each weight derivation mode, black indicates a weight of 0% for the position corresponding to the first reference block, white indicates a weight of 100%, and gray areas, depending on the depth of the color, indicate a weight greater than 0% and less than 100% for the position corresponding to the first reference block. The weight value for the position corresponding to the second reference block is 100% minus the weight value for the position corresponding to the first reference block.
[0143] It should be noted that the weight derivation methods of GPM and AWP can be different. For example, GPM determines the angle and offset according to each weight derivation mode, and then calculates the weight map corresponding to each weight derivation mode. AWP first determines a one-dimensional weight line according to each weight derivation mode, and then uses a method similar to intra-frame angle prediction to spread the one-dimensional weight line across the entire image to obtain the weight map corresponding to each weight derivation mode. Of course, in other alternative embodiments, the weight map corresponding to each weight derivation mode can also be called a weight matrix.
[0144] The following uses GPM as an example to illustrate the weight derivation method.
[0145] The encoder can determine the corresponding dividing line according to each weight derivation mode, and then determine the corresponding weight matrix based on the dividing line. For example, the encoder can use Table 1 to determine the angle index variable angleIdx and the distance index variable distanceIdx corresponding to the weight derivation mode to determine the weight derivation mode merge_gpm_partition_idx. The angle index variable angleIdx and the distance index variable distanceIdx can be considered as variables for determining the dividing line, that is, they are used to determine the angle and offset of the dividing line respectively. After the encoder determines the dividing line corresponding to each weight derivation mode, it can determine the weight matrix corresponding to each weight derivation mode based on the dividing line corresponding to each weight derivation mode.
[0146] Table 1
[0147]
[0148]
[0149] As shown in Table 1, the weight derivation mode includes 64 modes (for example, the 64 modes shown in Figure 8), and the value of its index (merge_gpm_partition_idx) is 0 to 63, wherein each of the 64 weight derivation modes may correspond to an angle index variable angleIdx and a distance index variable distanceIdx. Equivalently, each weight derivation mode may correspond to a dividing line. Of course, an angle index variable angleIdx or a distance index variable distanceIdx may correspond to the index of one or more weight derivation modes. Table 1 is only an example of the present application and should not be understood as a limitation to the present application.
[0150] Since all three components (such as Y, Cb, and Cr) can use GPM, the process of generating the GPM prediction sample matrix for one component can be divided into a sub-process. That is, for the GPM weighted prediction process (Weighted sample prediction process for geometric partitioning mode), all three components can call this process, but the parameters of the call are different. This application uses the luminance component as an example for explanation. For example, the prediction matrix predSamplesL[xL][yL] of the current luminance block can be derived from the GPM weighted prediction process; where xL = 0..cbWidth–1 and yL = 0..cbHeight-1. Here, nCbW is set to cbWidth and nCbH is set to cbHeight.
[0151] The inputs to the GPM weighted prediction process include: the width of the current block nCbW; the height of the current block nCbH; two (nCbW) x (nCbH) prediction sample matrices predSamplesLA and predSamplesLB; the GPM "partition" angle index variable angleIdx; the GPM distance index variable distanceIdx; and the component index variable cIdx, for example, when cIdx is 0, it can be used to represent the luma component. The output of the GPM weighted prediction process is: the GPM prediction sample matrix pbSamples[x][y] of (nCbW) x (nCbH); where x = 0..nCbW–1 and y = 0..nCbH-1.
[0152] The prediction sample matrix pbSamples[x][y] can be derived as follows:
[0153] For example, first, the variables nW, nH, shift1, offset1, displacementX, displacementY, partFlip, and shiftHor can be derived as follows:
[0154] nW=(cIdx==0)? nCbW:nCbW*SubWidthC.
[0155] nH=(cIdx==0)? nCbH:nCbH*SubHeightC.
[0156] shift1=Max(5,17-BitDepth), where BitDepth is the bit depth of the codec.
[0157] offset1=1<<(shift1-1).
[0158] displacementX=angleIdx.
[0159] displacementY=(angleIdx+8)%32.
[0160] partFlip=(angleIdx>=13&&angleIdx<=27)? 0:1.
[0161] shiftHor=(angleIdx%16==8||(angleIdx%16!=0&&nH>=nW))? 0:1.
[0162] Next, the variables offsetX and offsetY can be derived as follows:
[0163] If the value of shiftHor is 0:
[0164] offsetX=(-nW)>>1;
[0165] offsetY=((-nH)>>1)+(angleIdx<16?(distanceIdx*nH)>>3:-((distanceIdx*nH)>>3));
[0166] Otherwise (i.e. the value of shiftHor is 1):
[0167] offsetX=((-nW)>>1)+(angleIdx<16?(distanceIdx*nW)>>3:-((distanceIdx*nW)>>3);
[0168] offsetY=(-nH)>>1.
[0169] Then, the prediction sample matrix pbSamples[x][y] can be derived as follows; where x = 0..nCbW–1, y = 0..nCbH-1):
[0170] The variables xL and yL are derived as follows:
[0171] xL=(cIdx==0)? x:x*SubWidthC;
[0172] yL=(cIdx==0)? y:y*SubHeightC;
[0173] weightIdx=(((xL+offsetX)<<1)+1)*disLut[displacementX]+(((yL+offsetY)<<1)+1)*disLut[displacementY];
[0174] Among them, Table 2 disLut[displacementX]:
[0175] Table 2
[0176] idx02345681011121314disLut[idx]8884420-2-4-4-8-8idx161819202122242627282930disLut[idx]-8-8-8-4-4-2024488
[0177] weightIdxL=partFlip? 32+weightIdx:32–weightIdx;
[0178] wValue=Clip3(0,8,(weightIdxL+4)>>3);
[0179] pbSamples[x][y]=Clip3(0,(1<<BitDepth)-1,(predSamplesLA[x][y]*wValue+predSamplesLB[x][y]*(8-wValue)+offset1)> >shift1).
[0180] Where pbSamples[x][y] represents the predicted sample at point (x, y). wValue represents the weight of the predicted value predSamplesLA[x][y] of the prediction matrix for one prediction mode at point (x, y), and (8-wValue) represents the weight of the predicted value predSamplesLB[x][y] of the prediction matrix for another prediction mode at point (x, y).
[0181] It should be noted that for a weight derivation mode, it can be used to derive a weight value wValue for each point, and then calculate a GPM prediction value pbSamples[x][y]. In this way, the weight wValue does not have to be written in the form of a matrix, but it can be understood that if the wValue of each position is saved in a matrix, then it is a weight matrix. The principle is the same for calculating the weight of each point separately and weighting it to obtain the predicted value of GPM, or calculating all the weights and then uniformly weighting them to obtain the predicted sample matrix of GPM. The term weight matrix is used in many descriptions in this article to make the expression easier to understand, and drawing with the weight matrix is more intuitive. In fact, it can also be described according to the weight of each position. For example, the weight matrix derivation mode can also be said to be the weight derivation mode, and this application does not make specific restrictions on this.
[0182] In addition, the division of CU, PU and TU all belongs to the rectangular division method; however, GPM and AWP achieve the predicted non-rectangular division effect without division. GPM and AWP use the weight mask of two reference blocks, that is, the weight map or weight matrix mentioned above. This mask determines the weights of the two reference blocks when generating the prediction block, or it can be simply understood that part of the position of the prediction block comes from the first reference block and part of the position comes from the second reference block, and the transition area (blending area) is weighted by the corresponding positions of the two reference blocks to make the transition smoother. GPM and AWP do not divide the current block into two CUs or PUs, so the transformation, quantization, inverse transformation, inverse quantization, etc. of the residual after prediction also treat the current block as a whole.
[0183] It should be noted that GPM can combine two inter-frame prediction blocks using a weight matrix. This application extends this to combining two arbitrary prediction blocks. For example, two inter-frame prediction blocks, two intra-frame prediction blocks, or one inter-frame prediction block plus one intra-frame prediction block. Even in screen content encoding, prediction blocks in intra block copy (IBC) mode or palette mode can be used as one or two prediction blocks. For convenience of expression, this application collectively refers to intra-frame mode, inter-frame mode, IBC mode, and palette mode as prediction modes. A prediction mode can be understood as information based on which the codec can generate a prediction block for the current block. For example, in intra-frame prediction, the prediction mode can be a certain intra-frame prediction mode, such as DC mode, planar mode, various intra-frame angle prediction modes, etc. Of course, one or more auxiliary information can also be superimposed, such as the optimization method of intra-frame reference pixels, the optimization method after generating the preliminary prediction block (such as filtering), etc. Exemplarily, in inter-frame prediction, the prediction mode may be a merge mode, a Merge with Motion Vector Difference (MMVD) mode, or an Advanced Motion Vector Prediction (AMVP) mode. Exemplarily, the prediction mode may be a unidirectional prediction, a bidirectional prediction, or a multi-hypothesis prediction. Furthermore, if the inter-frame prediction mode uses a unidirectional prediction and one motion information can be determined, the prediction block can be determined based on the motion information. If the inter-frame prediction mode uses a bidirectional prediction and two motion information can be determined, the prediction block can be determined based on the motion information.
[0184] FIG11 is a schematic diagram of a GPM or AWP provided in an embodiment of the present application.
[0185] As shown in Figure 11, the information required by the GPM can be expressed as one weight derivation mode and two prediction modes. The weight derivation mode is used to determine the weight matrix or weights, while the two prediction modes each determine a prediction block or prediction value. The weight derivation mode is also called a partitioning mode or a weight matrix derivation mode. The two prediction modes can be the same or different prediction modes, including but not limited to intra-frame prediction mode, inter-frame prediction mode, IBC mode, and palette mode.
[0186] The following describes the content related to the transformation of the residual block.
[0187] During encoding, a prediction is first performed on the current block. This prediction leverages spatial or temporal correlation to produce an image identical or similar to the current block. While it's possible for the predicted block to be identical to the current block for a given block, it's difficult to guarantee this for all blocks in a video. This is particularly true for natural video, or video captured by cameras, where complex textures and noise often result in the predicted block being very similar to the current block, but with some differences. Furthermore, irregular motion, distortion, occlusion, and brightness changes in a video make it difficult to fully predict the current block. Therefore, a hybrid coding framework subtracts the predicted image from the original image of the current block to produce a residual image, or, in other words, the current block minus the predicted block to produce a residual block. This residual block is typically much simpler than the original image, so prediction can significantly improve compression efficiency. The residual block isn't encoded directly; instead, it's usually first transformed. This transform converts the residual image from the spatial domain to the frequency domain to remove correlation. After the residual image is transformed to the frequency domain, the energy is mostly concentrated in the low-frequency region, so the non-zero coefficients are mostly concentrated in the upper left corner. Quantization is then used to further compress the image. Since the human eye is not sensitive to high frequencies, a larger quantization step size can be used in high-frequency areas.
[0188] Image transformation technology transforms an original image to enable it to be represented using an orthogonal function or matrix. This transformation is linearly reversible in two dimensions. The original image is generally referred to as a spatial domain image, and the transformed image is called a transform domain image (also called a frequency domain image). The transform domain image can be reverse-transformed back to a spatial domain image. Image transformation not only more effectively reflects the image's characteristics, but also concentrates energy on a small amount of data, facilitating image storage, transmission, and processing.
[0189] In the field of image and video coding, after obtaining the residual block, the encoder can transform it. Transformations include, but are not limited to, discrete cosine transforms (DCT) and discrete sine transforms (DST). Due to the DCT's strong energy concentration characteristic, only certain regions (such as the upper left corner) of the original image have non-zero coefficients after the DCT transform. Of course, in video codecs, images are processed by dividing them into blocks, so transforms are also performed on a block-by-block basis. DCTs that can be used in video codecs include, but are not limited to, DCT2 and DCT8; and DSTs that can be used in video codecs include, but are not limited to, DST7. DCT2 is a common transform used in video compression standards; VVC can also use DCT8 and DST7. It's worth noting that while transforms are very useful in general video compression, not all blocks require them. In some cases, transforms can even yield less compression than non-transforms. Therefore, in some cases, the encoder can choose whether to use a transform for the current block.
[0190] When the encoder transforms the current block in the current image, it can use the basis function or the basis image to transform the residual block of the current block. The basis image is the image expression of the basis function.
[0191] FIG12 is an example of a DCT2 type base image provided in an embodiment of the present application.
[0192] As shown in Figure 12, the DCT2-type base image can be a base image composed of 8×8 small blocks drawn according to the basis function, where each small block is composed of 8×8 elements (sub-blocks). In a specific implementation, the base image composed of 8×8 small blocks can be used to transform the 8×8 block size, resulting in an 8×8 transform coefficient matrix.
[0193] As mentioned above, in addition to using DCT2 to perform basic transform on the residual block in VVC, DCT8 and DST7 can also be used to perform basic transform on the residual block, that is, the multiple transform selection (MTS) technology in VVC, where the transform type corresponding to the basis function used in the basic transform can also be called the transform kernel type used in the basic transform. When the encoder performs basic transform, based on different residual distribution characteristics, selecting the most appropriate transform kernel type can improve compression performance. The basic transform can also be called the main transform (core transform). MTS can select the transform kernel type through some syntax elements. The following is a list of MTS that selects the transform kernel type through syntax elements in combination with Table 3.
[0194] Table 3
[0195]
[0196]
[0197] As shown in Table 3, if the value of MTS_CU_flag is 0, the transform kernel type of the basic transform in both the horizontal and vertical directions is DCT2. If the value of MTS_CU_flag is 1, the value of MTS_Hor_flag is 0, and the value of MTS_Ver_flag is 0, the transform kernel type in the horizontal direction is DST7, and the transform kernel type in the vertical direction is DST7.
[0198] In the VVC standard, the syntax of MTS can also be rewritten or simplified, that is, VVC uses a syntax element mts_idx to determine the transform kernel type of the basic transform.
[0199] Table 4
[0200] mts_idx01234trTypeHor01212trTypeVer01122
[0201] As shown in Table 4, trTypeHor represents the transform kernel type of the horizontal transform, trTypeVer represents the transform kernel type of the vertical transform, and trTypeHor and trTypeVer are 0 for DCT2 type transform, 1 for DST7 type transform, and 2 for DCT8 type transform.
[0202] Since the distribution of residuals is correlated with the intra-frame prediction mode, the base transform can also exploit this correlation. One approach is to group the MTS transform kernel types based on the intra-frame prediction mode. An example of grouping is shown in the following table.
[0203] Table 5
[0204] 00<=mode<=112<=mode<=12213<=mode<=23324<=mode<=34
[0205] As shown in Table 5, if the intra prediction mode index is 0 or 1, the transform kernel type group with index 0 in the MTS is selected. In VVC, the mode with index 0 is Planar, and the mode with index 1 in VVC is DC. Both DC and Planar produce relatively flat prediction values. If the intra prediction mode index is 2 to 12, the transform kernel type group with index 1 in the MTS is selected. As can be seen from the intra prediction mode diagram, angles from 2 to 12 all point to the lower left.
[0206] It should be noted that each transform kernel type group may have one transform kernel type for horizontal and vertical transforms, or may have multiple transform kernel types for horizontal and vertical transforms. In other words, after selecting a transform kernel type group based on the intra-frame prediction mode, further subdivision can be done, such as further selecting the transform kernel type based on some identifiers or block size information, which will not be discussed here. The key point is that the basic transform can select a transform kernel type group based on the intra-frame prediction mode. It can also be seen that this method of selecting a transform kernel type group for the basic transform based on the intra-frame prediction mode may also lead to more detailed grouping of basic transforms in the future, and this application does not make specific limitations on this.
[0207] In addition, the transformation kernel type involved in the main transformation in this application may also be referred to as transformation matrix, transformation type or transformation kernel, etc., and the transformation kernel type group involved in the main transformation may also be referred to as transformation matrix group, transformation type group or transformation kernel group, etc., which have similar or identical meanings. This application does not make specific limitations on this. In other words, the selected transformation kernel type or transformation kernel type group involved in this application may also be referred to as selected transformation matrix or transformation matrix group, or may also be referred to as selected transformation type or selected transformation type group, or may also be referred to as selected transformation kernel or transformation kernel group. The transformation kernel type or transformation type may include DCT2, DCT8, DST7, etc., and may also include DCT5, DST4, DST1 or Identity Transform (IDTR), etc.
[0208] In addition, blocks of different sizes can use transform kernel types of corresponding sizes, which will not be described in detail in this application.
[0209] It is worth noting that since images are all two-dimensional, the computational complexity and memory overhead of directly performing a two-dimensional transformation are unacceptable to the hardware conditions. Therefore, the above-mentioned DCT2, DCT8, and DST7 transformations are all split into one-dimensional transformations in the horizontal and vertical directions, that is, they are performed in two steps. For example, the horizontal transformation is performed first and then the vertical transformation, or the vertical transformation is performed first and then the horizontal transformation. The above-mentioned transformation method is more effective for horizontal and vertical textures, but the effect is less effective for diagonal textures. Since horizontal and vertical textures are the most common, the above-mentioned transformation method is very useful for improving compression efficiency. However, with the development of technology, only processing the residuals of horizontal and vertical textures can no longer meet the demand for compression efficiency.
[0210] Based on this, the present application introduces the concept of secondary transform, that is, the encoder can perform secondary transform based on the primary transform to improve compression efficiency.
[0211] For example, a primary transform (also known as a base transform) can be used to process horizontal and vertical textures. For example, a primary transform includes, but is not limited to, the aforementioned DCT2, DCT8, and DST7 transforms. A secondary transform is used to process diagonal textures. For example, a secondary transform includes, but is not limited to, a low-frequency non-separable transform (LFNST). At the encoding end, the secondary transform is applied after the primary transform and before quantization. At the decoding end, the secondary transform is applied after inverse quantization and before the inverse of the primary transform.
[0212] FIG13 is an example of LFNST provided by an embodiment of the present application.
[0213] As shown in Figure 13, at the encoder, LFNST performs a secondary transform on the low-frequency coefficients in the upper left corner after the base transform. The primary transform decorrelates the image, concentrating energy in the upper left corner. The secondary transform further decorrelates the low-frequency coefficients of the primary transform. At the encoder, when 16 coefficients are input to a 4x4 LFNST, 8 coefficients are output; when 64 coefficients are input to an 8x8 LFNST, 16 coefficients are output. At the decoder, when 8 coefficients are input to a 4x4 inverse LFNST, 16 coefficients are output; when 16 coefficients are input to an 8x8 inverse LFNST, 64 coefficients are output.
[0214] When the encoder performs a secondary transform on a current block in the current image, it may use a transform matrix from the selected transform matrix group to transform the residual block of the current block. For example, when the secondary transform is LFNST, the transform matrix may be a matrix for transforming a certain diagonal texture, and the transform matrix group may include matrices for transforming certain similar diagonal textures.
[0215] FIG14 is an example of a transformation matrix group of LFNST provided in an embodiment of the present application.
[0216] As shown in (a) to (d) of Figure 14, LFNST may have four groups of transformation matrices, and the transformation matrices in the same group of transformation matrices have similar oblique textures. For example, the transformation matrix group shown in (a) of Figure 14 may be a transformation matrix group with an index of 0, the transformation matrix group shown in (b) of Figure 14 may be a transformation matrix group with an index of 1, the transformation matrix group shown in (c) of Figure 14 may be a transformation matrix group with an index of 2, and the transformation matrix group shown in (d) of Figure 14 may be a transformation matrix group with an index of 3.
[0217] It should be understood that the transformation matrix involved in the secondary transformation in this application may also be referred to as a transformation kernel, a transformation kernel type, or a basis function, etc., and the transformation matrix group involved in the secondary transformation may also be referred to as a transformation kernel group, a transformation kernel type group, or a basis function group, etc., which have similar or identical meanings. This application does not specifically limit this. In other words, the selected transformation matrix or transformation matrix group involved in this application may also be referred to as a selected transformation kernel type or a transformation kernel type group, or may also be referred to as a selected transformation type or a selected transformation type group, or may also be referred to as a selected transformation kernel or a transformation kernel group.
[0218] The following describes the application of LFNST to intra-coded blocks.
[0219] Intra-frame prediction uses the reconstructed pixels around the current block as reference to predict the current block. Since current videos are encoded from left to right and from top to bottom, the reference pixels that can be used by the current block are usually on the left and top. Angular prediction tiles the reference pixels to the current block at a specified angle as the prediction value, which means that the predicted block will have obvious directional texture, and the residual of the current block after angular prediction will also statistically show obvious angular characteristics. Therefore, the transformation matrix selected by LFNST can be bound to the intra-frame prediction mode. That is, after determining the intra-frame prediction mode, LFNST can use a set of transformation matrices whose texture direction is adapted to the angular characteristics of the intra-frame prediction mode.
[0220] For example, assuming that LFNST has a total of 4 groups of transformation matrices, each group has 2 transformation matrices. Table 6 shows the correspondence between intra prediction modes and transformation matrix groups.
[0221] Table 6
[0222]
[0223] As shown in Table 6, intra prediction modes 0 to 81 may be associated with four sets of transform matrix indices.
[0224] It is worth noting that the cross-component prediction modes used for chroma intra prediction are 81 to 83, while these modes are not available for luma intra prediction. The LFNST transform matrix can be transposed to handle more angles with one transform matrix group. For example, intra prediction modes 13 to 23 and intra prediction modes 45 to 55 both correspond to transform matrix group 2. However, intra prediction modes 13 to 23 are obviously close to horizontal modes, while intra prediction modes 45 to 55 are obviously close to vertical modes. The transform and inverse transform corresponding to intra prediction modes 45 to 55 need to be adapted by transposition.
[0225] In practice, since LFNST has four sets of transform matrices, the encoder can determine which set of transform matrices to use based on the intra-frame prediction mode used by the current block, and then determine the transform matrix to use from that set. This effectively leverages the correlation between the intra-frame prediction mode and the LFNST transform matrix set, reducing the transmission of the selected LFNST transform matrix in the bitstream. Whether LFNST is used for the current block, and if so, whether the first or second set is used, can be determined by the bitstream and certain conditions.
[0226] Of course, considering that there are 67 common intra-frame prediction modes and LFNST has only 4 sets of transformation matrices, multiple similar angle prediction modes can only correspond to one set of LFNST transformation matrices. This is a compromise between performance and complexity, because each transformation matrix requires storage space to store the coefficients of the transformation matrix. With the improvement of compression efficiency requirements and hardware capabilities, LFNST can also be designed to be more complex. For example, using larger transformation matrices, more transformation matrix groups, and using more transformation matrices in each transformation matrix group. For example, Table 7 shows another correspondence between intra-frame prediction modes and transformation matrix groups.
[0227] Table 7
[0228] Intra prediction mode -14-13-12-11-10-9-8-7-6-5-4-3-2-1 Index of transform matrix group 2222222222222 Intra prediction mode 012345678910111213 Index of transform matrix group 012345678910111213 Intra prediction mode 1415161718192021222324252627 Index of transform matrix group 1415161718192021222324252627 Intra prediction mode 28293031323334353 63738394041 Transformation matrix group index 2829303132333433323130292827 Intra prediction mode 4243444546474849505152535455 Transformation matrix group index 2625242322212019181716151413 Intra prediction mode 5657585960616263646566676869 Transformation matrix group index 12111098765432222 Intra prediction mode 7071727374757677787980 Transformation matrix group index 22222222222
[0229] As shown in Table 7, 35 transform matrix groups are used, each with three transform matrices. The correspondence between transform matrix groups and intra prediction modes can be implemented as follows: For intra prediction modes 0 to 34, the positive correspondence is to transform matrix groups 0 to 34, meaning that the larger the prediction mode number, the larger the transform matrix group index. For intra prediction modes 35 to 67, due to transposition, the negative correspondence is to transform matrix groups 2 to 33, meaning that the larger the prediction mode number, the smaller the transform matrix group index. For the remaining prediction modes, they can all be uniformly mapped to a transform matrix group with an index of 2. In other words, if transposition is not considered, each intra prediction mode corresponds to one transform matrix group. With this design, the residual corresponding to each intra prediction mode can obtain a more suitable LFNST transform matrix, improving compression performance.
[0230] Of course, in theory, the wide-angle mode can also achieve one-to-one, but the cost-effectiveness of this design is low, and this application will not provide further details on this.
[0231] In addition, for LFNST, in order to enable the MIP to adapt to the transformation matrix group, the transformation matrix group to which the planar mode is adapted in the present application can be used as the transformation matrix group adapted to the MIP.
[0232] It should be noted that LFNST is only an example of a secondary transform and should not be construed as limiting secondary transforms. For example, LFNST is a non-separable secondary transform. In other alternative embodiments, a separable secondary transform can be used to improve the compression efficiency of the residual error of oblique textures.
[0233] FIG15 is a schematic block diagram of a decoding framework 200 provided in an embodiment of the present application.
[0234] As shown in FIG15 , the decoding framework 200 may include an entropy decoding unit 210 , an inverse transform and dequantization unit 220 , a residual unit 230 , an intra-frame prediction unit 240 , an inter-frame prediction unit 250 , a loop filtering unit 260 , and a decoded image cache unit 270 .
[0235] After receiving and decoding the code stream, the entropy decoding unit 210 obtains a prediction block and a frequency domain residual block. For the frequency domain residual block, the inverse transform and inverse quantization unit 220 performs transformation and inverse quantization and other steps to obtain a time domain residual block. The residual unit 230 superimposes the prediction block predicted by the intra-frame prediction unit 240 or the inter-frame prediction unit 250 onto the time domain residual block after transformation and inverse quantization by the inverse transform and inverse quantization unit 220 to obtain a reconstructed block.
[0236] FIG16 is a schematic flow chart of a decoding method 300 provided in an embodiment of the present application. It should be understood that the decoding method 300 can be executed by a decoder. For example, it can be applied to the decoding framework 200 shown in FIG15 . For ease of description, the following description uses a decoder as an example.
[0237] As shown in FIG16 , the decoding method 300 may include some or all of the following:
[0238] S310, the decoder decodes the code stream to obtain the first transform coefficient of the current block;
[0239] S320: The decoder performs a first transform on the first transform coefficient to obtain a second transform coefficient of the current block;
[0240] S330: The decoder performs a second transform on the second transform coefficient to obtain a residual block of the current block;
[0241] S340, the decoder predicts the current block based on the first intra prediction mode and the second intra prediction mode derived from the prediction mode derivation mode to obtain a prediction block for the current block;
[0242] S350: The decoder obtains a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.
[0243] This application introduces a first transformation based on the prediction mode derivation mode and the second transformation, which can improve the decompression efficiency of the current block.
[0244] Exemplarily, the first transformation may be LFNST, that is, the present application combines LFNST with the prediction mode derivation mode, which can improve the compression efficiency of the residual of the oblique texture.
[0245] Of course, the adaptation method of the prediction mode derivation mode and LFNST is also applicable to other secondary transform methods. For example, LFNST is a non-separable secondary transform. In other alternative embodiments, the prediction mode derivation mode can also be applied to separable secondary transforms, which is not specifically limited in this application.
[0246] In some embodiments, the prediction mode derivation mode may include a decoder-side intra-mode derivation DIMD mode or a template-based intra-mode derivation TIMD mode.
[0247] In some embodiments, the S320 may include:
[0248] Decoding the code stream to obtain a first identifier and a second identifier;
[0249] If the first identifier is used to indicate that the prediction mode deriving mode is allowed to be used to predict the image blocks in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image blocks in the current sequence, then a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
[0250] Exemplarily, the current sequence is an image sequence including the current block.
[0251] Exemplarily, the first flag may be used to control whether the current sequence uses a prediction mode to derive a mode.
[0252] For example, if the value of the first identifier is a first value, the identifier allows the prediction mode derivation mode to be used for prediction of the image block in the current sequence; if the value of the first identifier is a second value, the identifier does not allow the prediction mode derivation mode to be used for prediction of the image block in the current sequence. As an implementation, the first value is 0 and the second value is 1. As another implementation, the first value is 1 and the second value is 0. Of course, the first value or the second value may also be other values.
[0253] Exemplarily, the second flag is used to control whether the current sequence uses the first transformation.
[0254] For example, if the value of the second flag is a third value, it indicates that the image blocks in the current sequence can be transformed using the first transform. If the value of the second flag is a fourth value, it indicates that the image blocks in the current sequence can be transformed using the first transform. In one implementation, the third value is 0 and the fourth value is 1. In another implementation, the third value is 1 and the fourth value is 0. Of course, the third value or the fourth value may also be other values.
[0255] Exemplarily, if the first flag is recorded as sps_timd / dimd_enabled_flag and the second flag is recorded as sps_lfnst_enabled_flag, when the values of sps_timd / dimd_enabled_flag and sps_lfnst_enabled_flag are both 1, a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
[0256] Exemplarily, if the first identifier is used to indicate that the prediction mode deriving mode is not allowed to be used to predict the image block in the current sequence, and / or the second identifier is used to indicate that the first transform is not allowed to be used to transform the image block in the current sequence, then the first transform coefficient is not subjected to the first transform, or in other words, the second transform can be directly performed on the first transform coefficient to obtain the residual value of the current block.
[0257] Of course, in other alternative embodiments, the first identifier and / or the second identifier may be replaced with identifiers at the levels of a picture, a slice, a largest coding unit (LCU), a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU). Alternatively, based on the first identifier and the second identifier, identifiers at the levels of a picture, a slice, an LCU, a CTU, a CU, a PU, or a TU may be added according to whether the prediction mode is used to derive the mode or whether the first transform is used. This embodiment of the present application does not specifically limit this.
[0258] In some embodiments, the S320 may include:
[0259] If the first identifier is used to indicate that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image block in the current sequence, decoding the bitstream to obtain a third identifier;
[0260] If the third identifier is used to indicate that the image block in the current sequence is allowed to use the prediction mode derivation mode and the first transform at the same time, a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
[0261] Exemplarily, the third flag is used to control whether the prediction mode derivation mode and the first transformation can be shared.
[0262] Exemplarily, if the value of the third flag is the fifth value, the flag allows the image blocks in the current sequence to use the prediction mode derivation mode and the first transform simultaneously; if the value of the third flag is the sixth value, the flag does not allow the image blocks in the current sequence to use the prediction mode derivation mode and the first transform simultaneously. As an implementation, the fifth value is 0 and the sixth value is 1. As another implementation, the fifth value is 1 and the sixth value is 0. Of course, the fifth value or the sixth value may also be other values.
[0263] Exemplarily, if the first identifier is recorded as sps_timd / dimd_enabled_flag, the second identifier is recorded as sps_lfnst_enabled_flag, and the third identifier is recorded as sps_timd / dimd_lfnst_enabled_flag, then when the values of sps_timd / dimd_enabled_flag and sps_lfnst_enabled_flag are both 1, determine whether sps_timd / dimd_lfnst_enabled_flag is 1, and when sps_timd / dimd_lfnst_enabled_flag is 1, perform a first transformation on the first transformation coefficient to obtain the second transformation coefficient.
[0264] Of course, in other alternative embodiments, the third identifier may be replaced with an identifier of a level such as a picture, a slice, a largest coding unit (LCU), a coding tree unit (CTU), a coding unit (CU), a prediction unit (PU), or a transform unit (TU). Alternatively, based on the third identifier, an identifier of a level such as a picture, a slice, an LCU, a CTU, a CU, a PU, or a TU may be added depending on whether the prediction mode is used to derive the mode or whether the first transform is used. This embodiment of the present application does not specifically limit this.
[0265] In some embodiments, the S320 may include:
[0266] If the first identifier is used to indicate that the prediction mode deriving mode is allowed to be used to predict the image block in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image block in the current sequence, then the decoder performs a first transform on the first transform coefficient to obtain the second transform coefficient when the height and / or width of the current block is greater than or equal to a first threshold.
[0267] Exemplarily, if the first identifier is recorded as sps_timd / dimd_enabled_flag, the second identifier is recorded as sps_lfnst_enabled_flag, and the third identifier is recorded as sps_timd / dimd_lfnst_enabled_flag, then when the values of sps_timd / dimd_enabled_flag and sps_lfnst_enabled_flag are both 1, the decoder determines the height and / or width of the current block, and when the height and / or width of the current block is greater than or equal to the first threshold, performs a first transform on the first transform coefficient to obtain the second transform coefficient.
[0268] Exemplarily, the first threshold may be 4, 8, 16, 32, 64 or other values.
[0269] In some embodiments, before S320, the method 300 may further include:
[0270] The decoder determines a set of transformation matrices used for the first transformation.
[0271] It should be noted that the prediction mode derivation mode combines two prediction modes (i.e., the first intra-frame prediction mode and the second intra-frame prediction mode) to predict the current block. The predicted blocks obtained by predicting the current block using different intra-frame prediction modes may have different texture characteristics. Therefore, if the prediction mode derivation mode is selected for the current block, the first intra-frame prediction mode may cause the predicted block of the current block to exhibit one texture characteristic, while the second intra-frame prediction mode may cause the predicted block of the current block to exhibit another texture characteristic. In other words, after the prediction of the current block, from a statistical perspective, the residual block of the current block will also exhibit two texture characteristics. In other words, the residual block of the current block does not necessarily conform to the pattern that can be reflected by a single prediction mode. Therefore, for the prediction mode derivation mode, before performing the first transform on the first transform coefficient, the decoder needs to first determine a transform matrix group that is adapted to its characteristics. However, since the transform matrix group used for the first transform is typically a transform matrix group defined based on a single intra-frame prediction mode, the prediction mode derivation mode requires further refinement of the scheme for determining the transform matrix used for the first transform. Various implementations are exemplified below.
[0272] In some embodiments, the transformation matrix group used in the first transformation is the same as the transformation matrix group used for planar mode or direct current (DC) mode adaptation.
[0273] Exemplarily, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder classifies the prediction mode derivation mode and the plane mode (or DC mode) into one category and adapts the transformation matrix group used for the first transformation according to the plane mode (or DC mode). In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the encoder can return the prediction mode of the current block as the plane mode (or DC mode), so that the decoder determines the transformation matrix group adapted to the plane mode (or DC mode) as the transformation matrix group used for the first transformation. In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder can consider that the transformation matrix group used in the first transformation for the current block can be the transformation matrix group adapted to the plane mode (or DC mode).
[0274] In this embodiment, since both the planar mode (or DC mode) and the prediction mode derivation mode can reflect a variety of texture characteristics, the transformation matrix group adapted to the planar mode or DC mode is determined as the transformation matrix group used by the first transformation. This not only enables decoding of the current block based on the prediction mode derivation mode and the first transformation, but also ensures that the texture characteristics of the transformation matrix group used by the first transformation are as close as possible to the texture characteristics of the residual block of the current block, thereby improving the decompression efficiency.
[0275] In some embodiments, the decoder determines a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode; wherein the transformation matrix group used by the first transformation is the same as the transformation matrix group adapted to the third intra prediction mode.
[0276] Exemplarily, the decoder may determine the transformation matrix group adapted to the third intra prediction mode as the transformation matrix group used by the first transformation.
[0277] Exemplarily, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder determines the third intra-frame prediction mode based on the first intra-frame prediction mode and the second intra-frame prediction mode, and classifies the prediction mode derivation mode and the third intra-frame prediction mode into one category, so that the decoder can adapt the transformation matrix group used for the first transform according to the third intra-frame prediction mode. In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the encoder can return the prediction mode of the current block as the third intra-frame prediction mode, so that the decoder determines the transformation matrix group adapted to the third intra-frame prediction mode as the transformation matrix group used for the first transform. In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder can consider that the transformation matrix group used in the first transform for the current block can be the transformation matrix group adapted to the third intra-frame prediction mode.
[0278] Of course, in other alternative embodiments, the decoder may not explicitly determine the third intra prediction mode first and then determine the transformation matrix group used for the first transformation based on the third intra prediction mode, but may directly use the transformation matrix group adapted to the third intra prediction mode as the transformation matrix group used for the first transformation.
[0279] In some embodiments, the decoder determines the default prediction mode between the first intra-frame prediction mode and the second intra-frame prediction mode as the third intra-frame prediction mode; or the decoder determines the third intra-frame prediction mode based on the weight of the first intra-frame prediction mode and / or the weight of the second intra-frame prediction mode; or the decoder determines the third intra-frame prediction mode based on the type of the first intra-frame prediction mode and the type of the second intra-frame prediction mode; or the decoder determines the third intra-frame prediction mode based on the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode.
[0280] Exemplarily, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, then when selecting the transformation matrix group used for the first transformation, the transformation matrix group used for the first transformation can be determined based on the first intra-frame prediction mode and the second intra-frame prediction mode. As an implementation method, the first intra-frame prediction mode can be used for determination in any case, that is, the transformation matrix group adapted to the first intra-frame prediction mode is determined as the transformation matrix group used for the first transformation in any case; or the second intra-frame prediction mode can be used for determination in any case, that is, the transformation matrix group adapted to the second intra-frame prediction mode is determined as the transformation matrix group used for the first transformation in any case. As another implementation, in some cases, the first intra-frame prediction mode can be used for determination, that is, in some cases, the transformation matrix group adapted to the first intra-frame prediction mode is determined as the transformation matrix group used for the first transformation; or in some cases, the second intra-frame prediction mode can be used for determination, that is, in some cases, the transformation matrix group adapted to the second intra-frame prediction mode is determined as the transformation matrix group used for the first transformation; or even in some cases, the planar mode or DC mode can be used for determination, that is, in some cases, the transformation matrix group adapted to the planar mode or DC mode is determined as the transformation matrix group used for the first transformation. The so-called determination by a certain prediction mode means that the prediction mode derivation mode and the certain prediction mode are classified into the same category, so that the decoder can adapt the transformation matrix group used for the first transformation according to the certain prediction mode. In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the certain prediction mode can be returned so that the decoder can adapt the transformation matrix group used for the first transformation according to the certain prediction mode. In other words, when the decoder checks the prediction mode of the current block, if the current block uses the prediction mode derivation mode, the decoder may consider that the transformation matrix group used in the first transformation for the current block may be a transformation matrix group adapted to the certain prediction mode.
[0281] In some embodiments, when the decoder determines the third intra-frame prediction mode based on the weight of the first intra-frame prediction mode and / or the weight of the second intra-frame prediction mode, the intra-frame prediction mode with the largest weight between the first intra-frame prediction mode and the second intra-frame prediction mode can be determined as the third intra-frame prediction mode.
[0282] Exemplarily, when the decoder determines the third intra-frame prediction mode based on the weight of the first intra-frame prediction mode and / or the weight of the second intra-frame prediction mode, the intra-frame prediction mode with a larger weight has a higher priority than the intra-frame prediction mode with a smaller weight. For example, if the weight of the first intra-frame prediction mode is greater than the weight of the second intra-frame prediction mode, the first intra-frame prediction mode is determined as the third intra-frame prediction mode; if the weight of the second intra-frame prediction mode is greater than the weight of the first intra-frame prediction mode, the second intra-frame prediction mode is determined as the third intra-frame prediction mode.
[0283] In some embodiments, when the decoder determines the third intra-frame prediction mode based on the type of the first intra-frame prediction mode and the type of the second intra-frame prediction mode, if the first intra-frame prediction mode and the second intra-frame prediction mode include an angular prediction mode and a non-angular prediction mode, the angular prediction mode is determined as the third intra-frame prediction mode.
[0284] Exemplarily, when the decoder determines the third intra-frame prediction mode based on the type of the first intra-frame prediction mode and the type of the second intra-frame prediction mode, the angular prediction mode has a higher priority as the third intra-frame prediction mode than the non-angular prediction mode. For example, if the first intra-frame prediction mode is an angular prediction mode and the second intra-frame prediction mode is a non-angular prediction mode (such as a planar mode or a DC mode), the first intra-frame prediction mode (i.e., the angular prediction mode) is determined as the third intra-frame prediction mode.
[0285] In some embodiments, if the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is less than or equal to a second threshold, the intra-frame prediction mode corresponding to the first prediction angle is determined as the third intra-frame prediction mode, wherein the first prediction angle is determined based on the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode; if the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is greater than the second threshold, the planar mode or the DC mode is determined as the third intra-frame prediction mode.
[0286] Exemplarily, when the decoder determines the third intra-frame prediction mode based on the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode, if the prediction angle of the first intra-frame prediction mode is close to the prediction angle of the second intra-frame prediction mode, the decoder may select the first intra-frame prediction mode, the second intra-frame prediction mode, or an intra-frame prediction mode whose prediction angle is between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode. For example, if the prediction angle of the first intra-frame prediction mode is significantly different from the prediction angle of the second intra-frame prediction mode, the decoder may determine the planar mode or the DC mode as the third intra-frame prediction mode.
[0287] In some embodiments, the decoder determines the transformation matrix group adapted to the prediction mode derivation mode as the transformation matrix group used for the first transformation.
[0288] For example, the decoder may derive a set of transform matrices adapted or dedicated to the prediction mode definition.
[0289] In some embodiments, the first transform is used to process textures in the current block along an oblique direction, and the second transform is used to process textures in the current block along a horizontal direction and textures in a vertical direction.
[0290] It should be understood that the first transform at the decoding end is the inverse transform of the first transform at the encoding end, and the second transform at the decoding end is the inverse transform of the second transform at the encoding end. For example, for the encoding end, the first transform is the secondary transform mentioned above, and the second transform can be the basic transform or main transform mentioned above. Then, for the decoding end, the first transform can be the inverse transform or inverse transform of the secondary transform mentioned above. For example, for the encoding end, the first transform can be an inverse LFNST, and the second transform can be a DCT2 type, a DCT8 type, or a DST7 type, etc.; then, for the decoder, the first transform can be an inverse (inverse) LFNST, and the second transform can be an inverse (inverse) DCT2 type, an inverse (inverse) DCT8 type, or an inverse (inverse) DST7 type, etc.
[0291] The decoding method according to the embodiment of the present application is described in detail above from the perspective of the decoder. The encoding method according to the embodiment of the present application will be described below from the perspective of the encoder in conjunction with Figure 17.
[0292] FIG17 is a schematic flow chart of an encoding method 400 provided in an embodiment of the present application. It should be understood that the encoding method 400 can be performed by an encoder. For example, it can be applied to the encoding framework 100 shown in FIG1 . For ease of description, the following description uses an encoder as an example.
[0293] As shown in FIG. 17 , the encoding method 400 may include:
[0294] S410, predicting a current block based on a first intra prediction mode and a second intra prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block;
[0295] S420, obtaining a residual block of the current block based on the prediction block of the current block;
[0296] S430, performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;
[0297] S440, performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block;
[0298] S450: Encode the fourth transform coefficient.
[0299] It should be understood that the first transform at the decoding end is the inverse transform of the fourth transform at the encoding end, and the second transform at the decoding end is the inverse transform of the third transform at the encoding end. For example, the third transform is the basic transform or main transform mentioned above, and the fourth transform is the secondary transform mentioned above. Accordingly, the first transform is the inverse transform (or inverse transform) of the secondary transform, and the second transform can be the inverse transform (or inverse transform) of the basic transform or the main transform. For example, the first transform can be an inverse (inverse) LFNST, and the second transform can be an inverse (inverse) DCT2 type, an inverse (inverse) DCT8 type, or an inverse (inverse) DST7 type, etc.; accordingly, the third transform can be a DCT2 type, a DCT8 type, or a DST7 type, etc., and the fourth transform can be a LFNST.
[0300] In some embodiments, the prediction mode derivation mode comprises a decoder-side intra-mode derivation mode or a template-based intra-mode derivation mode.
[0301] In some embodiments, the S450 may include:
[0302] encoding the first identifier, the second identifier, and the fourth transform coefficient;
[0303] The first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image blocks in the current sequence, and the second identifier is used to identify that the fourth transformation is allowed to be used to transform the image blocks in the current sequence.
[0304] In some embodiments, the S450 may include:
[0305] encoding the first identifier, the second identifier, the fourth transform coefficient, and the third identifier;
[0306] The third identifier is used to indicate that the image blocks in the current sequence are allowed to use the prediction mode derivation mode and the fourth transformation simultaneously.
[0307] In some embodiments, the S440 may include:
[0308] In a case where the height and / or width of the current block is greater than or equal to a first threshold, the fourth transform is performed on the third transform coefficient to obtain the fourth transform coefficient.
[0309] In some embodiments, before S440, the method 400 may further include:
[0310] A transformation matrix group used for the fourth transformation is determined.
[0311] In some embodiments, the transformation matrix group used by the fourth transformation is the same as the transformation matrix group adapted for the planar mode or the direct current (DC) mode.
[0312] In some embodiments, determining a transformation matrix group used for the fourth transformation includes:
[0313] Determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode;
[0314] The transformation matrix group used by the fourth transformation is the same as the transformation matrix group adapted to the third intra prediction mode.
[0315] In some embodiments, determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode includes:
[0316] determining a default prediction mode between the first intra-frame prediction mode and the second intra-frame prediction mode as the third intra-frame prediction mode; or
[0317] determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode; or
[0318] determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode; or
[0319] The third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
[0320] In some embodiments, determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode includes:
[0321] The intra-frame prediction mode with the largest weight between the first intra-frame prediction mode and the second intra-frame prediction mode is determined as the third intra-frame prediction mode.
[0322] In some embodiments, determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode includes:
[0323] If the first intra prediction mode and the second intra prediction mode include an angular prediction mode and a non-angular prediction mode, the angular prediction mode is determined as the third intra prediction mode.
[0324] In some embodiments, determining the third intra prediction mode based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode includes:
[0325] If an absolute value of a difference between a prediction angle of the first intra-frame prediction mode and a prediction angle of the second intra-frame prediction mode is less than or equal to a second threshold, determining the intra-frame prediction mode corresponding to the first prediction angle as the third intra-frame prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode:
[0326] If the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is greater than the second threshold, the planar mode or the DC mode is determined as the third intra-frame prediction mode.
[0327] In some embodiments, determining a transformation matrix group used for the fourth transformation includes:
[0328] The transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the fourth transformation.
[0329] In some embodiments, the fourth transform is used to process textures in the current block along an oblique direction, and the third transform is used to process textures in the current block along a horizontal direction and a vertical direction.
[0330] It should be understood that the encoding method can be understood as the inverse process of the decoding method. Therefore, the specific scheme of the encoding method 400 can refer to the relevant content of the decoding method 300. For the sake of ease of description, this application will not go into details.
[0331] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will no longer describe the various possible combinations separately. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the idea of the present application, they should also be regarded as the contents disclosed in the present application. It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0332] The method embodiment of the present application is described in detail above. The following describes in detail an apparatus embodiment of the present application in conjunction with Figures 18 to 20.
[0333] FIG18 is a schematic block diagram of a decoder 500 according to an embodiment of the present application.
[0334] As shown in FIG18 , the decoder 500 may include:
[0335] A decoding unit 510 is configured to decode a bitstream to obtain a first transform coefficient of a current block;
[0336] The transformation unit 520 is configured to:
[0337] Performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block;
[0338] Performing a second transform on the second transform coefficient to obtain a residual block of the current block;
[0339] A prediction unit 530 is configured to predict the current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block;
[0340] The reconstruction unit 540 is configured to obtain a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.
[0341] In some embodiments, the prediction mode derivation mode comprises a decoder-side intra-mode derivation mode or a template-based intra-mode derivation mode.
[0342] In some embodiments, the transform unit 520 is specifically configured to:
[0343] Decoding the code stream to obtain a first identifier and a second identifier;
[0344] If the first identifier is used to indicate that the prediction mode deriving mode is allowed to be used to predict the image blocks in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image blocks in the current sequence, then a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
[0345] In some embodiments, the transform unit 520 is specifically configured to:
[0346] If the first identifier is used to indicate that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image block in the current sequence, decoding the bitstream to obtain a third identifier;
[0347] If the third identifier is used to indicate that the image block in the current sequence is allowed to use the prediction mode derivation mode and the first transform at the same time, a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
[0348] In some embodiments, the transform unit 520 is specifically configured to:
[0349] If the first identifier is used to indicate that the prediction mode deriving mode is allowed to be used to predict the image block in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image block in the current sequence, then when the height and / or width of the current block is greater than or equal to a first threshold, the first transform coefficient is subjected to a first transform to obtain the second transform coefficient.
[0350] In some embodiments, before the transform unit 520 performs a second transform on the second transform coefficient to obtain the residual block of the current block, it is further configured to:
[0351] A transformation matrix group used for the first transformation is determined.
[0352] In some embodiments, the transform unit 520 is specifically configured to:
[0353] In some embodiments, the transformation matrix group used in the first transformation is the same as the transformation matrix group used for planar mode or direct current (DC) mode adaptation.
[0354] In some embodiments, the transform unit 520 is specifically configured to:
[0355] Determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode;
[0356] The transformation matrix group used in the first transformation is the same as the transformation matrix group adapted to the third intra prediction mode.
[0357] In some embodiments, the transform unit 520 is specifically configured to:
[0358] determining a default prediction mode between the first intra-frame prediction mode and the second intra-frame prediction mode as the third intra-frame prediction mode; or
[0359] determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode; or
[0360] determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode; or
[0361] The third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
[0362] In some embodiments, the transform unit 520 is specifically configured to:
[0363] The intra-frame prediction mode with the largest weight between the first intra-frame prediction mode and the second intra-frame prediction mode is determined as the third intra-frame prediction mode.
[0364] In some embodiments, the transform unit 520 is specifically configured to:
[0365] If the first intra prediction mode and the second intra prediction mode include an angular prediction mode and a non-angular prediction mode, the angular prediction mode is determined as the third intra prediction mode.
[0366] In some embodiments, the transform unit 520 is specifically configured to:
[0367] If an absolute value of a difference between a prediction angle of the first intra-frame prediction mode and a prediction angle of the second intra-frame prediction mode is less than or equal to a second threshold, determining the intra-frame prediction mode corresponding to the first prediction angle as the third intra-frame prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode:
[0368] If the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is greater than the second threshold, the planar mode or the DC mode is determined as the third intra-frame prediction mode.
[0369] In some embodiments, the transform unit 520 is specifically configured to:
[0370] The transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the first transformation.
[0371] In some embodiments, the first transform is used to process textures in the current block along an oblique direction, and the second transform is used to process textures in the current block along a horizontal direction and textures in a vertical direction.
[0372] FIG19 is a schematic block diagram of an encoder 600 according to an embodiment of the present application.
[0373] As shown in FIG19 , the encoder 600 may include:
[0374] A prediction unit 610 is configured to predict a current block based on a first intra prediction mode and a second intra prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block;
[0375] A residual unit 620, configured to obtain a residual block of the current block based on the prediction block of the current block;
[0376] The transformation unit 630 is configured to:
[0377] Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block;
[0378] Performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block;
[0379] The encoding unit 640 is configured to encode the fourth transform coefficient.
[0380] In some embodiments, the prediction mode derivation mode comprises a decoder-side intra-mode derivation mode or a template-based intra-mode derivation mode.
[0381] In some embodiments, the encoding unit 640 is specifically configured to:
[0382] encoding the first identifier, the second identifier, and the fourth transform coefficient;
[0383] The first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image blocks in the current sequence, and the second identifier is used to identify that the fourth transformation is allowed to be used to transform the image blocks in the current sequence.
[0384] In some embodiments, the encoding unit 640 is specifically configured to:
[0385] encoding the first identifier, the second identifier, the fourth transform coefficient, and the third identifier;
[0386] The third identifier is used to indicate that the image blocks in the current sequence are allowed to use the prediction mode derivation mode and the fourth transformation simultaneously.
[0387] In some embodiments, the transform unit 630 is specifically configured to:
[0388] In a case where the height and / or width of the current block is greater than or equal to a first threshold, the fourth transform is performed on the third transform coefficient to obtain the fourth transform coefficient.
[0389] In some embodiments, before performing a fourth transform on the third transform coefficient to obtain the fourth transform coefficient of the current block, the transform unit 630 is further configured to:
[0390] A transformation matrix group used for the fourth transformation is determined.
[0391] In some embodiments, the transformation matrix group used by the fourth transformation is the same as the transformation matrix group adapted for the planar mode or the direct current (DC) mode.
[0392] In some embodiments, the transform unit 630 is specifically configured to:
[0393] Determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode;
[0394] The transformation matrix group used by the fourth transformation is the same as the transformation matrix group adapted to the third intra prediction mode.
[0395] In some embodiments, the transform unit 630 is specifically configured to:
[0396] determining a default prediction mode between the first intra-frame prediction mode and the second intra-frame prediction mode as the third intra-frame prediction mode; or
[0397] determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode; or
[0398] determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode; or
[0399] The third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
[0400] In some embodiments, the transform unit 630 is specifically configured to:
[0401] The intra-frame prediction mode with the largest weight between the first intra-frame prediction mode and the second intra-frame prediction mode is determined as the third intra-frame prediction mode.
[0402] In some embodiments, the transform unit 630 is specifically configured to:
[0403] If the first intra prediction mode and the second intra prediction mode include an angular prediction mode and a non-angular prediction mode, the angular prediction mode is determined as the third intra prediction mode.
[0404] In some embodiments, the transform unit 630 is specifically configured to:
[0405] If an absolute value of a difference between a prediction angle of the first intra-frame prediction mode and a prediction angle of the second intra-frame prediction mode is less than or equal to a second threshold, determining the intra-frame prediction mode corresponding to the first prediction angle as the third intra-frame prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode:
[0406] If the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is greater than the second threshold, the planar mode or the DC mode is determined as the third intra-frame prediction mode.
[0407] In some embodiments, the transform unit 630 is specifically configured to:
[0408] The transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the fourth transformation.
[0409] In some embodiments, the fourth transform is used to process textures in the current block along an oblique direction, and the third transform is used to process textures in the current block along a horizontal direction and a vertical direction.
[0410] It should be understood that the device embodiment and the method embodiment can correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, the decoder 500 shown in Figure 18 can correspond to the corresponding subject in the method 300 for executing the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the decoder 500 are respectively for implementing the corresponding processes in various methods such as method 300. Similarly, the encoder 600 shown in Figure 19 can correspond to the corresponding subject in the method 400 for executing the embodiment of the present application, that is, the aforementioned and other operations and / or functions of the various units in the encoder 600 are respectively for implementing the corresponding processes in various methods such as method 400.
[0411] It should also be understood that the various units in the decoder 500 or encoder 600 involved in the embodiment of the present application can be separately or all merged into one or several other units to constitute, or some (some) units therein can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by multiple units, or the function of multiple units is realized by one unit. In other embodiments of the present application, the decoder 500 or encoder 600 can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units. According to another embodiment of the present application, the decoder 500 or encoder 600 involved in the embodiment of the present application can be constructed by running a computer program (including program code) that can execute each step involved in the corresponding method on a general-purpose computing device including processing elements and storage elements such as a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM), etc., to construct the decoder 500 or encoder 600 involved in the embodiment of the present application, and to implement the encoding method or decoding method of the embodiment of the present application. The computer program can be recorded on, for example, a computer-readable storage medium, loaded into an electronic device via the computer-readable storage medium, and run therein to implement the corresponding method of the embodiment of the present application.
[0412] In other words, the units mentioned above can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented in the form of a combination of hardware and software. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software in a decoding processor. Optionally, the software can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in combination with its hardware.
[0413] FIG20 is a schematic structural diagram of an electronic device 700 provided in an embodiment of the present application.
[0414] As shown in Figure 20, the electronic device 700 includes at least a processor 710 and a computer-readable storage medium 720. The processor 710 and the computer-readable storage medium 720 may be connected via a bus or other means. The computer-readable storage medium 720 is used to store a computer program 721, which includes computer instructions. The processor 710 is used to execute the computer instructions stored in the computer-readable storage medium 720. The processor 710 is the computing core and control core of the electronic device 700. It is suitable for implementing one or more computer instructions, and is specifically suitable for loading and executing one or more computer instructions to implement the corresponding method flow or corresponding function.
[0415] As an example, the processor 710 may also be referred to as a central processing unit (CPU). The processor 710 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.
[0416] As an example, the computer-readable storage medium 720 may be a high-speed RAM memory, or a non-volatile memory (Non-Volatile Memory), such as at least one disk storage; optionally, it may also be at least one computer-readable storage medium located away from the aforementioned processor 710. Specifically, the computer-readable storage medium 720 includes, but is not limited to: volatile memory and / or non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).
[0417] In one implementation, the electronic device 700 may be an encoder or encoding framework involved in an embodiment of the present application; a first computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the first computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the encoding method provided in an embodiment of the present application; in other words, the first computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed. To avoid repetition, they are not repeated here.
[0418] In one implementation, the electronic device 700 may be a decoder or decoding framework involved in an embodiment of the present application; a second computer instruction is stored in the computer-readable storage medium 720; the processor 710 loads and executes the second computer instruction stored in the computer-readable storage medium 720 to implement the corresponding steps in the decoding method provided in an embodiment of the present application; in other words, the second computer instruction in the computer-readable storage medium 720 is loaded by the processor 710 and the corresponding steps are executed, which will not be repeated here to avoid repetition.
[0419] According to another aspect of the present application, an embodiment of the present application further provides a coding and decoding system, including the encoder and decoder mentioned above.
[0420] According to another aspect of the present application, an embodiment of the present application further provides a computer-readable storage medium (Memory), which is a memory device in the electronic device 700 for storing programs and data. For example, a computer-readable storage medium 720. It can be understood that the computer-readable storage medium 720 here can include both the built-in storage medium in the electronic device 700 and, of course, the extended storage medium supported by the electronic device 700. The computer-readable storage medium provides a storage space, which stores the operating system of the electronic device 700. In addition, one or more computer instructions suitable for being loaded and executed by the processor 710 are also stored in the storage space. These computer instructions can be one or more computer programs 721 (including program codes).
[0421] According to another aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. For example, computer program 721 is provided. In this case, data processing device 700 may be a computer, processor 710 may read the computer instructions from computer-readable storage medium 720, and processor 710 may execute the computer instructions, causing the computer to perform the encoding method or decoding method provided in the various optional embodiments described above.
[0422] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process of the embodiment of the present application is run in whole or in part or the functions of the embodiment of the present application are implemented. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode.
[0423] Those skilled in the art will appreciate that the units and process steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0424] Finally, it should be noted that the above content is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.
Claims
1. A decoding method, characterized in that: include: Decoding the code stream to obtain the first transform coefficient of the current block; Performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block; Performing a second transform on the second transform coefficient to obtain a residual block of the current block; Predicting the current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block; A reconstructed block of the current block is obtained based on the prediction block of the current block and the residual block of the current block.
2. The method according to claim 1, characterized in that The prediction mode derivation mode includes a decoder-side intra-frame mode derivation mode or a template-based intra-frame mode derivation mode.
3. The method according to claim 1 or 2, characterized in that The performing a second transform on the second transform coefficient to obtain a residual block of the current block includes: Decoding the code stream to obtain a first identifier and a second identifier; If the first identifier is used to indicate that the prediction mode deriving mode is allowed to be used to predict the image blocks in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image blocks in the current sequence, then a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
4. The method according to claim 3, characterized in that The performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block includes: If the first identifier is used to indicate that the prediction mode derivation mode is allowed to be used to predict the image block in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image block in the current sequence, decoding the bitstream to obtain a third identifier; If the third identifier is used to indicate that the image block in the current sequence is allowed to use the prediction mode derivation mode and the first transform at the same time, a first transform is performed on the first transform coefficient to obtain the second transform coefficient.
5. The method according to claim 3, characterized in that The performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block includes: If the first identifier is used to indicate that the prediction mode deriving mode is allowed to be used to predict the image block in the current sequence, and the second identifier is used to indicate that the first transform is allowed to be used to transform the image block in the current sequence, then when the height and / or width of the current block is greater than or equal to a first threshold, the first transform coefficient is subjected to a first transform to obtain the second transform coefficient.
6. The method according to any one of claims 1 to 5, characterized in that Before performing a second transform on the second transform coefficient to obtain a residual block of the current block, the method further includes: A transformation matrix group used for the first transformation is determined.
7. The method according to claim 6, characterized in that The transformation matrix group used in the first transformation is the same as the transformation matrix group adapted for the planar mode or the DC mode.
8. The method according to claim 6, characterized in that The determining of the transformation matrix group used for the first transformation includes: Determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode; The transformation matrix group used in the first transformation is the same as the transformation matrix group adapted to the third intra prediction mode.
9. The method according to claim 8, characterized in that The determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode includes: determining a default prediction mode between the first intra-frame prediction mode and the second intra-frame prediction mode as the third intra-frame prediction mode; or determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode; or determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode; or The third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
10. The method according to claim 9, characterized in that The determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode includes: The intra-frame prediction mode with the largest weight between the first intra-frame prediction mode and the second intra-frame prediction mode is determined as the third intra-frame prediction mode.
11. The method according to claim 9, characterized in that The determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode includes: If the first intra prediction mode and the second intra prediction mode include an angular prediction mode and a non-angular prediction mode, the angular prediction mode is determined as the third intra prediction mode.
12. The method according to claim 9, characterized in that The determining the third intra prediction mode based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode includes: If an absolute value of a difference between a prediction angle of the first intra-frame prediction mode and a prediction angle of the second intra-frame prediction mode is less than or equal to a second threshold, determining the intra-frame prediction mode corresponding to the first prediction angle as the third intra-frame prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode: If the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is greater than the second threshold, the planar mode or the DC mode is determined as the third intra-frame prediction mode.
13. The method according to claim 6, characterized in that The determining of the transformation matrix group used for the first transformation includes: The transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the first transformation.
14. The method according to any one of claims 1 to 13, characterized in that The first transform is used to process textures in the current block along an oblique direction, and the second transform is used to process textures in the current block along a horizontal direction and a vertical direction.
15. A coding method, characterized in that: include: Predicting a current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode to obtain a prediction block of the current block; Obtaining a residual block of the current block based on the prediction block of the current block; Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block; Performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block; The fourth transform coefficients are encoded.
16. The method according to claim 15, characterized in that The prediction mode derivation mode includes a decoder-side intra-frame mode derivation mode or a template-based intra-frame mode derivation mode.
17. The method according to claim 15 or 16, characterized in that The encoding of the fourth transform coefficient comprises: Encoding the first identifier, the second identifier, and the fourth transform coefficient; The first identifier is used to identify that the prediction mode derivation mode is allowed to be used to predict the image blocks in the current sequence, and the second identifier is used to identify that the fourth transformation is allowed to be used to transform the image blocks in the current sequence.
18. The method according to claim 17, characterized in that The encoding of the first identifier, the second identifier, and the fourth transform coefficient includes: encoding the first identifier, the second identifier, the fourth transform coefficient, and the third identifier; The third identifier is used to indicate that the image blocks in the current sequence are allowed to use the prediction mode derivation mode and the fourth transformation simultaneously.
19. The method according to any one of claims 15 to 18, characterized in that The performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block includes: In a case where the height and / or width of the current block is greater than or equal to a first threshold, the fourth transform is performed on the third transform coefficient to obtain the fourth transform coefficient.
20. The method according to any one of claims 15 to 19, characterized in that Before performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block, the method further includes: A transformation matrix group used for the fourth transformation is determined.
21. The method according to claim 20, characterized in that The transformation matrix group used in the fourth transformation is the same as the transformation matrix group adapted for the planar mode or the DC mode.
22. The method according to claim 20, characterized in that The determining of the transformation matrix group used for the fourth transformation includes: Determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode; The transformation matrix group used by the fourth transformation is the same as the transformation matrix group adapted to the third intra prediction mode.
23. The method according to claim 22, characterized in that The determining a third intra prediction mode based on the first intra prediction mode and the second intra prediction mode includes: determining a default prediction mode between the first intra-frame prediction mode and the second intra-frame prediction mode as the third intra-frame prediction mode; or determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode; or determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode; or The third intra prediction mode is determined based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode.
24. The method according to claim 23, wherein The determining the third intra prediction mode based on the weight of the first intra prediction mode and / or the weight of the second intra prediction mode includes: The intra-frame prediction mode with the largest weight between the first intra-frame prediction mode and the second intra-frame prediction mode is determined as the third intra-frame prediction mode.
25. The method according to claim 23, characterized in that The determining the third intra prediction mode based on the type of the first intra prediction mode and the type of the second intra prediction mode includes: If the first intra prediction mode and the second intra prediction mode include an angular prediction mode and a non-angular prediction mode, the angular prediction mode is determined as the third intra prediction mode.
26. The method according to claim 23, wherein The determining the third intra prediction mode based on the prediction angle of the first intra prediction mode and the prediction angle of the second intra prediction mode includes: If an absolute value of a difference between a prediction angle of the first intra-frame prediction mode and a prediction angle of the second intra-frame prediction mode is less than or equal to a second threshold, determining the intra-frame prediction mode corresponding to the first prediction angle as the third intra-frame prediction mode, wherein the first prediction angle is determined according to the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode: If the absolute value of the difference between the prediction angle of the first intra-frame prediction mode and the prediction angle of the second intra-frame prediction mode is greater than the second threshold, the planar mode or the DC mode is determined as the third intra-frame prediction mode.
27. The method according to claim 20, characterized in that The determining of the transformation matrix group used for the fourth transformation includes: The transformation matrix group adapted to the prediction mode derivation mode is determined as the transformation matrix group used for the fourth transformation.
28. The method according to any one of claims 15 to 27, characterized in that The fourth transform is used to process the texture in the current block along an oblique direction, and the third transform is used to process the texture in the current block along a horizontal direction and a texture in a vertical direction.
29. A decoder, characterized in that include: A decoding unit, configured to decode a code stream to obtain a first transform coefficient of a current block; Transformation unit, used to: Performing a first transform on the first transform coefficient to obtain a second transform coefficient of the current block; Performing a second transform on the second transform coefficient to obtain a residual block of the current block; A prediction unit, configured to predict the current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode, to obtain a prediction block of the current block; The reconstruction unit is configured to obtain a reconstructed block of the current block based on the prediction block of the current block and the residual block of the current block.
30. An encoder, characterized in that include: A prediction unit, configured to predict a current block based on a first intra-frame prediction mode and a second intra-frame prediction mode derived from a prediction mode derivation mode, to obtain a prediction block of the current block; A residual unit, configured to obtain a residual block of the current block based on a prediction block of the current block; Transformation unit, used to: Performing a third transform on the residual block of the current block to obtain a third transform coefficient of the current block; Performing a fourth transform on the third transform coefficient to obtain a fourth transform coefficient of the current block; A coding unit is used to encode the fourth transformation coefficient.
31. An electronic device, characterized in that: include: a processor adapted to execute a computer program; A computer-readable storage medium having a computer program stored therein, wherein when the computer program is executed by the processor, the method according to any one of claims 1 to 14 or the method according to any one of claims 15 to 28 is implemented.
32. A computer-readable storage medium, characterized in that Used to store a computer program, the computer program causing a computer to execute the method according to any one of claims 1 to 14 or the method according to any one of claims 15 to 28.
33. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 14 or the method according to any one of claims 15 to 28 is implemented.
34. A code stream, characterized in that The code stream is the code stream described in the method according to any one of claims 1 to 14 or the code stream generated by the method according to any one of claims 15 to 28.
Citation Information
Cited By
Encoding method, decoding method, encoder, decoder, storage medium, and bitstream
WO2026137114A1
Decoding method and apparatus, encoding method and apparatus, and corresponding encoder and decoder
WO2026179213A1
