Video encoding and decoding method and device based on prediction enhancement

Abstract

The present invention proposes a video encoding and decoding method and device based on prediction enhancement, which belongs to the field of video processing technology. The video decoding method includes: decoding the code stream generated by the video encoder, obtaining the prediction information and flag of the current block, and the residual value of the current block; predicting the current block based on the prediction information of the current block to generate a prediction value; if the flag shows that the prediction enhancement module is turned on, using the available spatial neighbors of the current block to enhance the prediction value; adding the residual value of the current block to the prediction value to reconstruct the current block. The video encoding method includes: after the intra-frame predictor obtains the prediction value of the current block, the prediction value is input into the prediction enhancement module for enhancement; according to the prediction values ​​before and after enhancement, the final prediction value is selected and the flag indicating whether the prediction enhancement is turned on is set; the final prediction value is subtracted from the original signal to obtain the residual value, and after transformation and quantization, entropy coding is performed. The present invention improves the encoding and decoding performance.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video prediction enhancement method combining spatial information in a video codec framework. Background Art

[0002] Raw video data is much larger than images or audio data. Uncompressed video data is difficult to store and cannot be directly transmitted over a network. Therefore, to achieve efficient data management, a video encoder is typically used to encode the raw video data for efficient storage or transmission. A decoder then receives the encoded bitstream, decodes it, and produces the compressed video through a player.

[0003] Traditional video codecs utilize a block-based hybrid coding framework, which is typically standardized. The general video coding standard H.266 / VVC, AV1, and AVS3 are the latest video coding standards developed by JVET (composed of ITU-T VCEG and ISO / IEC MPEG), AOMedia, and China, respectively. Currently, JVET, building on H.266 / VVC, is focusing its video coding standards development on two main tracks: Beyond VVC, exploring more advanced non-neural network coding tools; and NNVC, exploring neural network-based coding tools.

[0004] However, with the continuous improvement of video image size, resolution and playback frame rate, the amount of data to be encoded is also showing a rapid growth trend. Therefore, a new codec technology is needed that can provide higher encoding and decoding efficiency and improve video image enhancement effects compared to existing compression technologies. In traditional intra-frame prediction methods, for the current coding unit in the video to be encoded, a prediction unit is usually generated from adjacent reference samples based on a fixed rule. However, this method may have difficulty generating accurate prediction units when processing complex textures. In addition, since the amount of information that can be used as reference samples in intra-frame prediction is often lower than the amount of information between temporally adjacent frames, the encoding and decoding performance of intra-frame prediction methods is generally lower than that of inter-frame prediction methods. Therefore, prediction enhancement methods can be used to reduce the residual signal between the original block by improving the generation of prediction units.

[0005] At present, the existing prediction enhancement schemes mainly focus on inter-frame prediction enhancement, which usually enhances motion compensation through neural networks. These enhancement operations are usually performed after the motion compensation is completed for the inter-frame coded square blocks. Different neural networks need to be trained independently for different coding block sizes and different quantization parameters (QP). Existing technologies usually perform residual enhancement by stacking residual convolutions or residual units, and then add the enhanced residual to the input data to obtain the improved motion compensation value. However, these existing enhancement schemes usually have high encoding and decoding complexity, the enhancement effect is only applicable to some square blocks, the data generation and training process is complicated (data needs to be generated and trained separately for different QPs and block sizes), and it takes up a lot of memory when integrated into the encoder.

[0006] Compared to inter-frame prediction enhancement, there is less research on intra-frame prediction enhancement in the existing technology. Existing intra-frame prediction enhancement solutions usually rely on neural networks for enhancement, which also has technical challenges such as high complexity and complex training process. Summary of the Invention

[0007] To overcome the high complexity and training process of existing prediction enhancement methods, the present invention aims to provide a video encoding and decoding method and apparatus with a prediction enhancement model. A video prediction enhancement method incorporating spatial information is designed within the encoding and decoding method. This method leverages available neighbor information and the spatial correlation of the current block to construct a prediction enhancement model, enhancing the predicted value and generating a refined prediction value, thereby improving video encoding and decoding efficiency and video quality.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A video decoding method based on prediction enhancement, characterized by comprising:

[0010] Decode the code stream generated by the video encoder to obtain the prediction information and flag of the current block and the residual value of the current block;

[0011] Predicting the current block based on the prediction information of the current block to generate a prediction value;

[0012] If the flag indicates that the prediction enhancement module is turned on, the prediction value is enhanced using the available spatial neighbors of the current block;

[0013] The residual value of the current block is added to the predicted value to reconstruct the current block.

[0014] A further technical solution of the present invention is that the step of enhancing the predicted value based on the available spatial neighbors of the current block comprises:

[0015] Get the reference area of ​​the current block;

[0016] Obtain the predicted value and reconstructed value of the reference area of ​​the current block;

[0017] Build a model of predicted and reconstructed values ​​for the reference area of ​​the current block;

[0018] The model of the predicted value and the reconstructed value of the reference area of ​​the current block is applied to the predicted value of the current block to obtain an enhanced predicted value of the current block.

[0019] A further technical solution of the present invention: the step of obtaining a reference area of ​​the current block includes:

[0020] Determine whether the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are available; if not all are available, no reference area is constructed, that is, no prediction enhancement is performed on the block; if the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are all available, the left side, upper left and upper side of the current prediction block are the reference areas; if the size of the current block is W*H, where W is the width of the current block and H is the height of the current block, the constructed reference area should contain W+H+1 pixels.

[0021] A further technical solution of the present invention is as follows: the step of constructing a model of predicted values ​​and reconstructed values ​​of a reference area of ​​a current block includes:

[0022] Model the predicted and reconstructed values ​​for the reference region:

[0023] L C (x,y)=α×L P (x,y)+β

[0024] Among them, L C and L P are the reconstructed value and predicted value of the current prediction block boundary, respectively. α and β are two model parameters.

[0025] The model parameters are obtained by solving the predicted values ​​and the reconstructed values ​​of the current reference area.

[0026] A further technical solution of the present invention is to apply a model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block, including:

[0027] When the model of the reference area prediction value and the reconstructed value is applied to the prediction enhancement, P is the prediction value of the current block and P' is the prediction value of the current block after enhancement:

[0028] P′[x,y]=α×P[x,y]+β.

[0029] A further technical solution of the present invention is as follows: the prediction enhancement module utilizes the available spatial neighbors of the current block to enhance the prediction value, and also includes clamping the enhanced prediction value of the current block, limiting it to a normal range using ClipPel, and returning the specified minimum brightness value if it is less than the specified minimum brightness value; and returning the specified maximum brightness value if it is greater than the specified maximum brightness value.

[0030] A further technical solution of the present invention is as follows: selecting the final prediction value and setting a flag indicating whether prediction enhancement is enabled based on the prediction values ​​before and after enhancement, including:

[0031] Based on the prediction values ​​before and after enhancement, the sum of the absolute errors (SAD) before and after enhancement is obtained, and the SAD values ​​before and after enhancement are compared. If the SAD value after enhancement is smaller, it indicates that the enhancement is effective, that is, the original prediction value is replaced with the enhanced value, and the prediction enhancement is turned on. If the SAD value after enhancement is larger, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value, and the prediction enhancement is turned off.

[0032] A video encoding method based on prediction enhancement, characterized in that the video encoder:

[0033] After the intra-frame predictor obtains the prediction value of the current block, the prediction value is input into the prediction enhancement module for enhancement; based on the prediction values ​​before and after enhancement, the final prediction value is selected and the flag indicating whether the prediction enhancement is turned on is set; the final prediction value is subtracted from the original signal to obtain the residual value, which is then transformed and quantized before entropy coding.

[0034] A video decoding device based on prediction enhancement, characterized by comprising

[0035] An entropy decoder configured to decode the bitstream generated by the video encoder to obtain prediction information and a flag of the current block and a residual value of the current block;

[0036] an intra-frame predictor configured to predict the current block based on the prediction information of the current block and generate a prediction value;

[0037] A prediction enhancement module is configured to enhance the prediction value using the available spatial neighbors of the current block if the flag indicates that the prediction enhancement module is turned on;

[0038] An adder is configured to add the residual value of the current block to the prediction value to reconstruct the current block.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention proposes a video encoding and decoding method and apparatus based on predictive enhancement. This method designs a novel predictive enhancement module that incorporates spatial information. This predictive enhancement module combines spatial correlation to construct a model of predicted and reconstructed values ​​for the reference area of ​​the current block. This model is then applied to the predicted value of the current block to obtain a more accurate prediction value, thereby improving encoding and decoding performance. This method differs from existing neural network predictive enhancement methods in that it can be implemented using a lower-complexity approach. The input of the present invention is not restricted in size or shape, including non-square areas, making it more flexible to use. This is a significant difference from predictive enhancement using neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The present invention is a video encoder framework that can implement the technology of the present invention.

[0042] Figure 2 are the current block and the reference block.

[0043] Figure 3 is the intra prediction mode and prediction direction.

[0044] Figure 4 It is the inter-frame prediction process and related techniques.

[0045] Figure 5 The present invention is a video decoder framework that can implement the technology of the present invention.

[0046] Figure 6 Encoder framework with predictive enhancement.

[0047] Figure 7 Decoder framework with prediction enhancement.

[0048] Figure 8 Design an overall idea for the prediction enhancement module.

[0049] Figure 9 Flowchart of the prediction enhancement module.

[0050] Figure 10 A framework for a realizable encoder with inter-frame prediction enhancement.

[0051] Figure 11 A framework for a realizable decoder with inter-frame prediction enhancement. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0053] like Figure 1 The figure shows the framework of a video encoding device that can implement the technology of the present invention. The framework of the video encoding device includes: an image segmenter, a predictor, a subtractor, a transformer, a quantizer, an entropy encoder, an inverse quantizer, an inverse transformer, an adder, a loop filter unit and a memory.

[0054] Each component of the encoder can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented as the function of executing the software corresponding to each component.

[0055] A video sequence is presented as a series of continuous video frames, which can be single or in parallel. During the video encoding process, each video frame to be encoded is divided into several regions, each of which is encoded separately. Specifically, a video frame may be divided into several independent tiles (Tiles) and / or slices (Slices) by an image splitter. These tiles or slices can be further combined to form tile groups or slice groups for more efficient encoding. During the encoding process, each tile or / and slice is subdivided into one or more Coding Tree Units (CTUs). These CTUs can be further divided into smaller Coding Trees (CUs) in a tree structure. The CU is the basic unit in the encoding and decoding process. Each CU contains specific coding information, which constitutes the CU's syntax structure. At the same time, the relevant information of all CUs belonging to the same CTU is organized into the CTU's syntax structure. Information that is common to all blocks in a slice is encoded as the syntax of the slice header, while information related to all blocks that make up one or more video frames is encoded as a picture parameter set (PPS) or a picture header. Shared information that spans multiple video frames is encoded into a sequence parameter set (SPS). Furthermore, if there is shared information between multiple SPSs, this information is encoded into a video parameter set (VPS). In addition, shared information applicable to a tile or tile group can also be encoded as the syntax of the tile or tile group header. The syntax contained in the SPS, PPS, slice header, and tile or tile group header can be collectively referred to as high-level syntax, which is an indispensable component of the video encoding process. These high-level syntaxes ensure the accuracy of video data during the encoding and decoding process.

[0056] The image splitter determines the size of the CTU. Information about the size of the CTU is encoded as the syntax of the SPS or PPS and is transmitted to the decoder. The image splitter splits each frame of the video into CTUs of a predetermined size, and then recursively splits the CTUs in a tree structure. The leaf nodes in the tree structure are called CUs. In the following, the block corresponding to the CU to be encoded or decoded is called the "current block", and the spatially adjacent pixel blocks in the current block are called reference blocks, such as Figure 2 shown.

[0057] The predictor is responsible for predicting the current block, thereby generating a predicted block. Predictors include intra-frame predictors and inter-frame predictors. Typically, the current block in an image can be coded using predictive coding. Specifically, prediction of the current block can be achieved using intra-frame prediction techniques (using data from the video frame to which the current block belongs) or inter-frame prediction techniques (using data from previously encoded video frames).

[0058] The intra-frame predictor uses the coded pixel values ​​in the current video frame to predict the pixel values ​​to be coded, thereby effectively removing the spatial correlation of the video. Intra-frame prediction can adopt multiple modes according to different prediction directions. For example, Figure 3 As shown by the solid line in the middle, intra prediction modes include two non-directional modes, namely planar mode and DC mode, and 65 different directional modes. Each prediction mode has its own unique neighboring pixel selection and algorithm formula.

[0059] In order to achieve more efficient directional prediction on a current block with a rectangular shape, additional use can be made of Figure 3 The dotted arrows indicate the directional modes (67 to 80 and -1 to -14). These modes are called "Wide Angle Intra Prediction (WAIP)". Figure 3 In the figure, the arrows indicate the positions of the reference samples used for prediction, but not the actual direction of the prediction. The prediction direction is opposite to the direction indicated by the arrows.

[0060] When the current block is rectangular, the wide-angle intra prediction mode can predict in the opposite direction of the specific direction mode without increasing the transmission bit. In this case, the selection of available wide-angle intra prediction modes can be determined by the aspect ratio of the current block. For example, when the height of the current block is smaller than the width, a wide-angle intra prediction mode with an angle less than 45 degrees (intra prediction modes #67 to #80) can be used. When the width of the current block is larger than the height, a wide-angle intra prediction mode with an angle greater than -135 degrees can be used.

[0061] To compensate for the shortcomings of angular prediction in utilizing spatial information, the intra-frame predictor can use Position Dependent Prediction Combination (PDPC) to correct the initial prediction values ​​obtained by intra-frame prediction. PDPC first calculates the initial values ​​for each prediction mode and then performs a weighted combination of prediction values ​​from different directions (such as horizontal and vertical) based on the position of the pixels within the block to obtain the corrected prediction values.

[0062] In addition to angular prediction, the intra predictor can also use matrix-based intra prediction technology (MIP), which obtains an output vector by multiplying the downsampled reference pixel block with a fixed weight matrix, and upsamples the output vector to the current block size to obtain the final prediction value.

[0063] The intra-frame predictor determines the intra-frame prediction mode to use to encode the current block. In some cases, the intra-frame predictor encodes the current block using multiple intra-frame prediction modes and selects the most suitable intra-frame prediction mode from these test modes. For example, the intra-frame predictor can perform rate-distortion analysis on multiple test intra-frame prediction modes, calculate the rate-distortion value for each mode, and then select the intra-frame prediction mode with the best rate-distortion characteristics.

[0064] Once the intra-frame predictor determines the optimal mode among multiple intra-frame prediction modes, it uses the neighboring pixels (reference pixels) and algorithm equations defined in that mode to predict the current block. The entropy encoder then encodes information about the selected intra-frame prediction mode and transmits it to the video decoding device so that the decoder can correctly decode and reconstruct the block.

[0065] The inter-frame predictor exploits temporal correlation in video, using pixel values ​​from adjacent coded frames to predict pixel values ​​in the current frame to be encoded, thereby removing temporal redundancy. The inter-frame predictor uses motion estimation (ME) to obtain a best-matching block in the coded image. It then calculates the displacement from this best-matching block to the current block to obtain a motion vector (MV). This MV is then used to perform motion compensation (MC) to obtain a predicted value for the current block. The pixel difference between the current block and the reference block is called the prediction residual. The inter-frame encoder encodes only the inter-image motion information and the prediction residual, significantly improving coding efficiency.

[0066] According to the encoding method of motion vector and prediction residual, inter-frame coding mode can be divided into three categories: Skip, Merge and Advanced Motion Vector Prediction (AMVP). Specifically, Skip mode uses predicted motion vector to represent MV information, and does not encode motion vector difference and prediction residual; Merge mode only encodes predicted motion vector and prediction residual, but not motion vector difference; AMVP mode encodes both predicted motion vector difference and residual.

[0067] The inter-frame prediction process can be divided into three steps: motion vector prediction, motion vector determination, and motion compensation. The specific process and related technologies are as follows: Figure 4 shown.

[0068] The subtractor generates a residual block by subtracting a prediction block generated by an intra predictor or an inter predictor from the current block.

[0069] The transformer transforms the residual signal in the residual block with spatial pixel values ​​into transform coefficients in the frequency domain. Through appropriate transformation, the dispersed distribution of image energy in the spatial domain can be converted into a relatively concentrated distribution in the transform domain, thereby achieving the purpose of removing redundancy. The transformation process can be performed by using the entire residual block as the transform unit, or by dividing the residual block into multiple sub-blocks and performing the transform operation on these sub-blocks as transform units. For sub-block transforms, the entropy encoder encodes a flag indicating that only the sub-block is being transformed, as well as the corresponding direction and position information, and notifies the video decoder of this information.

[0070] The converter can also independently transform the residual block in the horizontal and vertical directions. Various transform functions or matrices can be used in this transformation. For example, pairs of transform functions for horizontal and vertical transformations can be defined as a Multiple Transform Selection (MTS). The converter can select the transform function pair with the highest transform efficiency in the MTS and transform the residual block in the horizontal and vertical directions. The entropy encoder encodes information about the transform function pairs in the MTS and sends this information to the video decoder via the bitstream.

[0071] After the residual signal undergoes transformation, the transform coefficients often have a larger dynamic range. The quantizer is responsible for quantizing the transform coefficients output by the transformer, reducing the signal value space, thereby achieving better compression effect.

[0072] The quantizer processes the transform coefficients based on the quantization parameters and sends the quantized coefficients to the entropy encoder. The quantizer can use the correlation between the transform coefficients to perform dependent quantization, making the transform coefficients more compact in the M-dimensional vector space after quantization.

[0073] The entropy encoder encodes the quantized transform coefficient sequence using various coding schemes, including variable-length coding, Exponential Golomb coding, and Context-Based Adaptive Binary Arithmetic Coding (CABAC), to generate an encoded bitstream. Furthermore, the entropy encoder encodes information related to block segmentation, including CTU size, CTU segmentation flags, and the tree-structured partitioning of CTUs, to ensure that the video decoding device can perform block segmentation consistently with the video encoding device. Furthermore, the entropy encoder encodes prediction type information, indicating whether the current block is coded using intra-frame prediction or inter-frame prediction. Depending on the selected prediction type, the entropy encoder further encodes intra-frame prediction information (such as the intra-frame prediction mode) or inter-frame prediction information (such as the merge index in merge mode and the reference image index and motion vector difference in AMVP mode). Finally, the entropy encoder encodes information related to quantization, including the quantization parameter and quantization matrix.

[0074] The dequantizer dequantizes the quantized transform coefficients received from the quantizer to generate corresponding transform coefficients.

[0075] The inverse transformer converts these transform coefficients from the frequency domain back to the spatial domain to reconstruct the residual block.

[0076] The adder adds the reconstructed residual block to the prediction block generated by the predictor to reconstruct the current block and obtain the reconstructed block. When performing intra-frame prediction for the next block, the reconstructed block will be used as the reference block for the current block. The position relationship between the reference block and the current block is as follows: Figure 2 shown.

[0077] A loop filter is a filter used during the video encoding process to reduce common coding artifacts such as blocking artifacts, ringing artifacts, color deviation, and image blur that occur in block-based hybrid coding methods. The filtered image is used to encode subsequent images, effectively forming the coding "loop." The loop filter improves the quality of the encoded image while providing a high-quality reference image for subsequent encoding, resulting in better prediction and increased coding efficiency. The loop filter is typically composed of one or more of three modules: a deblocking filter (DBF), sample adaptive offset (SAO), and an adaptive loop filter (ALF).

[0078] The DBF filter smoothes the boundaries of reconstructed blocks to reduce discontinuities at block coding boundaries, thereby removing blocking artifacts. In block-based transforms and quantization, strong edges in an image can produce ripples around them after decoding due to quantization distortion of high-frequency AC coefficients. This distortion, known as ringing, severely impacts the subjective and objective quality of the video. The SAO filter uses a CTU as a unit, classifying reconstructed pixels into categories by selecting an appropriate classifier and applying boundary or sideband compensation to pixels in different categories to reduce ringing. The ALF filter, based on the Wiener filtering principle, uses the original and reconstructed image information to establish the Wiener-Hoff equation and solve a series of filter coefficients with minimum mean square error, thereby reducing decoding errors and effectively improving video frame quality. The filter coefficients are written into the bitstream via an entropy encoder.

[0079] The memory is used to store the reconstructed blocks after loop filtering. When all blocks in a video frame are reconstructed, the reconstructed entire frame image can be used as a reference image for inter-frame prediction of the current block in the subsequent encoded image.

[0080] like Figure 5 FIG. 1 shows a framework of a video decoding device that can implement the technology of the present invention. The framework of the video decoding device may include: an entropy decoder, an inverse quantizer, an inverse transformer, a predictor, an adder, a loop filter, and a memory.

[0081] Similar to Figure 1 Each component of the video encoding device and the video decoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0082] The entropy decoder decodes the bitstream generated by the video encoder and extracts information related to block segmentation to determine the current block to be decoded and obtain prediction information and residual information used to reconstruct the current block. The entropy decoder extracts CTU size information from the SPS or PPS to determine the size of the CTU and divides the video frame into CTUs of predetermined size based on this information. In addition, the CTU is set as the root node of the tree structure, and the entropy decoder extracts information related to CTU segmentation and obtains CUs by segmenting the CTU using the tree structure.

[0083] When determining the decoding for the current block, the entropy decoder uses the tree structure's segmentation information to extract prediction type information, indicating whether the current block is predicted using intra or inter prediction. For intra prediction, the decoder obtains syntax elements such as the intra prediction mode for the current block; for inter prediction, the decoder obtains relevant syntax elements including motion vectors, reference images, and prediction residuals. Furthermore, the entropy decoder extracts information related to quantization and residual signal information associated with the quantized transform coefficients of the current block.

[0084] The inverse quantizer inversely quantizes the transform coefficients quantized by the quantization parameter QP.

[0085] The inverse transformer performs inter-domain conversion on the transform coefficients generated by the inverse quantizer to restore the reconstructed residual information, thereby obtaining the residual block of the current block.

[0086] The predictor may include an intra predictor and an inter predictor. When the prediction type of the current block is intra prediction, the intra predictor is activated for use, and when the prediction type of the current block is inter prediction, the inter predictor is activated for use.

[0087] The intra predictor selects an intra prediction mode for the current block from multiple intra prediction modes according to the intra prediction mode syntax element extracted from the entropy decoder, and predicts the current block using adjacent reference pixels of the current block to obtain a predicted block.

[0088] The inter-frame predictor determines the motion vector of the current block and the reference image referred to by the motion vector by using the syntax elements of the inter-frame prediction mode extracted from the entropy decoder, and predicts the current block using the reference block in the reference image to obtain a predicted block.

[0089] The adder reconstructs the current block by adding the residual block output from the inverse transformer to the prediction block output from the inter-frame predictor or the intra-frame predictor. When performing intra-frame prediction on a subsequent block to be decoded, the pixels in the reconstructed current block can be used as reference pixels in a reference block.

[0090] The loop filter unit of the loop filter may include one or a combination of a DBF filter, an SAO filter, and an ALF filter. The DBF filter performs smoothing filtering on the boundaries between reconstructed blocks to remove discontinuous blocking effects caused by block unit decoding. The SAO filter and ALF perform additional filtering on the reconstructed blocks after DBF filtering to compensate for the difference between the reconstructed pixels and the original pixels caused by lossy coding, thereby improving the quality of the video frame. The filter coefficients of the filter are obtained from the decoded bitstream of the entropy decoder.

[0091] After loop filtering, the reconstructed blocks are stored in memory. Once all blocks in an image have been reconstructed, the entire reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent video frames that need to be decoded.

[0092] In some embodiments, the present invention relates to a video encoding and decoding apparatus as described above. More specifically:

[0093] The present invention first proposes an intra-frame luminance prediction enhancement module for video codecs. That is, in the intra-frame prediction encoder, after the original encoder obtains the intra-frame prediction value, the intra-frame prediction enhancement module proposed by the present invention is added to refine the prediction value. The sum of absolute differences (SAD) is used as an indicator to determine whether the prediction enhancement module is turned on, and the corresponding flag is written into the bitstream through the entropy encoder to reduce the residual error and improve the coding performance. The encoder framework with prediction enhancement is as follows: Figure 6 As shown in . In the original decoder, after the intra-frame predictor performs intra-frame prediction to obtain the intra-frame prediction value, the flag information indicating whether the intra-frame prediction enhancement module is enabled is obtained from the entropy decoder. If enabled, the prediction enhancement is completed and then added to the corresponding residual value to obtain the reconstructed value; if not enabled, no prediction enhancement is required. The decoder framework with prediction enhancement is shown in Figure 7 shown.

[0094] (Note: 1. The video encoder framework is extensible, see Example 2; 2. Intra-frame luminance prediction is extensible, see Examples 3 and 4; 3. Indication of whether the enhancement module is enabled is extensible, see Example 5)

[0095] The present invention proposes an intra-frame prediction enhancement module, which makes full use of the available spatial neighbor information for reconstruction around the current block, and uses the relationship between the predicted value and the reconstructed value of the available spatial neighbors to enhance the predicted value of the current block to reduce the prediction error. The idea and implementation of this module are the core protection points of the present invention. The implementation of this module consists of four steps. Step 1: Obtain the reference area of ​​the current block (Note: the reference area can be expanded, see Example 7). Step 2: Obtain the predicted value and reconstructed value of the reference area of ​​the current block. Step 3: Construct a model of the predicted value and reconstructed value of the reference area of ​​the current block (Note: the model construction can be expanded, see Example 6 and Example 8). Step 4: Apply the model in step 3 to the predicted value of the current block, thereby obtaining an enhanced predicted value of the current block. Steps 3 and 4 deeply embody the core idea of ​​the present invention.

[0096] The following embodiments are generally applicable to video codecs and prediction enhancement modules using video codecs.

[0097] Based on the enhancement method implemented by the prediction enhancement module, the present invention proposes a video decoding method based on prediction enhancement, including: decoding the code stream generated by the video encoder, obtaining the prediction information and flag of the current block, and the residual value of the current block; predicting the current block based on the prediction information of the current block to generate a prediction value; if the flag shows that the prediction enhancement module is turned on, using the available spatial neighbors of the current block to enhance the prediction value; and adding the residual value of the current block to the prediction value to reconstruct the current block.

[0098] Based on the prediction enhancement module, the present invention proposes a video decoding device based on prediction enhancement, including an entropy decoder, which is configured to decode the code stream generated by the video encoder, obtain the prediction information and flag of the current block, and the residual value of the current block; an intra-frame predictor, which is configured to predict the current block based on the prediction information of the current block and generate a prediction value; a prediction enhancement module, which is configured to enhance the prediction value using the available spatial neighbors of the current block if the flag shows that the prediction enhancement module is turned on; and an adder, which is configured to add the residual value of the current block to the prediction value to reconstruct the current block.

[0099] Example 1:

[0100] Based on H.266 / VVC, this paper proposes a new intra-frame prediction enhancement module that combines spatial information to improve coding performance. A video encoding and decoding method based on this intra-frame prediction enhancement module is implemented. The following method is used as an example to illustrate the implementation process of this technical solution.

[0101] In a video encoder, when intra-frame prediction is selected, after the intra-frame predictor obtains the prediction value for the current block, it is input into the prediction enhancement module for prediction enhancement. The effectiveness of the enhancement is determined by the SAD values ​​before and after enhancement. If the SAD value after enhancement is smaller, the enhancement is effective, and the original prediction value is replaced with the enhanced value. If the SAD value after enhancement is larger, the enhancement is invalid, and the original prediction value becomes the final prediction value. Whether prediction enhancement is enabled is sent to the entropy encoder as a flag and written into the bitstream. The final prediction value is subtracted from the original signal to obtain the residual. After transformation and quantization, it is entropy encoded using the entropy encoder. In the decoder, the entropy decoder decodes the bitstream to obtain information about the current block and its residual value. This information about the current block includes prediction-related information and a prediction enhancement flag. The intra-frame prediction value is calculated by predicting the relevant information. If the flag shows that the prediction enhancement module is turned on, that is, after the prediction enhancement is performed, the enhanced prediction value is added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value; if the flag shows that the prediction enhancement module is not turned on, that is, after the prediction value is calculated, it is directly added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value. Figure 6 and Figure 7The red part in the middle shows the location of the intra-frame prediction enhancement scheme in the codec under the H.266 / VVC framework. The prediction enhancement module is added by this technical solution.

[0102] The following is a detailed description of the prediction enhancement module. After the encoder predicts the current block, the prediction value is enhanced using the available spatial neighbors of the current block. Combined with the inter-block correlation, it is assumed that the relationship and regularity between the predicted value and the reconstructed value of the available spatial neighbors and the current block are the same. Taking the L-shaped reference area as an example, the overall design idea of ​​the prediction enhancement module is as follows: Figure 8 Where LC and LP represent the reconstructed value and predicted value of the reference region, respectively; P and P' represent the predicted value and enhanced predicted value of the current block, respectively.

[0103] like Figure 9 As shown in Figure 2, the implementation process of the prediction enhancement module is as follows:

[0104] Step 1: Get the reference area of ​​the current block.

[0105] Step 2: Get the predicted value and reconstructed value of the reference area of ​​the current block.

[0106] Step 3: Build a model of the predicted and reconstructed values ​​of the reference area of ​​the current block.

[0107] Step 4: Apply the model in step 3 to the predicted value of the current block to obtain an enhanced predicted value of the current block.

[0108] The specific process of obtaining the reference area of ​​the current block in step 1 is as follows:

[0109] Analyze whether the current block boundary is available. In this embodiment, determine whether the predicted and reconstructed pixels of the upper left, upper side and left side boundaries (1 row / 1 column) of the current prediction block are available. If not all are available, no reference area is constructed, that is, the block is not predicted and enhanced. If the predicted and reconstructed pixels of the upper left, upper side and left side boundaries (1 row / 1 column) of the current prediction block are all available, then the left, upper left and upper side (1 row / 1 column) of the current prediction block are the reference areas. If the size of the current block is W*H, W is the width of the current block, and H is the height of the current block, the constructed reference area should contain (W+H+1) pixels. In general, only the prediction blocks that contain valid reference areas are enhanced. Apart from this, there are no restrictions on the size, shape and position of the prediction blocks. The reference areas are as follows Figure 2 As shown, it consists of 1 row / column of luminance samples above and to the left of the current block.

[0110] Step 3: The process of building a model of the predicted value and reconstructed value of the reference area of ​​the current block is as follows:

[0111] Step 1: Build a model for the predicted and reconstructed values ​​of the reference region. For each pixel position in the constructed reference region, there is a corresponding predicted and reconstructed value. In this invention, a model is used to describe and build the predicted and reconstructed values ​​for the same pixel position in the reference region. The construction process is now explained using a linear model as an example.

[0112] L C (x,y)=α×L P (x,y)+β

[0113] Among them, L C and L P are the reconstructed value and predicted value of the current prediction block boundary respectively. α and β are two model parameters.

[0114] Step 2: Use the least squares method to solve the model parameters and find the best function matching of the data by minimizing the sum of squares of the errors to obtain a curve that best represents the relationship between the predicted value and the reconstructed value of the reference area of ​​the current block. First, calculate the mean of the predicted value and the reconstructed value in the reference area. and Among them, n represents the total number of n pixels in the reference area. The current n should be W+H+1, L P (x i ,y i ) and L C (x i ,y i ) respectively represent (x i ,y i ) The predicted and reconstructed values ​​at the position. The slope α of the linear model is calculated as follows:

[0115]

[0116] The corresponding intercept is calculated as follows:

[0117]

[0118] The process of obtaining an enhanced prediction value of the current block in step 4 is as follows:

[0119] Step 1: When applying the above linear model to prediction enhancement, P is the predicted value of the current block, and P' is the predicted value of the current block after enhancement.

[0120] P′[x,y]=α×P[x,y]+β

[0121] Step 2: Since the enhanced prediction value obtained by the linear model may not be within the brightness range (for 8-bit video, the normal range is 0-255; for 10-bit video, the normal range is 0-1023), it is necessary to clamp the enhanced prediction value of the current block. Use ClipPel to limit it to the normal range. If it is less than the specified minimum brightness value, the specified minimum brightness value is returned; if it is greater than the specified maximum brightness value, the specified maximum brightness value is returned.

[0122] Example 2:

[0123] The difference from Example 1 is that Example 1 relies on the coding framework of VVC, and Example 2 can implement this method on other coding frameworks, such as AV1, NNVC, ECM, AVS3, etc., but is not limited to the above-mentioned coding frameworks.

[0124] Example 3:

[0125] The difference between Example 3 and Example 1-2 is that Example 1-2 implements intra-frame brightness prediction enhancement, while Example 3 can be implemented in different prediction links, such as migrating the method of the present invention to inter-frame prediction enhancement. In the encoder, after the prediction value of the current block is obtained by inter-frame prediction, the prediction value is input into the prediction enhancement module for enhancement. The SAD value before and after enhancement is used to determine whether the enhancement is effective. If the SAD value after enhancement is small, it indicates that the enhancement is effective, that is, the original prediction value is replaced by the enhanced value; if the SAD value after enhancement is large, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value. Whether to turn on prediction enhancement is written into the bitstream in the form of a flag through the entropy encoder. The final prediction value is subtracted from the original signal to obtain a residual, which is then transformed and quantized and then entropy encoded. In the decoder, the information of the current block and the residual value of the current block are decoded from the bitstream through the entropy decoder. The information of the current block includes: prediction-related information, prediction enhancement flag, etc. The inter-frame prediction value is calculated using the prediction information. If the flag shows that the prediction enhancement module is turned on, that is, after the prediction enhancement is performed, the enhanced prediction value is added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value; if the flag shows that the prediction enhancement module is not turned on, that is, after the prediction value is calculated, the reconstructed value is directly added to the residual after inverse quantization and inverse transformation.

[0126] Figure 10 and 11 The position of the inter-frame prediction enhancement scheme in the codec within the framework of the codec that can implement the technology of the present invention is given. The red prediction enhancement module in the figure is added by this technical solution.

[0127] Example 4:

[0128] The difference between Example 4 and Examples 1-3 is that, whereas Examples 1-3 performed prediction enhancement on luma, this example can also perform prediction enhancement on chroma. When the method of the present invention is used for both luma and chroma, a more accurate prediction block can be obtained by adding control variables and making a decision to select whether to use the prediction enhancement method for luma or chroma.

[0129] Example 5:

[0130] The difference between Example 5 and Examples 1-4 is that in Examples 1-4, the encoder determines whether the enhancement is effective by using the SAD values ​​before and after enhancement, and writes a corresponding flag into the bitstream. On the decoder, after entropy decoding, the flag indicates whether the prediction enhancement module is enabled. This part can be expanded as follows:

[0131] All prediction blocks containing valid reference areas are enhanced without the need to pass a flag. That is, on the encoder side, enhancement is performed directly after obtaining the intra-frame prediction value. On the decoder side, enhancement is performed directly after obtaining the intra-frame prediction value.

[0132] Enhance all prediction blocks containing valid reference regions, without the need for a flag. The relationship between the current block's information (such as prediction mode and reference samples) and whether enhancement is effective is analyzed. On the encoder side, enhancement is determined based on the current block's information. On the decoder side, enhancement is also determined directly based on the current block's information, without the need for a flag.

[0133] 3. The intra-frame prediction enhancement module is introduced as a new mode, competing with other prediction modes. On the encoder side, prediction enhancement competes with other prediction modes, comparing the rate-distortion cost with other modes and setting a flag indicating whether the enhancement module is enabled. On the decoder side, if the enhancement flag is enabled, the predicted value is enhanced before the reconstructed value is calculated.

[0134] 4. The indicator for determining whether enhancement is effective can be replaced with RDCost (Rate Distortion Cost). If the RDCost value after enhancement is smaller, it indicates that enhancement is effective, and the predicted value before enhancement is replaced by the predicted value after enhancement. If the RDCost value after enhancement becomes larger, it indicates that enhancement is ineffective, and the predicted value before enhancement is used as the final predicted value. Whether to enable predictive enhancement is written into the bitstream in the form of a flag. The final predicted value is subtracted from the original signal to obtain the residual, which is then transformed and quantized and entropy encoded. On the decoding end, after entropy decoding, the flag indicates whether the predictive enhancement module is enabled.

[0135] Example 6:

[0136] The difference between Example 6 and Examples 1-5 is that, whereas Step 3 of Examples 1-5 utilizes a linear model to construct a model for the predicted and reconstructed values ​​of the reference region, this embodiment can also utilize other models, such as convolutional models and MMLM models, to describe the relationship between the predicted and reconstructed values ​​of the reference region, but is not limited to the aforementioned models. Furthermore, the embodiment can adaptively select and construct a model for prediction enhancement based on different intra-frame prediction modes or other information about the current block, such as prediction information of the current block.

[0137] Example 7:

[0138] The difference between Example 7 and Examples 1-6 is that the reference area constructed in step 1 of Examples 1-6 is the boundary of the upper left, upper side, and left side of the current block by one row / one column. In this embodiment, the reference area can be constructed as N rows and N columns (N>1), or the reference area of ​​the current block can be constructed with M rows and M columns (M≥1) at the lower left, left, upper left, upper side, and upper right, or a reference area of ​​any other shape can be constructed.

[0139] Example 8:

[0140] The difference between Example 8 and the above-mentioned Examples 1-7 is that, in step 3 of the above-mentioned Examples 1-7, all pixel values ​​are used to calculate model parameters when constructing the model. This embodiment can also select some pixels to calculate model parameters to further reduce complexity.

[0141] In summary, the present invention makes better use of the spatial correlation of video frames, reducing prediction error and improving encoding performance. BD-rate is an objective metric used in video encoding. Within a certain bitrate or quality range, this metric can compare the rate-distortion performance of two different video codecs, or different configurations of the same video codec. A negative value indicates improved compression efficiency.

[0142] The proposed method was implemented on the VVC standard test code VTM-20.0 platform, and the proposed method achieved intra-frame luminance prediction enhancement for blocks with valid reference regions. Under AI configuration, the configuration file encoder_intra_vtm.cfg was selected, the number of encoded frames was 8 (with TemporalSubsampleRatio set to 1), and the standard test sequence was encoded and decoded with quantization parameters (QP) of 22, 27, 32, and 37. The bitrate and peak signal-to-noise ratio (PSNR) of the video encoding were recorded. The encoded objects were standard VVC test videos with the following names and resolutions: BasketballPass (416×240), BQSquare (416×240), BlowingBubbles (416×240), and RaceHorses (416×240). On average, the bitrates of the Y, U, and V components were reduced by 0.40%, 0.77%, and 0.74%, respectively, compared to VVC, verifying the effectiveness of the proposed method.

[0143] Table 1 Coding performance of the present invention compared with VTM20.0 under AI configuration

[0144]

[0145] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and that in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A video decoding method based on prediction enhancement, characterized in that: include: Decode the code stream generated by the video encoder to obtain the prediction information and flag of the current block and the residual value of the current block; Predicting the current block based on the prediction information of the current block to generate a prediction value; If the flag indicates that the prediction enhancement module is turned on, the prediction value is enhanced using the available spatial neighbors of the current block; Reconstruct the current block by adding the residual value of the current block to the predicted value; The enhancing the predicted value based on the available spatial neighbors of the current block includes: Get the reference area of ​​the current block; Obtain the predicted value and reconstructed value of the reference area of ​​the current block; Build a model of predicted and reconstructed values ​​for the reference area of ​​the current block; Apply the model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block; The constructing of a model of predicted values ​​and reconstructed values ​​of a reference area of ​​the current block includes: Model the predicted and reconstructed values ​​for the reference region: in, LC and LP are the reconstructed value and predicted value of the current prediction block boundary, and are two model parameters; The model parameters are obtained by solving the predicted values ​​and the reconstructed values ​​of the current reference area.

2. The video decoding method based on prediction enhancement according to claim 1, characterized in that The obtaining of the reference area of ​​the current block includes: Determine whether the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are available; if not all are available, no reference area is constructed, that is, no prediction enhancement is performed on the block; if the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are all available, the left side, upper left and upper side of the current prediction block are the reference areas; if the size of the current block is W*H, where W is the width of the current block and H is the height of the current block, the constructed reference area should contain W+H+1 pixels.

3. The video decoding method based on prediction enhancement according to claim 1, characterized in that Apply the model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block, including: When the model of the predicted and reconstructed values ​​of the reference area is applied to the prediction enhancement, P is the predicted value of the current block, P’ is the predicted value of the current block after enhancement: 。 4. The video decoding method based on prediction enhancement according to claim 1, characterized in that The prediction enhancement module uses the available spatial neighbors of the current block to enhance the prediction value, and also includes clamping the enhanced prediction value of the current block, limiting it to a normal range using ClipPel. If it is less than the specified minimum brightness value, the specified minimum brightness value is returned; if it is greater than the specified maximum brightness value, the specified maximum brightness value is returned.

5. The video decoding method based on prediction enhancement according to claim 1, characterized in that: The process of selecting a final prediction value and setting a flag indicating whether prediction enhancement is enabled based on the prediction values ​​before and after enhancement includes: Based on the prediction values ​​before and after enhancement, the sum of the absolute errors (SAD) before and after enhancement is obtained, and the SAD values ​​before and after enhancement are compared. If the SAD value after enhancement is smaller, it indicates that the enhancement is effective, that is, the original prediction value is replaced with the enhanced value, and the prediction enhancement is turned on. If the SAD value after enhancement is larger, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value, and the prediction enhancement is turned off.

6. A video coding method based on prediction enhancement, characterized in that: The video encoder: After the intra predictor obtains the prediction value of the current block, the prediction value is input into the prediction enhancement module for enhancement; based on the prediction values ​​before and after enhancement, the final prediction value is selected and the flag indicating whether the prediction enhancement is turned on is set; Subtract the final predicted value from the original signal to obtain the residual value, which is then transformed and quantized before being entropy coded. The step of inputting the predicted value into the prediction enhancement module for enhancement includes: Get the reference area of ​​the current block; Obtain the predicted value and reconstructed value of the reference area of ​​the current block; Build a model of predicted and reconstructed values ​​for the reference area of ​​the current block; Apply the model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block; The constructing of a model of predicted values ​​and reconstructed values ​​of a reference area of ​​the current block includes: Model the predicted and reconstructed values ​​for the reference region: in, LC and LP are the reconstructed value and predicted value of the current prediction block boundary, and are two model parameters; The model parameters are obtained by solving the predicted values ​​and the reconstructed values ​​of the current reference area.

7. A video decoding device based on prediction enhancement, characterized in that: include An entropy decoder configured to decode the bitstream generated by the video encoder to obtain prediction information and a flag of the current block and a residual value of the current block; an intra-frame predictor configured to predict the current block based on the prediction information of the current block and generate a prediction value; A prediction enhancement module is configured to enhance the prediction value using the available spatial neighbors of the current block if the flag indicates that the prediction enhancement module is turned on; an adder configured to add a residual value of a current block to a predicted value to reconstruct the current block; The enhancing the predicted value based on the available spatial neighbors of the current block includes: Get the reference area of ​​the current block; Obtain the predicted value and reconstructed value of the reference area of ​​the current block; Build a model of predicted and reconstructed values ​​for the reference area of ​​the current block; Apply the model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block; The constructing of a model of predicted values ​​and reconstructed values ​​of a reference area of ​​the current block includes: Model the predicted and reconstructed values ​​for the reference region: in, LC and LP are the reconstructed value and predicted value of the current prediction block boundary, and are two model parameters; The model parameters are obtained by solving the predicted values ​​and the reconstructed values ​​of the current reference area.

Images