Video coding and decoding method and device based on prediction enhancement

Abstract

The invention provides a video encoding and decoding method and device based on prediction enhancement, and belongs to the technical field of video processing. The video decoding method comprises the following steps: decoding a code stream generated by a video encoder, and obtaining prediction information and flag of a current block and a residual value of the current block; predicting the current block based on the prediction information of the current block to generate a prediction value; if the flag shows that the prediction enhancement module is started, enhancing the prediction value by using the available spatial neighbors of the current block; and reconstructing the current block by adding the residual value of the current block to the predicted value. The video coding method comprises the following steps: after an intra-frame predictor obtains a predicted value of a current block, inputting the predicted value into a prediction enhancement module for enhancement; according to the predicted values before and after enhancement, selecting a final predicted value and setting a flag for predicting whether enhancement is started or not; and subtracting the final predicted value from the original signal to obtain a residual value, and performing entropy coding after transformation and quantization. According to the invention, the coding and decoding performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video prediction enhancement method combining spatial information under a video codec framework. Background Art

[0002] Raw video data has a larger data volume than image or audio data. Uncompressed video data is difficult to store, and it is even more difficult to directly transmit raw video data on the network. Therefore, in order to achieve efficient data management, a video encoder is usually used to encode the raw video data so that the video data can be stored or transmitted efficiently. Subsequently, a decoder is used to receive the encoded bitstream, decode it, and obtain the compressed video through a player.

[0003] Traditional video codecs use a block-based hybrid coding framework, which is usually standardized. The general video coding H.266 / VVC standard, AV1 and AVS3 are the latest video coding standards developed by JVET, AOMedia and China, which are composed of ITU-T VCEG and ISO / IEC MPEG. At this stage, JVET is also focusing on the development of video coding standards in two major directions based on H.266 / VVC: Beyond VVC explores more advanced, non-neural network coding tools; NNVC explores neural network-based coding tools.

[0004] However, with the continuous improvement of the size, resolution and playback frame rate of video images, the amount of data to be encoded is also showing a rapid growth trend. Therefore, a new codec technology is needed that can provide higher encoding and decoding efficiency and improve the video image enhancement effect compared with existing compression technologies. In the traditional intra-frame prediction method, for the current coding unit in the video to be encoded, a prediction unit is usually generated from adjacent reference samples based on a fixed rule. However, this method may have difficulty in generating accurate prediction units when processing complex textures. In addition, since the amount of information that can be used as a reference sample in intra-frame prediction is often lower than the amount of information between temporally adjacent frames, the encoding and decoding performance of the intra-frame prediction method is generally lower than that of the inter-frame prediction method. Therefore, the prediction enhancement method can be used to reduce the residual signal between the original block by improving the generation of the prediction unit.

[0005] At present, the existing prediction enhancement schemes are mainly focused on inter-frame prediction enhancement, which usually enhances motion compensation through neural networks. These enhancement operations are usually performed after the inter-frame coded square blocks complete motion compensation. Different neural networks need to be trained independently for different coding block sizes and different quantization parameters (Quantization Parameter, QP). The prior art usually performs residual enhancement by stacking residual convolutions or residual units, and then adds the enhanced residual to the input data to obtain the improved motion compensation value. However, these existing enhancement schemes usually have high encoding and decoding complexity, the enhancement effect is only applicable to some square blocks, the data generation and training process is complicated (data needs to be generated and trained separately for different QPs and block sizes), and it takes up a lot of memory when integrated into the encoder.

[0006] Compared with inter-frame prediction enhancement, there are fewer studies on intra-frame prediction enhancement in the prior art. Existing intra-frame prediction enhancement solutions usually rely on neural networks for enhancement, so there are also technical difficulties such as high complexity and complex training process. Summary of the invention

[0007] In order to overcome the problems of high complexity and complex training process of existing prediction enhancement methods, the present invention aims to provide a video coding method and device with a prediction enhancement model. In the coding method, a video prediction enhancement method combining spatial information is designed to fully utilize the available neighbor information and the spatial correlation of the current block to build a prediction enhancement model, enhance the prediction value, and generate a refined prediction value to improve the video coding efficiency and video quality.

[0008] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0009] A video decoding method based on prediction enhancement, characterized by comprising:

[0010] Decode the bitstream generated by the video encoder to obtain the prediction information and flag of the current block and the residual value of the current block;

[0011] Predicting the current block based on the prediction information of the current block to generate a prediction value;

[0012] If the flag indicates that the prediction enhancement module is turned on, the prediction value is enhanced using the available spatial neighbors of the current block;

[0013] The residual value of the current block is added to the predicted value to reconstruct the current block.

[0014] A further technical solution of the present invention is as follows: the step of enhancing the predicted value based on the available spatial neighbors of the current block comprises:

[0015] Get the reference area of ​​the current block;

[0016] Obtain the predicted value and the reconstructed value of the reference area of ​​the current block;

[0017] Construct a model of predicted and reconstructed values ​​for the reference area of ​​the current block;

[0018] The model of the predicted value and the reconstructed value of the reference area of ​​the current block is applied to the predicted value of the current block to obtain an enhanced predicted value of the current block.

[0019] A further technical solution of the present invention: the step of obtaining a reference area of ​​the current block includes:

[0020] Determine whether the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are available; if not all are available, no reference area is constructed, that is, no prediction enhancement is performed on the block; if the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are all available, the left side, upper left and upper side of the current prediction block are the reference areas; if the size of the current block is W*H, where W is the width of the current block and H is the height of the current block, the constructed reference area should contain W+H+1 pixels.

[0021] A further technical solution of the present invention is as follows: the model of building the prediction value and the reconstruction value of the reference area of ​​the current block includes:

[0022] Model the predicted and reconstructed values ​​for the reference region:

[0023] L C (x,y)=α×L P (x,y)+β

[0024] Among them, L C and L P are the reconstructed value and predicted value of the boundary of the current prediction block, respectively. α and β are two model parameters;

[0025] The model parameters are obtained by solving the predicted values ​​and the reconstructed values ​​of the current reference area.

[0026] A further technical solution of the present invention is to apply the model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block, including:

[0027] When the model of the reference area prediction value and the reconstruction value is applied to the prediction enhancement, P is the prediction value of the current block, and P' is the prediction value of the current block after enhancement:

[0028] P′[x,y]=α×P[x,y]+β.

[0029] A further technical solution of the present invention is as follows: the prediction enhancement module utilizes the available spatial neighbors of the current block to enhance the prediction value, and also includes clamping the enhanced prediction value of the current block, limiting it within a normal range using ClipPel, and returning the specified minimum brightness value if it is less than a specified minimum brightness value; and returning the specified maximum brightness value if it is greater than a specified maximum brightness value.

[0030] A further technical solution of the present invention: the method of selecting the final prediction value and setting a flag indicating whether prediction enhancement is turned on according to the prediction values ​​before and after enhancement comprises:

[0031] Based on the prediction values ​​before and after enhancement, the SAD value of the sum of absolute errors before and after enhancement is obtained, and the SAD values ​​before and after enhancement are compared. If the SAD value after enhancement is small, it indicates that the enhancement is effective, that is, the original prediction value is replaced with the enhanced value, and the prediction enhancement on flag is set; if the SAD value after enhancement is large, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value, and the prediction enhancement off flag is set.

[0032] A video encoding method based on prediction enhancement, characterized in that the video encoder:

[0033] After the intra-frame predictor obtains the prediction value of the current block, the prediction value is input into the prediction enhancement module for enhancement; according to the prediction values ​​before and after enhancement, the final prediction value is selected and the flag indicating whether the prediction enhancement is turned on is set; the final prediction value is subtracted from the original signal to obtain the residual value, which is then transformed and quantized before entropy coding.

[0034] A video decoding device based on prediction enhancement, characterized in that it includes

[0035] An entropy decoder configured to decode a bitstream generated by a video encoder, obtain prediction information and a flag of a current block, and a residual value of the current block;

[0036] An intra-frame predictor configured to predict a current block based on prediction information of the current block and generate a prediction value;

[0037] A prediction enhancement module, which is configured to enhance the prediction value using the available spatial neighbors of the current block if the flag indicates that the prediction enhancement module is turned on;

[0038] An adder is configured to reconstruct the current block by adding the residual value of the current block to the prediction value.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] The present invention proposes a video encoding and decoding method and device based on prediction enhancement, designs a new prediction enhancement module combining spatial information, and the prediction enhancement module combines spatial correlation to build a model of prediction value and reconstruction value of the reference area of ​​the current block, and applies the model to the prediction value of the current block to obtain a more accurate prediction value and improve the encoding and decoding performance. Different from the existing neural network prediction enhancement method, the present invention can be implemented only with a method of lower complexity. The input of the present invention is not limited in size and shape, including non-square areas, and is more flexible to use, which is also a contribution that is obviously different from using neural networks for prediction enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The present invention is a video encoder framework that can implement the technology of the present invention.

[0042] Figure 2 are the current block and the reference block.

[0043] Figure 3 is the intra prediction mode and prediction direction.

[0044] Figure 4 It is the inter-frame prediction process and related techniques.

[0045] Figure 5 The present invention is a video decoder framework that can implement the technology of the present invention.

[0046] Figure 6 Encoder framework with predictive enhancement.

[0047] Figure 7 Decoder framework with prediction enhancement.

[0048] Figure 8 Design an overall idea for the prediction enhancement module.

[0049] Fig. 9 Flowchart of the prediction enhancement module.

[0050] Fig.10 A framework for a realizable encoder with inter-prediction enhancement.

[0051] Fig.11 A feasible decoder framework with inter-frame prediction enhancement. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] like Figure 1 The video encoding device framework that can implement the technology of the present invention is shown in FIG. The video encoding device framework includes: an image segmenter, a predictor, a subtractor, a transformer, a quantizer, an entropy encoder, an inverse quantizer, an inverse transformer, an adder, a loop filter unit and a memory.

[0054] Each component of the encoder can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented as the function of executing the software corresponding to each component.

[0055] The video sequence is presented by a series of continuous video frames, which can be single or multiple parallel sequences. During the video encoding process, each video frame to be encoded will be divided into several regions, and each region will be encoded separately. Specifically, a video frame may be divided into several independent tiles (Tiles) or / and slices (Slices) through an image segmenter. These tiles or slices can be further combined to form tile groups or slice groups for more efficient encoding. During the encoding process, each tile or / and slice will be subdivided into one or more coding tree units (Coding Tree Unit, CTU). These CTUs can be further divided into smaller coding units (Coding Tree, CU) in a tree structure. CU is the basic unit in the encoding and decoding process. Each CU contains specific coding information, which constitutes the syntax structure of the CU. At the same time, the relevant information of all CUs belonging to the same CTU will be organized into the syntax structure of the CTU. Information related to all blocks in a slice is encoded as the syntax of the slice header, while information related to all blocks that make up one or more video frames is encoded as a picture parameter set (PPS) or a picture header. Shared information across multiple video frames is encoded in a sequence parameter set (SPS). Furthermore, if there is shared information between multiple SPSs, this information is encoded in a video parameter set (VPS). In addition, shared information applicable to a tile or tile group can also be encoded as the syntax of a tile or tile group header. The syntax contained in the SPS, PPS, slice header, and tile or tile group header can be collectively referred to as high-level syntax, which is an indispensable part of the video encoding process. These high-level syntaxes ensure the accuracy of video data during encoding and decoding.

[0056] The image segmenter determines the size of the CTU. Information about the size of the CTU is encoded as the syntax of the SPS or PPS and transmitted to the decoder. The image segmenter segments each frame of the video into CTUs of a predetermined size, and then recursively segments the CTUs in a tree structure. The leaf nodes in the tree structure are called CUs. In the following, the block corresponding to the CU to be encoded or decoded is called the "current block", and the spatially adjacent pixel blocks in the current block are called reference blocks, such as Figure 2 shown.

[0057] The predictor is responsible for predicting the current block to generate a predicted block. The predictor includes an intra-frame predictor and an inter-frame predictor. Generally, the current block in the image can be encoded using predictive coding. Specifically, the prediction of the current block can be achieved using intra-frame prediction technology (using data from the video frame to which the current block belongs) or inter-frame prediction technology (using data from previously encoded video frames).

[0058] The intra-frame predictor uses the coded pixel values ​​in the current video frame to predict the pixel values ​​to be coded, thereby effectively removing the spatial correlation of the video. Intra-frame prediction can adopt multiple modes according to different prediction directions. For example, Figure 3 As shown in the solid line, the intra prediction mode includes two non-directional modes, namely the planar mode and the DC mode, and 65 different directional modes. Each prediction mode has its own unique neighboring pixel selection and algorithm formula.

[0059] In order to achieve more efficient directional prediction on a current block with a rectangular shape, additional use can be made of Figure 3 The dotted arrows in the figure show the directional modes (67 to 80 and -1 to -14). These modes are called "Wide Angle Intra Prediction (WAIP)". Figure 3 In the figure, the arrows indicate the positions of the reference samples used for prediction, but not the actual direction of the prediction. The prediction direction is opposite to the direction indicated by the arrows.

[0060] When the current block is rectangular, the wide-angle intra prediction mode can predict in the opposite direction of the specific direction mode without adding extra bits to the transmission. In this case, the selection of the available wide-angle intra prediction mode can be determined by the aspect ratio of the current block. For example, when the height of the current block is less than the width, a wide-angle intra prediction mode with an angle less than 45 degrees (intra prediction modes #67 to #80) can be used. When the width of the current block is greater than the height, a wide-angle intra prediction mode with an angle greater than -135 degrees can be used.

[0061] In order to compensate for the inadequacy of angle prediction in utilizing spatial information, the intra-frame predictor can use the prediction value correction technology (Position Dependent Prediction Combination, PDPC) to correct the initial prediction value obtained by intra-frame prediction. PDPC first calculates the initial value of each prediction mode, and then performs a weighted combination based on the position of the pixel in the block and the prediction values ​​from different directions (such as horizontal and vertical) to obtain the corrected prediction value.

[0062] In addition to angular prediction, the intra predictor can also use matrix-based intra prediction technology (MIP) to obtain an output vector by multiplying the downsampled reference pixel block with a fixed weight matrix, and upsample the output vector to the current block size to obtain the final prediction value.

[0063] The intra predictor determines the intra prediction mode to use to encode the current block. In some cases, the intra predictor encodes the current block using multiple intra prediction modes and selects the most appropriate intra prediction mode from these test modes. For example, the intra predictor can perform rate-distortion analysis on multiple test intra prediction modes, calculate the rate-distortion value for each mode, and then select the intra prediction mode with the best rate-distortion characteristics.

[0064] Once the intra predictor determines the best mode among multiple intra prediction modes, it uses the neighboring pixels (reference pixels) and algorithm equations defined in the mode to predict the current block. Subsequently, the entropy encoder encodes the information of the selected intra prediction mode and transmits it to the video decoding device so that the decoder can correctly decode and reconstruct the block.

[0065] The inter-frame predictor uses the temporal correlation of the video and uses the pixel values ​​in the adjacent encoded video frames to predict the pixel values ​​in the current video frame to be encoded, so as to remove the temporal redundancy of the video. The inter-frame predictor obtains a best matching block in the encoded image through motion estimation (ME), calculates the displacement from the best matching block to the current block to obtain the motion vector (MV), and uses MV to perform motion compensation (MC) to obtain the predicted value of the current block. The pixel difference between the current block and the reference block is called the prediction residual. The inter-frame encoder only encodes the motion information and prediction residual between images, thereby greatly improving the coding efficiency.

[0066] According to the encoding method of motion vector and prediction residual, inter-frame coding mode can be divided into three categories: Skip, Merge and Advanced Motion Vector Prediction (AMVP). Specifically, Skip mode uses predicted motion vector to represent MV information, and does not encode motion vector difference and prediction residual; Merge mode only encodes predicted motion vector and prediction residual, but not motion vector difference; AMVP mode encodes both predicted motion vector difference and residual.

[0067] The inter-frame prediction process can be divided into three steps: motion vector prediction, motion vector determination, and motion compensation. The specific process and related technologies are as follows: Figure 4 shown.

[0068] The subtractor generates a residual block by subtracting a prediction block generated by an intra predictor or an inter predictor from the current block.

[0069] The transformer transforms the residual signal in the residual block with spatial pixel values ​​into transform coefficients in the frequency domain. Through appropriate transformation, the dispersed distribution of image energy in the spatial domain can be transformed into a relatively concentrated distribution in the transform domain, thereby achieving the purpose of removing redundancy. The transformation process can be performed by using the entire residual block as a transform unit, or by dividing the residual block into multiple sub-blocks and performing the transform operation on these sub-blocks as transform units. For sub-block transforms, the entropy encoder encodes a flag indicating the operation of transforming only the sub-block, as well as the corresponding direction and position information, and notifies the video decoder of this information.

[0070] The transformer can also transform the residual block independently in the horizontal direction and vertical direction respectively. In this transformation, various transformation functions or matrices can be used. For example, paired transformation functions for horizontal and vertical transformations can be defined as a multiple transform set (MTS). The transformer can select the transformation function pair with the highest transformation efficiency in the MTS and transform the residual block in the horizontal direction and vertical direction respectively. The entropy encoder will encode the information about the transformation function pair in the MTS and send this information to the video decoder through the bitstream.

[0071] After the residual signal is transformed, the transform coefficients often have a larger dynamic range. The quantizer is responsible for quantizing the transform coefficients output by the transformer to reduce the signal value space, thereby achieving better compression effect.

[0072] The quantizer processes based on the quantization parameters and transmits the quantized coefficients to the entropy encoder. The quantizer can use the correlation between the transform coefficients for dependent quantization, so that the transform coefficients are more compact in the M-dimensional vector space after quantization.

[0073] The entropy encoder encodes the quantized transform coefficient sequence using a variety of coding schemes including variable length coding, exponential Golomb coding, context-based adaptive binary arithmetic coding (CABAC) to generate an encoded bitstream. In addition, the entropy encoder also encodes information related to block segmentation, including CTU size, CTU segmentation flag, CTU division method according to tree structure, etc., to ensure that the video decoding device can perform block segmentation in a consistent manner with the video encoding device. In addition, the entropy encoder encodes the prediction type information indicating whether the current block is intra-frame prediction coding or inter-frame prediction coding. Depending on the selected prediction type, the entropy encoder further encodes intra-frame prediction information (such as intra-frame prediction mode, etc.) or inter-frame prediction information (such as merge index in merge mode, and reference image index and motion vector difference in AMVP mode, etc.). Finally, the entropy encoder also encodes information related to quantization, including information such as quantization parameters and quantization matrices.

[0074] The dequantizer dequantizes the quantized transform coefficients received from the quantizer to generate corresponding transform coefficients.

[0075] The inverse transformer converts these transform coefficients from the frequency domain back to the spatial domain to reconstruct the residual block.

[0076] The adder adds the reconstructed residual block to the prediction block generated by the predictor to reconstruct the current block and obtain the reconstructed block. When performing intra-frame prediction for the next block, the reconstructed block will be used as the reference block for the current block. The position relationship between the reference block and the current block is as follows: Figure 2 shown.

[0077] The loop filter is used to filter during the video encoding process to reduce common coding distortions such as block effect, ringing effect, color deviation and image blur in block-based hybrid coding methods. The filtered image is used for the encoding of subsequent images, that is, it is in the encoding "loop". On the one hand, the loop filter improves the quality of the encoded image, and on the other hand, it provides a high-quality reference image for subsequent encoded images, thereby obtaining better prediction effects and improving coding efficiency. The loop filter is generally composed of one or more of the three modules: De-Blocking Filter (DBF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF).

[0078] The DBF filter smoothes the reconstructed block boundaries to reduce the discontinuity at the block coding boundaries and achieve the purpose of removing the block effect. In block-based transformation and quantization, for strong edges in the image, due to the quantization distortion of the high-frequency AC coefficient, ripples will be generated around the edges after decoding. This distortion is called the ringing effect, which seriously affects the subjective and objective quality of the video. The SAO filter uses CTU as a unit, divides the reconstructed pixels into categories by selecting a suitable classifier, and uses boundary compensation or sideband compensation for pixels of different categories to reduce the ringing phenomenon. The ALF filter is based on the Wiener filtering principle. It establishes the Wiener-Hoff equation through the original image information and the reconstructed image information to solve a series of filter coefficients with the minimum mean square error, so as to reduce the decoding error and effectively improve the quality of the video frame. The filter coefficients are written into the bitstream through the entropy encoder.

[0079] The memory is used to store the reconstructed blocks after loop filtering. When all blocks in a video frame are reconstructed, the reconstructed whole frame image can be used as a reference image for inter-frame prediction of the current block in the subsequent encoded image.

[0080] like Figure 5 The video decoding device framework that can implement the technology of the present invention is shown in FIG. The video decoding device framework may include: an entropy decoder, an inverse quantizer, an inverse transformer, a predictor, an adder, a loop filter, and a memory.

[0081] Similar to Figure 1 Each component of the video encoding device and the video decoding device can be implemented as hardware or software, or as a combination of hardware and software. In addition, the function of each component can be implemented as software, and the microprocessor can also be implemented to execute the function of the software corresponding to each component.

[0082] The entropy decoder extracts information related to block segmentation by decoding the bitstream generated by the video encoder to determine the current block to be decoded and obtain prediction information and residual information for reconstructing the current block. The entropy decoder extracts CTU size information from the SPS or PPS to determine the size of the CTU, and divides the video frame into CTUs of a predetermined size accordingly. In addition, the CTU is set as the root node of the tree structure, and the entropy decoder extracts information related to CTU segmentation, and obtains the CU by segmenting the CTU using the tree structure.

[0083] When determining the decoding of the current block, the entropy decoder extracts the prediction type information indicating whether the current block is intra-predicted or inter-predicted by using the partitioning information of the tree structure. For intra-prediction, the decoder obtains syntax elements such as the intra-prediction mode of the current block; if it is inter-prediction, the decoder obtains relevant syntax elements including motion vectors, reference images, and prediction residuals. In addition, the entropy decoder also extracts information related to quantization and residual signal information related to the quantized transform coefficients of the current block.

[0084] The inverse quantizer inversely quantizes the transform coefficients quantized by the quantization parameter QP.

[0085] The inverse transformer performs inter-domain conversion on the transform coefficients generated by the inverse quantizer to restore the reconstructed residual information, thereby obtaining a residual block of the current block.

[0086] The predictor may include an intra predictor and an inter predictor. When the prediction type of the current block is intra prediction, the intra predictor is activated for use, and when the prediction type of the current block is inter prediction, the inter predictor is activated for use.

[0087] The intra predictor selects an intra prediction mode of a current block from a plurality of intra prediction modes according to the intra prediction mode syntax element extracted from the entropy decoder, and predicts the current block using adjacent reference pixels of the current block to obtain a predicted block.

[0088] The inter-frame predictor determines the motion vector of the current block and the reference image referred to by the motion vector by using the syntax elements of the inter-frame prediction mode extracted from the entropy decoder, and predicts the current block using the reference block in the reference image to obtain a predicted block.

[0089] The adder reconstructs the current block by adding the residual block output from the inverse transformer to the prediction block output from the inter-frame predictor or the intra-frame predictor. When intra-frame prediction is performed on a subsequent block to be decoded, the pixels in the reconstructed current block can be used as reference pixels in the reference block.

[0090] The loop filter unit of the loop filter may include one or a group of DBF filters, SAO filters and ALF filters. The DBF filter performs smoothing filtering on the boundaries between the reconstructed blocks to remove the discontinuous block effects caused by block unit decoding. The SAO filter and ALF perform additional filtering on the reconstructed blocks after DBF filtering to compensate for the difference between the reconstructed pixels and the original pixels caused by lossy coding and improve the quality of the video frame. The filter coefficients of the filter are obtained from the decoded bit stream of the entropy decoder.

[0091] The reconstructed blocks after loop filtering will be stored in the memory. When all blocks in an image are reconstructed, the entire reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent video frames that need to be decoded.

[0092] In some embodiments the present invention relates to a video encoding and decoding device as described above. More specifically:

[0093] The present invention first proposes an intra-frame brightness prediction enhancement module for video codecs, that is, in the intra-frame prediction encoder, after the original encoder obtains the intra-frame prediction value, the intra-frame prediction enhancement module proposed by the present invention is added to refine the prediction value. The sum of absolute differences (SAD) is used as an indicator to determine whether the prediction enhancement module is turned on, and the corresponding flag is written into the bitstream through the entropy encoder to reduce the residual and improve the encoding performance. The encoder framework with prediction enhancement is as follows Figure 6 As shown in . In the original decoder, after the intra-frame predictor performs intra-frame prediction to obtain the intra-frame prediction value, the entropy decoder obtains the flag information indicating whether the intra-frame prediction enhancement module is enabled. If enabled, the prediction enhancement is completed and then added to the corresponding residual value to obtain the reconstructed value; if not enabled, no prediction enhancement is required. The decoder framework with prediction enhancement is shown in Figure 7 shown.

[0094] (Note: 1. The video encoder framework can be extended, see Example 2; 2. Intra-frame brightness prediction can be extended, see Example 3 and Example 4; 3. Indicating whether the enhancement module is turned on can be extended, see Example 5)

[0095] The present invention proposes an intra-frame prediction enhancement module, which makes full use of the available spatial neighbor information for reconstruction around the current block, and uses the relationship between the predicted value and the reconstructed value of the available spatial neighbors to enhance the predicted value of the current block to reduce the prediction error. The idea and implementation of this module are the core protection points of the present invention. The implementation of this module consists of four steps. Step 1: Obtain the reference area of ​​the current block (Note: the reference area can be expanded, see Example 7). Step 2: Obtain the predicted value and reconstructed value of the reference area of ​​the current block. Step 3: Construct a model of the predicted value and reconstructed value of the reference area of ​​the current block (Note: the model construction is expandable, see Example 6 and Example 8). Step 4: Apply the model in step 3 to the predicted value of the current block, so as to obtain an enhanced predicted value of the current block. Steps 3 and 4 deeply embody the core idea of ​​the present invention.

[0096] The following embodiments may be generally applicable to video codecs and prediction enhancement modules using video codecs.

[0097] Based on the enhancement method implemented by the prediction enhancement module, the present invention proposes a video decoding method based on prediction enhancement, including: decoding the code stream generated by the video encoder, obtaining the prediction information and flag of the current block, and the residual value of the current block; predicting the current block based on the prediction information of the current block to generate a prediction value; if the flag shows that the prediction enhancement module is turned on, enhancing the prediction value using the available spatial neighbors of the current block; and reconstructing the current block by adding the residual value of the current block to the prediction value.

[0098] Based on a prediction enhancement module, the present invention proposes a video decoding device based on prediction enhancement, including an entropy decoder, which is configured to decode a code stream generated by a video encoder to obtain prediction information and a flag of a current block, and a residual value of the current block; an intra-frame predictor, which is configured to predict the current block based on the prediction information of the current block and generate a prediction value; a prediction enhancement module, which is configured to enhance the prediction value using the available spatial neighbors of the current block if a flag shows that the prediction enhancement module is turned on; and an adder, which is configured to add the residual value of the current block to the prediction value to reconstruct the current block.

[0099] Embodiment 1:

[0100] Based on H.266 / VVC, the present invention proposes a new intra-frame prediction enhancement module that combines spatial information to improve coding performance. A video encoding and decoding method is implemented based on the intra-frame prediction enhancement module. The following method is taken as an example to illustrate the implementation process of the technical solution.

[0101] In the video encoder, when intra-frame prediction is selected, after the intra-frame predictor obtains the prediction value of the current block, the prediction value is input into the prediction enhancement module for prediction enhancement. The SAD value before and after the enhancement is used to determine whether the enhancement is effective. If the SAD value after enhancement is small, it indicates that the enhancement is effective, that is, the original prediction value is replaced with the enhanced value; if the SAD value after enhancement is large, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value. Whether to turn on prediction enhancement is sent to the entropy encoder in the form of a flag and written into the bitstream. The final prediction value is subtracted from the original signal to obtain the residual, which is entropy encoded using the entropy encoder after transformation and quantization. In the decoder, the entropy decoder decodes the bitstream to obtain the information of the current block and the residual value of the current block. The information of the current block includes: prediction related information, prediction enhancement flag, etc. The intra-frame prediction value is calculated by predicting relevant information. If the flag shows that the prediction enhancement module is turned on, that is, after the prediction enhancement is performed, the enhanced prediction value is added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value; if the flag shows that the prediction enhancement module is not turned on, that is, after the prediction value is calculated, it is directly added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value. Figure 6 and Figure 7The red part in the middle shows the location of the intra-frame prediction enhancement scheme in the codec under the H.266 / VVC framework. The prediction enhancement module is added by this technical solution.

[0102] The following is a detailed description of the prediction enhancement module. After the encoder predicts the current block, the prediction value is enhanced using the available spatial neighbors of the current block. Combined with the inter-block correlation, it is believed that the relationship and law between the prediction value and the reconstructed value of the available spatial neighbors and the current block are the same. Taking the L-type reference area as an example, the overall design idea of ​​the prediction enhancement module is as follows: Figure 8 . Among them, LC and LP represent the reconstruction value and prediction value of the reference area respectively, and P and P' represent the prediction value and enhanced prediction value of the current block respectively.

[0103] like Fig. 9 As shown in Figure 2, the implementation process of the prediction enhancement module is as follows:

[0104] Step 1: Get the reference area of ​​the current block.

[0105] Step 2: Get the predicted value and reconstructed value of the reference area of ​​the current block.

[0106] Step 3: Build a model of the predicted and reconstructed values ​​of the reference area of ​​the current block.

[0107] Step 4: Apply the model in step 3 to the predicted value of the current block to obtain an enhanced predicted value of the current block.

[0108] The specific process of obtaining the reference area of ​​the current block in step 1 is as follows:

[0109] Analyze whether the current block boundary is available. In this embodiment, determine whether the predicted and reconstructed pixels at the upper left, upper side and left side boundaries (1 row / 1 column) of the current prediction block are available. If not all are available, the reference area is not constructed, that is, the block is not predicted and enhanced. If the predicted and reconstructed pixels at the upper left, upper side and left side boundaries (1 row / 1 column) of the current prediction block are all available, the left side, upper left and upper side (1 row / 1 column) of the current prediction block are the reference areas. If the size of the current block is W*H, W is the width of the current block, and H is the height of the current block, the constructed reference area should contain (W+H+1) pixels. In general, only the prediction blocks that contain valid reference areas are enhanced. Apart from this, there are no restrictions on the size, shape and position of the prediction blocks. The reference area is as follows Figure 2 As shown, it consists of 1 row / column of brightness samples above and to the left of the current block.

[0110] Step 3: The process of constructing the model of the predicted value and the reconstructed value of the reference area of ​​the current block is:

[0111] Step 1: Establish a model of the predicted value and the reconstructed value of the reference area. For each pixel position of the constructed reference area, there is a corresponding predicted value and reconstructed value. In the present invention, a model is used to describe and construct the model of the predicted value and the reconstructed value at the same pixel position of the reference area. Now take the linear model as an example to illustrate the construction process.

[0112] L C (x,y)=α×L P (x,y)+β

[0113] Among them, L C and L P are the reconstructed value and predicted value of the current prediction block boundary respectively. α and β are two model parameters.

[0114] Step 2: Use the least squares method to solve the model parameters and find the best function match for the data by minimizing the sum of squares of the errors to obtain a curve that best represents the relationship between the predicted value and the reconstructed value of the reference area of ​​the current block. First, calculate the mean of the predicted value and the reconstructed value in the reference area. and Among them, n represents the total number of pixels in the reference area. The current n should be W+H+1, L P (x i ,y i ) and L C (x i ,y i ) respectively represent (x i ,y i ) The predicted value and reconstructed value at the position. The slope α of the linear model is calculated as follows:

[0115]

[0116] The corresponding intercept is calculated as follows:

[0117]

[0118] Step 4: The process of obtaining an enhanced prediction value of the current block is as follows:

[0119] Step 1: When the above linear model is applied to prediction enhancement, P is the prediction value of the current block, and P' is the prediction value of the current block after enhancement.

[0120] P′[x,y]=α×P[x,y]+β

[0121] Step 2: Since the enhanced prediction value obtained according to the linear model may not be within the brightness range (for 8-bit video, the normal range is 0-255; for 10-bit video, the normal range is 0-1023), it is necessary to clamp the enhanced prediction value of the current block. Use ClipPel to limit it to the normal range. If it is less than the specified minimum brightness value, the specified minimum brightness value is returned; if it is greater than the specified maximum brightness value, the specified maximum brightness value is returned.

[0122] Embodiment 2:

[0123] The difference from Example 1 is that the coding framework based on Example 1 is VVC, and Example 2 can implement this method on other coding frameworks, such as AV1, NNVC, ECM, AVS3, etc., but is not limited to the above-mentioned coding frameworks.

[0124] Embodiment 3:

[0125] The difference between Example 3 and Example 1-2 is that Example 1-2 implements intra-frame brightness prediction enhancement, and Example 3 can be implemented in different prediction links, such as migrating the method of the present invention for inter-frame prediction enhancement. In the encoder, after the prediction value of the current block is obtained by inter-frame prediction, the prediction value is input into the prediction enhancement module for enhancement. The SAD value before and after the enhancement is used to judge whether the enhancement is effective. If the SAD value after enhancement is small, it indicates that the enhancement is effective, that is, the original prediction value is replaced by the enhanced value; if the SAD value after enhancement is large, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value. Whether to turn on the prediction enhancement is written into the bitstream through the entropy encoder in the form of a flag. The final prediction value is subtracted from the original signal to obtain the residual, which is entropy encoded after transformation and quantization. In the decoder, the information of the current block and the residual value of the current block are decoded from the bitstream by the entropy decoder. The information of the current block includes: prediction related information, prediction enhancement flag, etc. The inter-frame prediction value is calculated through the prediction information. If the flag shows that the prediction enhancement module is turned on, that is, after the prediction enhancement is performed, the enhanced prediction value is added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value; if the flag shows that the prediction enhancement module is not turned on, that is, after the prediction value is calculated, it is directly added to the residual after inverse quantization and inverse transformation to obtain the reconstructed value.

[0126] Fig.10 and 11 The position of the inter-frame prediction enhancement scheme in the codec is given in the framework of the codec that can implement the technology of the present invention. The red prediction enhancement module in the figure is added by this technical solution.

[0127] Embodiment 4:

[0128] The difference between the fourth embodiment and the above-mentioned embodiments 1-3 is that the above-mentioned embodiments 1-3 are for brightness prediction enhancement, and the present embodiment can be for chrominance prediction enhancement. When the brightness and chrominance use the method of the present invention at the same time, by adding control variables, it is possible to select whether the brightness and chrominance use the prediction enhancement method in the form of decision-making to obtain a more accurate prediction block.

[0129] Embodiment 5:

[0130] The difference between Embodiment 5 and Embodiments 1-4 is that Embodiments 1-4 use the SAD value before and after the enhancement to determine whether the enhancement is effective at the encoding end, and write the corresponding flag into the bitstream. At the decoding end, after entropy decoding, the prediction enhancement module is turned on according to the flag indication. For this part, the following extensions can be made:

[0131] All prediction blocks containing valid reference areas are enhanced without the need to pass a flag. That is, at the encoding end, enhancement is performed directly after obtaining the intra-frame prediction value. At the decoding end, enhancement is performed directly after obtaining the intra-frame prediction value.

[0132] All prediction blocks containing valid reference areas are enhanced without the need to send a flag. The relationship between the information of the current block (such as prediction mode, reference samples, etc.) and whether the enhancement is effective is analyzed. At the encoding end, the enhancement is determined based on the information of the current block. At the decoding end, the enhancement is also determined directly based on the information of the current block without the need to send a flag.

[0133] 3. The intra-frame prediction enhancement module is used as a new mode to compete with other prediction modes. On the encoding side, prediction enhancement will compete with other prediction modes, and set a flag indicating whether the enhancement module is turned on by comparing the rate-distortion cost with other modes. On the decoding side, if the enhanced flag is turned on, the predicted value will be enhanced and then the reconstructed value will be calculated.

[0134] 4. The indicator for judging whether the enhancement is effective can be replaced with RDCost (Rate Distortion Cost). If the RDCost value after enhancement is smaller, it means that the enhancement is effective. In this case, the prediction value before enhancement is replaced by the prediction value after enhancement. If the RDCost value after enhancement becomes larger, it means that the enhancement is invalid. In this case, the prediction value before enhancement is used as the final prediction value. Whether to enable prediction enhancement is written into the bitstream in the form of a flag. The final prediction value is subtracted from the original signal to obtain the residual, which is then entropy encoded after transformation and quantization. At the decoding end, after entropy decoding, the flag indicates whether to enable the prediction enhancement module.

[0135] Embodiment 6:

[0136] The difference between Example 6 and the above-mentioned Examples 1-5 is that, in step 3 of the above-mentioned Examples 1-5, a linear model is used to construct a model of the predicted value and the reconstructed value of the reference area. This embodiment can also use other models to describe the relationship between the predicted value and the reconstructed value of the reference area, such as a convolution model, an MMLM model, etc., but is not limited to the above-mentioned models. And for different intra-frame prediction modes or other information of the current block, such as the prediction information of the current block, a model can be adaptively selected and constructed for prediction enhancement.

[0137] Embodiment 7:

[0138] The difference between the embodiment 7 and the above embodiments 1-6 is that the reference area constructed in step 1 of the above embodiments 1-6 is the boundary of 1 row / 1 column in the upper left, upper side and left side of the current block. In this embodiment, the reference area can be constructed as N rows and N columns (N>1), or the reference area of ​​the current block can be constructed with M rows and M columns in the lower left, left side, upper left, upper side and upper right (M≥1), or a reference area of ​​any other shape can be constructed.

[0139] Embodiment 8:

[0140] The difference between Example 8 and the above-mentioned Examples 1-7 is that, in step 3 of the above-mentioned Examples 1-7, all pixel values ​​are used to calculate model parameters when constructing the model, and this example can also select some pixels to calculate model parameters to further reduce complexity.

[0141] In summary, the present invention makes fuller use of the spatial correlation of video frames to reduce the prediction error and improve the encoding performance. BD-rate is an objective metric used in video encoding. Within a certain bit rate or quality range, this metric can compare the rate-distortion performance of two different video codecs or different configurations of the same video codec. When the value is negative, it means that the compression efficiency is improved.

[0142] The method proposed in the present invention is carried out on the VVC standard test code VTM-20.0 platform, and the proposed method is used to realize intra-frame brightness prediction enhancement for blocks with valid reference areas. Under the AI ​​configuration, the configuration file selects encoder_intra_vtm.cfg, the number of encoded frames is 8 frames (TemporalSubsampleRatio is set to 1), and the standard test sequence is encoded and decoded when the quantization parameter QP is 22, 27, 32, and 37, and the bit rate and peak signal-to-noise ratio PSNR during video encoding are recorded. The objects of encoding are standard VVC test videos, and their names and resolutions are BasketballPass (416×240), BQSquare (416×240), BlowingBubbles (416×240), and RaceHorses (416×240). From the average value, compared with VVC, the bit rates of the Y, U, and V components are reduced by 0.40%, 0.77%, and 0.74%, respectively, thereby verifying the effectiveness of the present method.

[0143] Table 1 Coding performance of the present invention compared with VTM20.0 under AI configuration

[0144]

[0145] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A video decoding method based on prediction enhancement, characterized in that: include: Decode the bitstream generated by the video encoder to obtain the prediction information and flag of the current block and the residual value of the current block; Predicting the current block based on the prediction information of the current block to generate a prediction value; If the flag indicates that the prediction enhancement module is turned on, the prediction value is enhanced using the available spatial neighbors of the current block; The residual value of the current block is added to the predicted value to reconstruct the current block.

2. The video decoding method based on prediction enhancement according to claim 1, characterized in that: The step of enhancing the predicted value based on the available spatial neighbors of the current block includes: Get the reference area of ​​the current block; Obtain the predicted value and the reconstructed value of the reference area of ​​the current block; Construct a model of predicted and reconstructed values ​​for the reference area of ​​the current block; The model of the predicted value and the reconstructed value of the reference area of ​​the current block is applied to the predicted value of the current block to obtain an enhanced predicted value of the current block.

3. The video decoding method based on prediction enhancement according to claim 2, characterized in that: The obtaining of the reference area of ​​the current block includes: Determine whether the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are available; if not all are available, no reference area is constructed, that is, no prediction enhancement is performed on the block; if the predicted and reconstructed pixels at the upper left, upper side and left side boundaries of the current prediction block are all available, the left side, upper left and upper side of the current prediction block are the reference areas; if the size of the current block is W*H, where W is the width of the current block and H is the height of the current block, the constructed reference area should contain W+H+1 pixels.

4. The video decoding method based on prediction enhancement according to claim 2, characterized in that: The step of constructing a model of predicted values ​​and reconstructed values ​​of a reference area of ​​the current block includes: Model the predicted and reconstructed values ​​for the reference region: L C (x,y)=α×L P (x,y)+β Among them, L C and L P are the reconstructed value and predicted value of the boundary of the current prediction block, respectively. α and β are two model parameters; The model parameters are obtained by solving the predicted values ​​and the reconstructed values ​​of the current reference area.

5. The video decoding method based on prediction enhancement according to claim 2, characterized in that: Apply the model of the predicted value and the reconstructed value of the reference area of ​​the current block to the predicted value of the current block to obtain an enhanced predicted value of the current block, including: When the model of the reference area prediction value and the reconstruction value is applied to the prediction enhancement, P is the prediction value of the current block, and P' is the prediction value of the current block after enhancement: P′[x,y]=α×P[x,y]+β.

6. The video decoding method based on prediction enhancement according to claim 2, characterized in that: The prediction enhancement module utilizes the available spatial neighbors of the current block to enhance the prediction value, and also includes clamping the enhanced prediction value of the current block, limiting it within a normal range using ClipPel. If it is less than the specified minimum brightness value, the specified minimum brightness value is returned; if it is greater than the specified maximum brightness value, the specified maximum brightness value is returned.

7. The video decoding method based on prediction enhancement according to claim 1, characterized in that: The step of selecting the final prediction value and setting a flag indicating whether prediction enhancement is turned on according to the prediction values ​​before and after enhancement includes: Based on the prediction values ​​before and after enhancement, the SAD value of the sum of absolute errors before and after enhancement is obtained, and the SAD values ​​before and after enhancement are compared. If the SAD value after enhancement is small, it indicates that the enhancement is effective, that is, the original prediction value is replaced with the enhanced value, and the prediction enhancement on flag is set; if the SAD value after enhancement is large, it indicates that the enhancement is invalid, that is, the original prediction value is the final prediction value, and the prediction enhancement off flag is set.

8. A video coding method based on prediction enhancement, characterized in that: The video encoder: After the intra-frame predictor obtains the prediction value of the current block, the prediction value is input into the prediction enhancement module for enhancement; according to the prediction values ​​before and after enhancement, the final prediction value is selected and the flag indicating whether the prediction enhancement is turned on is set; the final prediction value is subtracted from the original signal to obtain the residual value, which is then transformed and quantized before entropy coding.

9. A video decoding device based on prediction enhancement, characterized in that: include An entropy decoder configured to decode a bitstream generated by a video encoder, obtain prediction information and a flag of a current block, and a residual value of the current block; An intra-frame predictor configured to predict a current block based on prediction information of the current block and generate a prediction value; A prediction enhancement module, which is configured to enhance the prediction value using the available spatial neighbors of the current block if the flag indicates that the prediction enhancement module is turned on; An adder is configured to reconstruct the current block by adding the residual value of the current block to the prediction value.

Images