Compression artifact reduction method incorporating multi-level interframe correlation

Abstract

The application provides a compression artifact suppression method combined with multi-level interframe correlation. The method mainly relates to a means combining multi-level motion compensation and deep convolution mapping to perform quality enhancement on HEVC decoded video. The application mainly utilizes the characteristics of interframe correlation of the video, and then designs a multi-level motion compensation network to perform motion compensation on two adjacent frames of a current frame, and then fuses features of the current frame with the two adjacent frames. Then, attention feature extraction is performed on the current frame and the fused multi-frame feature map respectively. Finally, deep residual learning and cross-layer cascade are performed to realize deep fusion, mapping and reconstruction of multi-level features, so that quality enhancement of the current frame is realized.

Description

Technical Field

[0001] This invention relates to the research on video quality improvement methods in the field of video coding, and in particular to a compression artifact suppression method that combines multi-level inter-frame correlation. Background Technology

[0002] With the ever-increasing volume of internet information and data, and the growing amount of data transmitted between terminals, how to transmit data within limited bandwidth has become a significant concern. We also encounter more and more video data in our daily lives. It is estimated that by 2022, video transmission will account for 80% of all internet transmissions globally, and high-definition video has become a basic requirement for daily viewing. Due to limitations in video encoding and decoding technologies, it is difficult to balance transmission efficiency with video quality. Therefore, a technology focused on efficiently improving the quality of compressed video is crucial. Traditional encoding and decoding standards such as H.264 and H.265 have integrated deblocking filtering and sample adaptive filtering, which, while improving the quality of compressed video to some extent, have only achieved limited visual enhancement. With the development of deep learning, various image and video processing technologies based on deep learning have made significant progress, leading to advancements in methods for improving the quality of compressed images and videos.

[0003] Traditional image denoising techniques mostly employ linear or nonlinear filters, such as Gaussian filters, using operations like weighting individual pixel values ​​within an image. While effective against certain types of noise, these techniques often require replacing the filter if the noise characteristics change. However, noise removal methods based on convolutional neural networks (CNNs) effectively circumvent this drawback. DnCNN proposes a method that uses a single type of noise to train a model, such as Gaussian noise, to perform multi-task processing, including blind Gaussian denoising, SISR, or JPEG deblocking. The use of residual learning to avoid gradient vanishing improves network stability, reflecting the generalization ability of neural networks to some extent. Meanwhile, Yang designed a network to directly process HEVC decoded video streams, achieving targeted denoising of compressed videos based on the HEVC standard. Adaptive Block Transform Neural Network (ASN) and Multi-Frame Video Quality Enhancement Network (MFQE) have been proposed. The former utilizes the CU block information from H.265 block coding to enhance the removal of compression artifacts, while the latter leverages the inter-frame correlation characteristics of video to perform motion compensation on adjacent frames from multiple input frames, thereby enhancing the quality of the current frame.

[0004] However, most current video quality enhancement methods still have limitations. Some methods focus on single-frame video enhancement, emphasizing image quality enhancement techniques while ignoring the temporal and spatial correlations between video frames. Meanwhile, some methods consider the encoder's CU block information and integrate it into the quality enhancement part, but the results are still unsatisfactory. Other methods utilize the inter-frame correlation but ignore the low-quality characteristics of the video decoded frames and fail to consider the impact of compression artifacts of different sizes, resulting in feature extraction that does not conform to the distribution characteristics, leading to insignificant enhancement effects in the final enhanced frames. Summary of the Invention

[0005] With the ever-increasing volume of internet information and data, and the growing amount of data transmitted between terminals, how to transmit data within limited bandwidth has become a significant concern. We also encounter more and more video data in our daily lives. It is estimated that by 2022, video transmission will account for 80% of all internet transmissions globally, and ultra-high-definition video has become a basic requirement for daily viewing. With the increasing prevalence of 4K and even 8K video, there is a strong demand for efficient video encoding and decoding technologies. However, high compression rates often result in high distortion, so much related research focuses on improving the quality of distorted image frames.

[0006] Traditional encoding and decoding standards such as H.264 and H.265 have integrated deblocking filtering and sample adaptation, which improve the quality of compressed video to some extent, but the actual visual improvement is limited. With the development of deep learning, various image and video processing technologies based on deep learning have made significant progress, and methods for improving the quality of compressed images and videos have also advanced. In this invention, we utilize a multi-level inter-frame motion prediction and compensation network to perform motion compensation on multiple frames, specifically including the following process steps:

[0007] (1) Remove artifacts from compressed video;

[0008] (2) The first part of this invention includes a multi-level motion compensation network that utilizes the strong inter-frame correlation between the forward and backward frames and the current frame to perform global-to-local motion compensation. It adopts a four-layer motion compensation method that progressively upsamples the entire image by 8 times. The compensation of each layer can be expressed as follows:

[0009]

[0010] The overall expression for this network can be represented as:

[0011]

[0012] This invention also includes a feature depth extraction mapping part, which first includes a multi-dimensional feature extraction part. First, the input image is expanded in dimension, then average pooling is performed on different channels at the same pixel position to obtain the average value of each channel. The expanded feature map is then weighted by channel. Next, the output image of this part is subjected to average pooling and max pooling of each channel to extract representative feature values ​​in the feature map spatial domain. After convolution, this is multiplied by the input feature map of this sub-part to achieve pixel weighting in the spatial domain, increasing the weight ratio of key feature maps. Following the multi-dimensional feature extraction, a... The deep feature mapping and reconstruction part consists of a seven-layer structure that integrates residual learning and dense cross-layer connections. The residual learning sub-part sums the previous convolutional results of the current convolutional layer as input. Each convolutional layer is a Convolution-ReLU structure with a standard 3×3 kernel. The dense cross-layer connections cascade the inputs of each layer at the end of the sub-part to achieve deep fusion of deep and shallow features. This fused feature map is input into the reconstruction part, which uses a two-layer convolutional structure to gradually reduce dimensionality and achieve better reconstruction results.

[0013] (3) The objective function of the compression artifact removal network in this invention can be expressed as:

[0014]

[0015] Where L loss The loss function of this invention employs a strategy combining mean absolute value error loss and mean squared error loss to form the loss function of the network of this invention. It uses the mean absolute value error loss of the large weight λ1 and the mean squared error loss of the small weight λ2, and can be expressed as:

[0016]

[0017] (4) This invention selects a seven-frame sequence of Vimeo90K as the dataset for training the network; all video sequences are processed in accordance with the ITU-R and BT.601 standards, that is, firstly, the continuous images are converted into YUV format video sequences, and then compressed by HM16.0 to obtain compressed video. The compression configuration refers to the configuration file encoder_lowdelay_P_main.cfg, QP is set to 37, and IPPP inter-frame coding is used. Finally, the compressed video and the original video are converted into RGB domain images respectively for experimental input and effect observation.

[0018] The advantages and beneficial technical effects of this invention compared with the prior art are as follows:

[0019] (1) This invention proposes a multi-level deep feature mapping multi-frame input network for HEVC compressed video quality enhancement tasks.

[0020] (2) After four layers of inter-frame motion compensation by inputting the current frame and the two adjacent frames of the current frame, attention mechanism enhancement is performed to improve the influence of key features. Finally, feature depth mapping and reconstruction are achieved through seven layers of residual learning and dense cross-connection.

[0021] (3) Experimental results show that our algorithm can achieve good results in both objective and subjective evaluation indicators. It can achieve good results at QP values ​​of 37 and 42. It improves PSNR performance by an average of 0.68 dB and 0.64 dB compared to the standard HEVC, respectively, and improves SSIM performance by 0.0111 and 0.0154 compared to the standard HEVC, respectively. Attached Figure Description

[0022] Figure 1 This is the overall network framework for a compression artifact suppression method that incorporates multi-level inter-frame correlation.

[0023] Figure 2 This is a multi-level motion compensation map that incorporates a compression artifact suppression method based on multi-level inter-frame correlation.

[0024] Figure 3 This diagram illustrates the fusion attention mechanism for compressive artifact suppression methods that incorporate multi-level inter-frame correlation.

[0025] Figure 4 This is a visual comparison chart of the present invention and the HEVC standard. Detailed Implementation

[0026] The present invention will be further described in detail below with reference to the embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention based on the above-described invention, and these improvements and adjustments should still fall within the scope of protection of the present invention.

[0027] (1) This invention proposes a post-processing algorithm for removing compression artifacts by combining multi-level inter-frame correlation. To verify the effectiveness of the proposed algorithm, it was applied to a video compressed using the HEVC standard HM16.0 test model. The experiment used the LDP-based IPPP standard inter-frame coding configuration file encoder_lowdelay_P_main.cfg. Sixteen standard test video sequences were selected from four resolutions: 416×240, 832×480, 1280×720, and 1920×1080, with quantization QP values ​​of 37 and 42, respectively. All experiments were run using MATLAB R2016a and PyCharm 2020 on an environment with an i5-2320@3.00GHz CPU and an Nvidia RTX 2080 GPU.

[0028] (2) In this experiment, only the training set with QP=37 was used for training. For the test sequence, the reference frame when processing the current frame is the previous frame and the next frame to perform blind quality enhancement of the video sequence.

[0029] Table 1 describes the layer design of the quality enhancement network in this invention. Convolutional layers are denoted by Conv, and the stride is set to 1. The feature extraction part of the fusion attention mechanism has two symmetrical parts that extract features from the current frame, the compensated frame (combining two frames), and the single current frame, respectively. The filter configurations for both parts are identical, so the description of the second part is omitted using "...". The convolutional layers in the deep feature mapping part are residual connections, and the filter configurations are identical; therefore, all filters except the first filter are omitted using "...".

[0030] Table 1. Configuration of the algorithm network in this invention.

[0031]

[0032]

[0033] The algorithm of this invention and other HEVC quality improvement algorithms mainly use two commonly used evaluation metrics, PSNR and SSIM, to compare the performance of the algorithm of this invention, as well as DCAD, DSCNN, and MFQE. All comparison results are based on the improvement value of HEVC. For both PSNR and SSIM, the higher the value, the better the image quality, and the SSIM value ranges from -1 to 1. As shown in Table 2, the comparison results of this invention and other algorithms under PSNR can be seen. It can be found that this invention has significantly better results than other methods when the quantization step size is 37 and 42, improving the quality of HEVC decoded frames by 0.683dB and 0.647dB under the two compression conditions, respectively.

[0034] Table 2 Comparison of this algorithm with other algorithms based on PSNR (dB)

[0035]

[0036] As shown in Table 3, the present invention also achieves better results than other algorithms under the SSIM metric, with an average improvement of 0.0111 and 0.0148 compared to the HEVC standard decoded frame at quantization step sizes of 37 and 42, respectively.

[0037] Table 3 Comparison of this algorithm with other SSIM-based algorithms

[0038]

[0039] Table 3 continues the comparison between this algorithm and other SSIM-based algorithms.

[0040]

Claims

1. A compression artifact suppression method combining multi-level inter-frame correlation, characterized in that: (1) a multi-level motion compensation network is constructed for inter-frame motion compensation; The inter-frame correlation of forward and backward frames is used to perform multi-level motion compensation from global to local for a current frame; the motion compensation network adopts a four-layer motion compensation structure which is up-sampled by 8 times to a full image, and each layer outputs is represented as: the total output of the multi-level motion compensation network is represented as: (2) a multi-dimensional feature extraction and reconstruction network is constructed for feature processing: The network includes a multi-dimensional feature extraction part and a deep feature mapping and reconstruction part connected in series; The multi-dimensional feature extraction part processes the output obtained in step (1): first, the input feature map is dimensionally expanded, the average value of the same pixel position in different channels of the expanded feature map is obtained by average pooling, and the expanded feature map is weighted according to the average value; Then the channel-weighted feature map is subjected to average pooling and maximum value pooling within a single channel, spatial domain representative feature values are extracted and convolved, and the convolution result is multiplied by the feature map after the aforementioned channel weighting to realize pixel weighting in the spatial domain; The deep feature mapping and reconstruction part processes the feature map after multi-dimensional feature extraction: the deep feature mapping and reconstruction part includes a structure of seven layers of fusion residual learning and cross-layer dense connection; wherein, residual learning refers to adding the convolution result before the current convolution layer as the input of the current layer, and each layer of convolution structure is Convolution-ReLU with a convolution kernel size of 3x3; Cross-layer dense connection refers to concatenating the output feature maps of each of the seven convolution layers to realize the fusion of deep and shallow features; the fused feature map is input into the reconstruction part, which gradually reduces the dimension and reconstructs through two convolution layer structures, and outputs the video frame after removing artifacts; (3) the network is trained using a loss function: Objective function of network may be represented as: The loss function loss adopts a mean absolute error loss and a mean squared error loss combined strategy, denoted as: Wherein, λ1 and λ2 are the large and small weights of the balance respectively; Seven-frame sequences of Vimeo90K are selected as the data set for training the network; all video sequences are processed in accordance with the standards of ITU-R and BT.601, i.e. first, the continuous images are converted into YUV format video sequences, then the compressed video is obtained by HM16.0 compression, the compression configuration refers to the configuration file encoder_lowdelay_P_main.cfg, the QP is set to 37, the inter-frame encoding mode of IPPP is adopted, and finally the compressed video and the original video are converted into RGB domain images for input and effect observation of the experiment.

Images