A noise removal method based on inter-frame information using GAN

Abstract

The present invention proposes a noise removal method based on inter-frame information using GAN. The method mainly involves a method of directly using the information of sequentially decoded adjacent frames to improve the quality of the current frame without distinguishing between intra-frame and inter-frame coding modes. The method utilizes the characteristic that adjacent frames of HEVC sequentially decoded frames have a correlation, and the compensated current frame and the current frame are fused and input into the quality enhancement part through a motion compensation network, and then the corresponding enhanced current frame is obtained after feature extraction, mapping, and image reconstruction. The method also utilizes the advantages of GAN, and designs its discriminator as an average truth discrimination, which gives a more accurate discrimination result and helps the parameter convergence of the generator. At the same time, the loss function characteristics of GAN are used to continuously approach the real result, and an image frame with compression artifacts removed that is more in line with subjective visual perception and objective evaluation indicators is obtained.

Description

Technical Field

[0001] The present invention relates to research on a video quality improvement method in the field of video coding, and in particular to a noise removal method based on inter-frame information using GAN. Background Art

[0002] Image quality improvement tasks are mainly divided into denoising, super-resolution, deblurring, etc. Many new methods have also been proven effective through experiments. In the early days, some scholars proposed a group-based sparse image restoration method. A single image super-resolution SRCNN method based on deep learning uses fewer convolutional layers to achieve super-resolution at a faster speed. At the same time, a similar image denoising network ARCNN was applied to JPEG images. A fast super-resolution network FSRCNN was proposed based on its original super-resolution network. By canceling bicubic interpolation and changing the feature extraction and nonlinear mapping modules, image super-resolution was accelerated to meet the growing requirements of real-time image super-resolution. A deep learning method of residual network (Residue Learning) has an important impact on image feature learning. Based on its residual structure, the gradient will not disappear as the network deepens, and the accuracy is constantly improving. This is because the previous data is continuously retained and added to each layer of convolution results as the input of the next layer during the forward propagation process, so that the gradient can be retained. VDSR is a neural network based on residual information learning to achieve image super-resolution. By directly adding the interpolated image to the output through a long shortcut, the network learns the residual value of the high-resolution image and the interpolated low-resolution image, reducing the difficulty of network training. Although this method is simple, the effect is very obvious. SRGAN applies generative adversarial networks to the field of image super-resolution. It adds the Loss function to the VGG pre-trained model to calculate the loss difference between the real image and the generated image, and achieves a certain improvement on the basis of the traditional loss function. A post-processing deep learning method for HEVC intra-coded frames (VRCNN) uses multi-size convolution kernels to extract features from HEVC compressed images by drawing on the ideas of ARCNN. Due to the existence of multi-size convolution kernels, it can extract information with a larger receptive field. Unlike ARCNN, it uses the combination of the output of multi-size convolution kernels in dimensions, which increases the number of extracted features while keeping the depth unchanged, reducing the difficulty of training. At the same time, using long shortcut to train residual images makes training relatively easy. A more effective denoising network DnCNN can achieve multi-task processing by training a single denoising model such as Gaussian noise, such as blind Gaussian denoising, SISR or JPEG deblocking. A deep learning method that increases the receptive field by performing standard convolution and dilated convolution DCSR with the same size convolution kernel to achieve single image super-resolution. Based on the characteristics of continuous correlation of video, STMC uses an optical flow estimation network to estimate video frame motion and proposes an end-to-end training continuous frame super-resolution, which combines the low-quality motion estimation information of the previous frame and the current frame with the previous high-quality motion estimation information to obtain the high-quality motion estimation information of the current frame to achieve quality improvement.The channel-focused image super-resolution network RCAN uses convolutional layers to not only learn image feature maps, but also add multipliers (channel statistics) to the feature maps on each channel to change (rescale) the feature map data size. A generative adversarial network ESRGAN that combines the Densenet and Resnet structures achieves image super-resolution. Combining CU size for noise removal and using partition-aware adaptive transformation neural network (ASN) takes advantage of different CU divisions under different CTUs to improve the quality of CU blocks of different sizes in a targeted manner. The neural network STMC based on spatial-temporal characteristics uses the correlation between the previous and next frames and the current frame to achieve super-resolution of the current frame through progressive motion compensation. By improving the input frame of STMC, that is, through the high-quality frame acquisition network to make the motion compensation frame of the current frame a PQF in the field, the multi-frame quality enhancement network MFQE obtains a better quality enhancement effect for the current frame. These methods improve the image in terms of objective evaluation indicators such as PSNR. Summary of the invention

[0003] As the amount of Internet information and the amount of data transmitted between terminals increase day by day, how to transmit data under limited bandwidth has become a problem worthy of attention. We are also exposed to more and more video data in our daily lives. It is estimated that by 2022, global video transmission will account for 80% of the entire Internet transmission. Nowadays, ultra-high-definition video has become a basic requirement for people's daily viewing, and 4K and even 8K videos are increasing, which has created a strong demand for efficient video encoding and decoding technology. However, the result of high compression rate is high distortion, so many related studies are mainly aimed at improving the quality of distorted image frames.

[0004] Through observation, we know that the current methods for removing HEVC compression artifacts are either unsatisfactory or use the video that is fully decoded by the decoding end for post-processing, that is, the information of the entire sequence after being fully decoded or at least a large part of the information is required, but there is no emphasis on a quality enhancement method based on the transmission order of the bitstream, that is, the processing is performed after the decoding of the current frame is completed, and the frame information after the current frame is not required, which has a certain real-time performance. In the present invention, we use the characteristics of the generative adversarial network to extract information from adjacent frames, and generate high-quality images with fewer compression artifacts through the quality enhancement network therein. The generator includes a motion compensation network and a quality enhancement part, and the discriminator includes two sub-discriminators, and the average truth discriminator is used to realize the specific discrimination between the generated image and the original image. Specifically, the following steps are included:

[0005] (1) Generate high-quality image networks;

[0006] (2) The generator contains a neural network that uses adjacent previous frames to perform motion compensation on the current frame. The network includes a three-layer motion estimation and compensation network from coarse to fine with downsampling of 8 times, 4 times, and 2 times respectively. Its various parts can be expressed as:

[0007]

[0008]

[0009]

[0010] The overall expression of the network can be expressed as:

[0011]

[0012] The generator in the present invention further includes a quality enhancement network after the motion compensation network, which uses the current frame obtained by the motion compensation network and the current frame obtained by HEVC decoding as input, first passes through a feature extraction part, which includes two parallel sub-extraction layers with the same connection status, wherein a multi-scale convolution layer is used to make compression artifacts of different sizes perceptible, and the means of dilated convolution is used to increase the perception area while reducing the parameter amount, so that the network can achieve faster convergence in the later training stage; all convolution layers in this part adopt the Convolution-ReLU structure, and the front and rear layers are three parallel convolution layers; then there is a non-fixed convolution kernel number feature mapping layer, which adopts the residual learning method, applies the non-fixed convolution kernel number strategy to each residual block intermediate layer, and then restores the number of convolution kernels required for residual learning, which is also the structure of Convolution-ReLU; the last layer of this part is a reconstruction layer, which adopts two convolution layers, both of which have the structure of Convolution-ReLU, and finally obtains an enhanced image frame;

[0013] (3) The objective function of the generative adversarial network in the present invention can be expressed as:

[0014]

[0015] The discriminator in the present invention adopts a fully convolutional network, which consists of two sub-discriminators and uses an average truth discriminator. The discriminator can be expressed as follows under the conditions of self-update and generator update:

[0016]

[0017]

[0018] In addition to the above discriminator results, the loss function in the present invention also introduces L1 loss function and L2 loss function, so the total loss function can be expressed as:

[0019]

[0020] (4) The seven-frame sequence of Vimeo90K was selected as the training set of the present invention; the data set was processed in accordance with the standards of ITU-R and BT.601, that is, the seven-frame sequence images were first converted into YUV and then compressed by HM16.0 to obtain a compressed video. The compression configuration referred to the configuration file encoder_lowdelay_P_main.cfg, the QP was set to 37, and the IPPP inter-frame coding method was adopted. Then, the compressed video and the original video were classified and converted into RGB domain images for experimental input and observation.

[0021] The advantages and beneficial technical effects of the present invention compared with the prior art are as follows:

[0022] (1) The present invention proposes a noise removal method based on HEVC inter-frame information using GAN, which extracts inter-frame motion information by utilizing the characteristics of inter-frame correlation and compensates the previous frame to obtain a motion compensated frame.

[0023] (2) The motion compensated frame and the current frame are input into the quality enhancement part, which uses multi-scale parallel dilated convolution to extract features. After that, it passes through seven layers of residual layers with non-fixed number of convolution kernels, and finally the obtained feature map is input into the reconstruction layer to obtain the restored image.

[0024] (3) Experimental results show that our algorithm can obtain good objective evaluation index and subjective evaluation index effects. It can achieve good results when QP is equal to 37 and 42. The PSNR performance is 0.55dB and 0.48dB higher than the standard HEVC average, respectively. The SSIM performance is 0.0097 and 0.0127 higher than the standard HEVC, respectively. The performance of the subjective evaluation index Perceptual Index (PI) is also good, reaching an average decrease of 0.21 and 0.40 respectively compared with the standard. It can be seen that the present invention can improve the subjective and objective effects and has a strong ability to improve image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the overall network framework of the noise removal method based on inter-frame information using GAN.

[0026] Figure 2 It is the feature extraction layer of the noise removal method based on inter-frame information using GAN.

[0027] Figure 3 Motion compensation network diagram for the noise removal method based on inter-frame information using GAN.

[0028] Figure 4The figure shows the visual effect comparison between the present invention and the HEVC standard. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field may make some non-essential improvements and adjustments to the present invention according to the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0030] (1) The present invention proposes a post-processing algorithm for improving the visual perception of HEVC compressed video based on GAN. To verify the effectiveness of the proposed algorithm, the proposed algorithm is used for the video compressed by the HEVC standard HM16.0 test model. The experiment of the present invention selects the IPPP standard inter-frame coding configuration file encoder_lowdelay_P_main.cfg based on LDP. A total of 16 standard test video sequences are selected from five resolution size videos of 416×240, 832×480, 1280×720, 1920×1080, and 2560×1600, and the quantization QP is 37 and 42 respectively. All experiments of the present invention are run using MATLAB R2016a and Pycharm 2020 tools on a CPU i5-2320@3.00GHz and a GPU Nvidia GeForce GTX 2080 environment.

[0031] (2) In this experiment, only the training set with QP=37 is used for training. The test sequence is blindly improved by using the coding mode that does not distinguish between intra-frame and inter-frame. The PI perceptual index represents the distance between the distribution of reconstructed video frames and the distribution of images in the training data set. The smaller the value, the better.

[0032] Table 1 describes the design of each layer of the generator in the present invention, where the convolution layer is represented by Conv, and the convolution step size is set to 1. There is residual learning in the feature mapping part, so the two filters are a combination. Since there are seven residual blocks, the description of the ninth to fifteenth residual layers is omitted by using "..." in the middle.

[0033] Table 1 Configuration of the algorithm generator G of the present invention

[0034]

[0035] Table 2 describes the design of the discriminator in the present invention, wherein the convolutional layer is represented by Conv, and the filter size, number of filters and step size of the convolutional layer are shown in the table.

[0036] Table 2 Configuration of the discriminator D of the algorithm of the present invention

[0037]

[0038] In order to further explore the visual perception improvement effect of the proposed GAN network, the algorithm proposed in the present invention is compared with the relevant HEVC standard compressed video in subjective indicators. Table 3 shows the test results based on the perceptual index PI on the standard HEVC test sequence.

[0039] Table 3 Results of the present invention and HEVC standard coding based on PI

[0040]

[0041] In addition to the subjective evaluation indicators, the objective effects of this algorithm and other HEVC quality improvement algorithms are also compared. The two commonly used evaluation indicators, PSNR and SSIM, are mainly used to compare the effects of the algorithm of the present invention and DCAD and DSCNN. All comparison results are based on the improvement values ​​of HEVC.

[0042] Table 4 Comparison of this algorithm with other algorithms based on PSNR (dB) and SSIM

[0043]

[0044] Table 4 Comparison of this algorithm with other algorithms based on PSNR (dB) and SSIM

[0045]

[0046]

Claims

1. A noise removal method based on inter-frame information using GAN, Features: (1) Feed the compressed video sequence into the generator to generate high-quality images: The generator contains a neural network that uses adjacent previous frames to perform motion compensation on the current frame. The network contains three layers of motion estimation and compensation networks from coarse to fine with downsampling of 8 times, 4 times, and 2 times respectively. Its parts can be expressed as: In the above formula They represent the motion estimation and compensation networks with downsampling of 8 times, 4 times, and 2 times, respectively. These three networks constitute the overall motion estimation and compensation network f est , Indicates the current low-quality frame. represents the adjacent previous frame, Represents the compensation frame obtained by the first two subnets of the motion compensation network, Represents the compensated frame obtained by the entire motion compensation network; The overall expression of the network can be expressed as: The generator also includes a quality enhancement network after the motion compensation network, which uses the current frame obtained by the motion compensation network and the current frame obtained by HEVC decoding as input, and first passes through a feature extraction part, which contains two parallel sub-extraction layers with the same connection status; all convolutional layers in this part adopt the Convolution-ReLU structure, and the front and back layers are three parallel convolutional layers; followed by a non-fixed convolution kernel number feature mapping layer, which adopts the residual learning method to apply the non-fixed convolution kernel number strategy to each residual block intermediate layer, and then restores the number of convolution kernels required for residual learning, which is also a Convolution-ReLU structure; the last layer of this part is the reconstruction layer, which uses two convolutional layers, both of which have the Convolution-ReLU structure, and finally obtains an enhanced image frame; (2) Send the enhanced image frame to the discriminator for adversarial training: The objective function of the generative adversarial network can be expressed as: In the above formula, N is the total number of frames, G is the capital letter of the generator, and G() represents the generator, which consists of a motion compensation network and a quality enhancement network. represents the high-quality i-th frame obtained by the generator, represents the original high-quality i-th frame, l G It represents the difference between the high-quality image generated by the generator and the original image. The specific calculation method is given in formula (8). represents the value of parameter θ that minimizes the sum in brackets; The discriminator in the method adopts a fully convolutional network, which consists of two sub-discriminators and an average truth discriminator. The discriminator can be expressed as follows under the conditions of self-update and generator update: Relativistic Average Discriminator (RaD) is a discriminator design used in generative adversarial networks. Formulas (6) and (7) are the default formulas of the discriminator, where C() represents the feature extraction function, E() represents the expected operation, represents an expectation operation, indicating that in the probability distribution (p x ,p y ) is the mean value of the function of the random variable (x,y) under f Represents the generated sample, which in this invention represents the high-quality frame obtained by the generator x r Represents a real sample, which in this invention represents the original high-quality frame In addition to the results of the above discriminator, the loss function of the method also introduces the L1 loss function and the L2 loss function. The L1 loss function is also called the mean absolute error (MAE), and the L2 loss function is also called the mean square error (MSE). Therefore, the total loss function can be expressed as: In the above formula, L1 , l L2 Represent the absolute error loss and mean square error loss between the generated high-quality frame and the original high-quality frame, respectively, 1 ,λ 2 , α 1 , α 2 Represents different error losses l L1 , l L2 The total error loss l G The different weights of the total loss function l G The inputs of the different loss errors in are the generated high-quality frames and the original high-quality frames, which is consistent with the l in the objective function of the generative adversarial network. G The inputs are the same, they are the same parameter; The seven-frame sequence of Vimeo90K is selected as the training set of the present invention for training: the data set is processed in accordance with the standards of ITU-R and BT.601, that is, the seven-frame sequence pictures are first converted into YUV and then compressed by HM16.0 to obtain a compressed video. The compression configuration refers to the configuration file encoder_lowdelay P main.cfg, the QP is set to 37, the IPPP inter-frame encoding method is adopted, and then the compressed video and the original video are classified and converted into RGB domain images.

2. In the noise removal method based on inter-frame information using GAN as described in claim 1, the parallel sub-extraction layer of the feature extraction part adopts multi-scale parallel dilated convolution.

3. According to the noise removal method based on inter-frame information using GAN as described in claim 1, the two sub-discriminators used by the average authenticity discriminator are both 9 layers.

Images