A meta-learning based compressed image post-processing method

Abstract

The application discloses a compressed image post-processing method based on meta learning, respectively constructs an image enhancement network and a compression degradation network, and uses deep learning to complete network pre-training on a large-scale data set; then, combining a meta learning method, meta data sets are constructed through the compression degradation network, internal and external circulation training of the image enhancement network under the MAML network is carried out, and the adaptability of the image enhancement network is improved. When facing unknown enhancement tasks, the quality enhancement effect can be improved by learning the characteristics of a small amount of new data and fine-tuning the parameters of the image enhancement network. The method can adapt to various application scenarios and improves the portability of the network.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to quality enhancement of reconstructed images, specifically to a post-processing method for compressed images based on meta-learning. Background Technology

[0002] With the development of network bandwidth, images and videos have become the main content of network transmission, and their proportion continues to grow. To reduce the high transmission and storage costs caused by the increasing demand for high-resolution images, image compression techniques are needed to reduce their bitrate to ensure smooth network transmission, while also maintaining a certain level of quality of experience (QoE) for end users. However, lossy image compression is always accompanied by undesirable artifacts such as blockiness, motion blur, and ringing, especially at high compression ratios or low bitrates, all of which can lead to an unpleasant visual experience for end users.

[0003] To alleviate this phenomenon, image post-processing techniques have been proposed. The goal of image post-processing is to improve the visual quality of compressed reconstructed images and approach the ideal image by eliminating as many coding effects as possible. That is, to estimate the ideal image X from the compressed reconstructed image y and improve its quality. Reeves and Kim first proposed eliminating block artifacts with spatially varied low-pass filters, and many adaptive filtering methods have subsequently emerged. Some classic image restoration algorithms, including constrained least squares, convex set projection, and maximum a posteriori probability estimation, have also been applied in post-processing methods. With the emergence of deep learning in various fields, methods for enhancing images using deep learning have also been developed. Ma et al. proposed MFRnet based on dense residual blocks, Schiopu proposed a post-processing method based on a multi-head self-attention mechanism, which can reduce the BD-rate by 14.66% compared to VVC, and Bayliss proposed a low-complexity filtering method through pruning.

[0004] These methods have achieved excellent results. However, they rarely utilize the internal information of the image itself. Instead, they learn the mapping relationship by stacking a large number of parameters, which results in low efficiency. In addition, these methods often lack strong generalization ability. When faced with test data not included in the training set, they can only achieve local optima, which greatly affects the quality enhancement effect. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a compressed image post-processing method based on meta-learning. This method utilizes both external datasets and internal image information, and can generate network parameters adapted to the current input with only a small amount of data for fine-tuning, achieving excellent results and overcoming the limitation of traditional deep learning methods that rely on large amounts of training data.

[0006] A compressed image post-processing method based on meta-learning specifically includes the following steps:

[0007] Step 1: Network Pre-training

[0008] s1.1 Construct an image enhancement network E and a compression degradation network D. The image enhancement network E is used to enhance the quality of the decoded image, and the compression degradation network D is used to perform low-pass filtering, blur filtering, JPEG compression, and feature estimation on the image.

[0009] s1.2 Decode the video image Using the original video image I as samples and as labels, a large-scale training set is constructed for the image augmentation network E; using the original video image I as samples, the video decoded image... As labels, a large-scale training set for the compressed degenerate network D is constructed.

[0010] s1.3. Using the large-scale training set constructed in s1.2, pre-train the image enhancement network E and the compression degradation network D respectively.

[0011] Step 2: Construction of the outer ring road training set

[0012] Using different data than in step one, decode the video image. Using the original video image I as samples and as labels, the outer loop training set of the image augmentation network E is constructed; using the original video image I as samples, the decoded video image... As labels, construct the outer loop training set of the compressed degenerate network D.

[0013] Step 3: Construction of the Inner Loop Metadataset

[0014] The samples in the training set of the outer loop of the image augmentation network E Input the pre-trained compressed degenerate network D from step one, and input the image output by compressed degenerate network D. As a sample, video decoded image As labels, construct the inner loop meta dataset of image augmentation network E.

[0015] Step 4: Network Update

[0016] s4.1 Using the inner loop metadata obtained in step 3, the image augmentation network E pre-trained in step 1 is trained in the inner loop using the MAML algorithm. Then, using the outer loop training set obtained in step 2, the image augmentation network E is trained in the outer loop using the MAML algorithm.

[0017] s4.2. Use the outer loop training set of the compressed degenerate network D obtained in step two to train the compressed degenerate network D and update the network parameters.

[0018] s4.3 Repeat steps two through four until the image enhancement network E and the compression degradation network D converge.

[0019] Step 5: Post-processing of compressed images

[0020] For the compressed and reconstructed image x, it is used as a label and input into the compressed degradation network D trained in step four. The image y output by the compressed degradation network D is used as the sample corresponding to the label. Then, image y is input into the image enhancement network E trained in step four, and the inner loop is trained according to the method in s4.1, updating the parameters of the image enhancement network E again. Finally, the compressed and reconstructed image x is input into the updated image enhancement network E to obtain the reconstructed image with enhanced quality, and then processed.

[0021] The present invention has the following beneficial effects:

[0022] Deep learning is used to pre-train image enhancement networks and compression degradation networks on large-scale datasets. Meta-learning methods are combined with compression degradation networks to construct meta-datasets to improve the adaptability of image enhancement networks. In actual testing tasks, good accuracy can be achieved on new tasks by learning and fine-tuning with a small amount of new data. This enables the network to adapt to various application scenarios and improves its portability. Attached Figure Description

[0023] Figure 1 This is a diagram of the VDSR network structure in Example 1;

[0024] Figure 2 This is a schematic diagram of the compression degradation network D in the embodiment;

[0025] Figure 3 This is a schematic diagram of the feature extraction method in the embodiment;

[0026] Figure 4 This is a schematic diagram of the attention module in the embodiment;

[0027] Figure 5 This is a schematic diagram of spatial attention operations in the embodiment;

[0028] Figure 6 This is a schematic diagram of the inner loop training of the image enhancement network E in the embodiment;

[0029] Figure 7 This is a schematic diagram of the outer loop training of the image enhancement network E in the embodiment;

[0030] Figure 8 This is a flowchart illustrating the post-processing of the compressed image in this embodiment;

[0031] Figure 9The compressed reconstructed image used in Example 1;

[0032] Figure 10 The image is the one enhanced by the pre-trained image enhancement network E in Example 1;

[0033] Figure 11 The image is the one enhanced by the meta-learning-based compressed image post-processing method in Example 1;

[0034] Figure 12 This is a diagram of the VRCNN network structure in Example 2. Detailed Implementation

[0035] The present invention will be further explained below with reference to the accompanying drawings;

[0036] Example 1

[0037] A compressed image post-processing method based on meta-learning specifically includes the following steps:

[0038] Step 1: Network Pre-training

[0039] s1.1, constructing as follows Figure 1 The VDSR network shown serves as the image enhancement network E. It performs 11 convolutions and ReLU activations on the decoded image, followed by another convolution to enhance its quality. The network is constructed as follows: Figure 2 The compression degradation network D shown sequentially performs low-pass filtering, blur filtering, and JPEG compression on the image, and then utilizes... Figure 3 The convolutional neural network shown performs feature estimation. The feature estimation includes three downsampling operations and three upsampling operations. Each downsampling operation is performed by four convolutions and one attention module, and each upsampling operation is performed by one attention module and one convolution. The result of the third downsampling operation is then processed by an attention module before undergoing the first upsampling operation. The result of the first upsampling operation is concatenated with the result of the second downsampling operation, and then processed by the second upsampling operation. The result of the second upsampling operation is then concatenated with the result of the first downsampling operation, and then processed by the third upsampling operation, outputting the feature extraction result. Figure 4 As shown, the attention module assigns weights to the input data twice. First, after layer normalization and four convolution operations, the input data is assigned weight a1, which is then concatenated with the original input data to obtain the first weight assignment result. Second, after layer normalization, four convolution operations, and spatial attention operations, the first weight assignment result is assigned weight a2, which is then concatenated with the first weight assignment result to output the final weight assignment result. Figure 5 As shown, the spatial attention operation performs average pooling and final pooling on the input image, followed by a convolution and a sigmoid operation, and then assigns weights to the input image.

[0040] s1.2 Decode the video image Using the original video image I as samples and as labels, a large-scale training set is constructed for the image augmentation network E; using the original video image I as samples, the video decoded image... As labels, a large-scale training set for the compressed degenerate network D is constructed.

[0041] s1.3. Using the large-scale training set constructed in s1.2, pre-train the image enhancement network E and the compression degradation network D respectively.

[0042] Step 2: Construction of the outer ring road training set

[0043] Using different data than in step one, decode the video image. Using the original video image I as samples and as labels, the outer loop training set of the image augmentation network E is constructed; using the original video image I as samples, the decoded video image... As labels, construct the outer loop training set of the compressed degenerate network D.

[0044] Step 3: Construction of the Inner Loop Metadataset

[0045] The samples in the training set of the outer loop of the image augmentation network E Input the pre-trained compressed degenerate network D from step one, and input the image output by compressed degenerate network D. As a sample, video decoded image As labels, construct the inner loop meta dataset of image augmentation network E.

[0046] Step 4: Network Update

[0047] s4.1, such as Figure 6 As shown, using the inner loop metadata obtained in step three, the MAML algorithm is used to train the image augmentation network E pre-trained in step one in the inner loop. The specific steps are as follows:

[0048] s4.1.1 Initialize the network parameters θ of the image augmentation network E and the inner loop learning rate α of the MAML algorithm.

[0049] s4.1.2, Randomly divide the data in the inner loop metadata set obtained in step three into batches, T 内 ={T1,T2,...,T i ,...T I};

[0050] s4.1.3. Input the batched data from s4.1.2 sequentially to train the image enhancement network E. Each batch of data T is input... i The network parameters will be updated once afterward. in Let L be the gradient of the loss function L of the image augmentation network E.

[0051] s4.2, such as Figure 7 As shown, using the outer loop training set obtained in step two, the image enhancement network E is trained using the MAML algorithm. The specific steps are as follows:

[0052] s4.2.1 Initialize the network parameters θ of the image augmentation network E and the outer learning rate β of the MAML algorithm.

[0053] s4.2.2, Randomly divide the data in the outer loop training set obtained in step two into batches, T 外 ={T1,T2,...,T l ,...T L Update the network parameters θ: During the network parameter update process in the outer loop, the updated network parameters are used as described in s4.1. Calculate the loss function.

[0054] s4.3. Use the outer loop training set of the compressed degenerate network D obtained in step two to train the compressed degenerate network D and update the network parameters.

[0055] s4.4 Repeat steps two through four until the image enhancement network E and the compression degradation network D converge.

[0056] Step 5: Post-processing of compressed images

[0057] like Figure 8 As shown, for the compressed and reconstructed image x, it is used as a label and input into the compressed degradation network D trained in step four. The image y output by the compressed degradation network D is used as the sample corresponding to the label. Then, image y is input into the image enhancement network E trained in step four, and the inner loop training is performed according to the method in s4.1, updating the parameters of the image enhancement network E again. Finally, the compressed and reconstructed image x is input into the updated image enhancement network E to obtain the reconstructed image with enhanced quality, and then processed.

[0058] To demonstrate the effectiveness of this method, this embodiment uses image data from the DIV2K dataset to complete the network training process described above, and randomly selects two reconstructed images for a comparative experiment on quality enhancement, such as... Figure 9 As shown, the PSNR of the reconstructed image is 32.79 dB and 34.00 dB. First, the image enhancement network E pre-trained in step one is used for quality enhancement, and the results are as follows. Figure 10 As shown, the enhanced image has a PSNR of 33.05 dB and 34.12 dB. Then... Figure 9The reconstructed image shown is enhanced using this method, and the results are shown in Figure 11. The PSNR of the enhanced image is 33.12 dB and 34.26 dB. It can be seen that this method can effectively improve image quality.

[0059] Example 2

[0060] This embodiment is based on embodiment 1, and uses, as follows Figure 12 The VRCNN network shown performs a convolution and ReLU activation on the input image, then divides it into two branches. A convolution and ReLU activation is then performed on each branch. The results of the two branches are concatenated, then divided into two branches again, and a convolution and ReLU activation are performed on each branch. Finally, the results of the two branches are concatenated and then a convolution and ReLU activation are performed again to achieve quality enhancement.

Claims

1. A meta-learning based compressed image post-processing method, characterized in that: Specifically comprising the following steps: Step one, network pre-training s1.1, construct an image enhancement network E for quality enhancement of the decoded image; construct a compression degradation network D for low-pass filtering, blur filtering, JPEG compression and feature estimation of the image; s1.2, decoding the video image As a sample, the video original image I is taken as a label to construct a large-scale training set of the image enhancement network E; taking the video original image I as a sample, the video decoding image As a label, a large-scale training set of the compression degradation network D is constructed; s1.3, using the large-scale training set constructed in s1.2, pre-training the image enhancement network E and the compression degradation network D respectively; Step two, outer loop training set construction using different data from step one, video decoding image As a sample, the video original image I is taken as a label to construct the outer loop training set of the image enhancement network E; taking the video original image I as a sample, the video decoding image as a label, the outer loop training set of the compression degradation network D is constructed; Step three, inner loop metadata set construction Decoding the video image in the outer loop training set of the image enhancement network E As a label, input the pre-trained compression degradation network D in step one, and get the output image As a sample, build the inner loop metadata set of the image enhancement network E; Step four, network update s4.1, using the inner loop metadata set obtained in step three, using MAML algorithm to perform inner loop training on the image enhancement network E pre-trained in step one, and then using the outer loop training set obtained in step two, using MAML algorithm to perform outer loop training on the image enhancement network E again, update the network parameters; s4.2, using the outer loop training set of the compression degradation network D obtained in step two to train the compression degradation network D, update the network parameters; s4.3, repeat steps two to four until the image enhancement network E and the compression degradation network D converge; Step five, compressed image post-processing For the compressed reconstructed image x, input it into the compression degradation network D trained in step four as a label, and get the output image y as the corresponding sample of the label, then input the sample y into the image enhancement network E trained in step four, and perform inner loop training according to the method of s4.1 to update the parameters of the image enhancement network E again; finally, input the compressed reconstructed image x into the image enhancement network E updated again to get the quality enhanced reconstructed image, and complete the post-processing.

2. The meta-learning based compressed image post-processing method of claim 1, wherein: The VDSR network or VRCNN network is used as the image enhancement network E.

3. The meta-learning based compressed image post-processing method of claim 1, wherein: The feature estimation part of the compression degradation network D includes 3 times of downsampling and 3 times of upsampling, each downsampling is completed by 4 times of convolution and 1 attention module, and each upsampling is completed by 1 attention module and 1 convolution; wherein the result of the third downsampling is input into an attention module, and then the first upsampling is performed; the result of the first upsampling is spliced with the result of the second downsampling, and then the second upsampling is performed; the result of the second upsampling is spliced with the result of the first downsampling, and then the third upsampling is performed, and the feature extraction result is output; The attention module gives the input data two times of weight assignment, the first time is to perform layer normalization and 4 times of convolution operation on the input data, and then assign weight a1, and then splice it with the original input data to get the first weight assignment result; the second time is to perform layer normalization, 4 times of convolution operation and spatial attention operation on the first weight assignment result, and then assign weight a2, and then splice it with the first weight assignment result to output the final weight assignment result; the spatial attention operation is performed on the input image, and then one convolution and one sigmoid operation are performed, and then the input image is assigned a weight value.

4. The meta-learning based compressed image post-processing method of claim 1, wherein: The steps of inner loop training of the image enhancement network E include: s4.1.1, initialize the network parameters θ of the image enhancement network E and the inner loop learning rate α of the MAML algorithm; s4.1.2, randomly divide the data in the inner loop metadata set obtained in step three into batches, T = {T1, T2,..., T i ,...T I} s4.1.3, inputting the data in batches after s4.1.2 in sequence, training the image enhancement network E, and updating the network parameters once every batch T i After that, the network parameters will be updated once, wherein is the gradient of the loss function L of the image enhancement network E.

5. The meta-learning based compressed image post-processing method of claim 4, wherein: The steps of the outer loop training of the image enhancement network E include: s4.2.1, initializing the network parameters θ of the image enhancement network E and the outer learning rate β of the MAML algorithm; s4.2.2, randomly divide the data in the outer loop training set obtained in step two into batches, T 外 = {T1, T2,..., T l ,...T L}; update the network parameters θ: In the network parameter update process of , use the updated network parameters of s4.1 to calculate the loss function.

Images