A method for restoring a blurred image based on an improved generative adversarial network

Abstract

The application discloses a blurred image restoration method based on an improved generative adversarial network, establishes a generative adversarial network model, designs an improved generator network structure, adds a mixed dilated convolution module and a skip connection in the generator, expands the receptive field of the network and fuses image information of deep and shallow networks, and simultaneously replaces a residual block with a residual dense block; a network loss function is designed, an L1 loss term is added in an original content loss, an image pair composed of a blurred image and a corresponding clear image is used to train the network model; the blurred image is input into the trained network model, the blurred image is processed by using the generator, and the restoration of the blurred image is realized. The application can train the network model by designing the generative adversarial network structure and the loss function, using the dataset composed of the blurred image and the clear image, and using the trained generator model to restore the blurred image, and the image deblurring effect is stable, and the high-precision restoration requirement of the blurred image can be met.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image processing, and particularly relates to a blurred image restoration method based on an improved generative adversarial network. BACKGROUND

[0002] In recent years, with the continuous development of computer vision technology, non-contact optical measurement methods are increasingly widely used in the field of aerospace. However, during the exposure of the camera, due to factors such as camera shaking or rapid movement of the object being photographed, the image will be blurred to some extent, which will ultimately affect the subsequent image analysis and processing. Therefore, how to restore the blurred image to a clear image is a problem that needs to be solved.

[0003] In early blurred image processing schemes, the motion between the camera and the object to be measured is regarded as uniform linear motion, which simplifies the mathematical model of the image. Common deblurring methods include inverse filtering, Wiener filtering, and constrained least squares recovery. In actual applications, the reasons for image blurring are complex, the corresponding blur kernel form is unknown, and the mathematical model of the image is difficult to accurately establish. Therefore, traditional image restoration methods are difficult to use. In recent years, deep learning-based image blind restoration methods have emerged, which have achieved restoration processing of complex blurred images.

[0004] In 2022, CN115222609A discloses an underwater image restoration method and a generative adversarial network model training method based on a generative adversarial network model. A skip connection is added to the network, and an L1 loss term with a weight of 100 is added to the loss function, which enhances the detail feature extraction capability of the generator and avoids the loss of details in the downsampling process. However, when L1 loss is used as the optimization objective function in this method, the weight is too large, which will cause the quality of the generated image to decrease. In 2022, CN114359106A discloses a video blurred image restoration method based on an improved generative adversarial network. A weighted network is added to the original network to focus on the important area features in the image. However, the generator used in this method contains a Dropout network. Although this structure can reduce the overfitting phenomenon in the training process, it will also cause the quality of the restored image to decrease. At the same time, the adversarial loss function used in this method is the original GAN adversarial loss function, which will cause the gradient to disappear when the generator or discriminator model is too good.

[0005] In view of the problems of model collapse and poor robustness in existing blurred image restoration methods based on a generative adversarial network, it is necessary to research a blurred image restoration method based on an improved generative adversarial network. SUMMARY

[0006] The technical problem solved by the present application is to overcome the shortcomings of the above-mentioned methods. For the problem of blurred image blind restoration, a blurred image restoration method based on an improved generative adversarial network is invented.

[0007] The technical scheme of the present application:

[0008] A blurred image restoration method based on an improved generative adversarial network, first, a generative adversarial network model is established, the generative adversarial network model is composed of a generator and a discriminator, an improved generator network structure is designed, a hybrid dilated convolution module and a skip connection are added in the generator, the receptive field of the generator network is expanded and the image information of the deep and shallow networks is fused, at the same time, the original residual block is replaced by a residual dense block to improve the stability of network training; a network loss function is designed, an L1 loss term is added in the original content loss to reduce the difference between the generated image and the target image; the image pair composed of the blurred image and the corresponding clear image is used to train the generative adversarial network model; the blurred image is input into the trained generative adversarial network model, and the generator is used to process the blurred image to realize the restoration of the blurred image; the specific steps are as follows:

[0009] (1) Establishing a generative adversarial network model

[0010] A hybrid dilated convolution module and a skip connection are added in the original generator model of the generative adversarial network, and a residual dense block is replaced; the improved generator is composed of a down-sampling module, a hybrid dilated convolution module, a feature enhancement extraction module and an up-sampling module, wherein the down-sampling module is composed of a down-sampling convolution block, the hybrid dilated convolution module is composed of a dilated convolution block, the feature enhancement extraction module is composed of a residual dense block, and the up-sampling module is composed of a transposed convolution block, a convolution layer and an activation layer; a skip connection is added between the down-sampling and up-sampling modules; the discriminator is composed of a down-sampling convolution block;

[0011] (2) Designing a network loss function

[0012] The network loss function is divided into two parts: adversarial loss and content loss, the weight of the content loss is λ1, the content loss is composed of two parts: average absolute loss (L1 loss) and perceptual loss, and the weights of the two parts are λ2 and λ3 respectively:

[0013]

[0014] The calculation of the adversarial loss adopts WGAN-GP with gradient penalty term:

[0015]

[0016] Wherein, E is the expected value, P g is the set of generated clear images, P r is the set of real clear images, is the image set composed of the generated image and the real clear image, D(x) and respectively represent that the image data is from the set P x and the discrimination result obtained after passing through the discriminator D, x~P r and respectively represent that the image data is from the set P g , P r and λ is the weight of the gradient penalty term;

[0017] The L1 loss calculates the average deviation between the generated image and the clear image:

[0018]

[0019] wherein x and y are pixel coordinates of the image, I S is the real clear image, G(I B ) is the image generated after the blurred image I B passes through the generator G;

[0020] The perceptual loss is defined as the Euclidean distance between the feature maps obtained after the real image I S and the generated image G(I B ) pass through the pre-trained VGG19 network model:

[0021]

[0022] wherein W i,j and H i,j are the width and height of the feature map respectively, and φ i,j is the feature map obtained after the input image passes through the i-th max-pooling layer and the j-th convolution in the pre-trained VGG19 network model;

[0023] (3) Training the network model

[0024] The blurred image and the corresponding clear image with a size of 256*256 are used to train the established generative adversarial network model;

[0025] (4) Blurred image restoration

[0026] The trained generator is used to process the blurred image to complete the restoration of the blurred image.

[0027] The beneficial effects of the present application are: a generative adversarial network model is established, and a generator and a discriminator network model are designed; a mixed dilated convolution module and a skip connection are added in the generator to expand the receptive field of the network and fully fuse the image information of the deep and shallow networks, thereby relieving the problem of gradient disappearance in the training process, and at the same time, the original multiple residual blocks in the network are replaced by residual dense blocks, rich local features are extracted through dense connection, and the network training is more stable; a loss function of the network is designed, the loss function is divided into an adversarial loss and a content loss, an L1 loss term is added in the original content loss function, the difference between the generated image and the target image is reduced, and the image restoration effect is improved; the generative adversarial network model is trained by using a fuzzy image and a corresponding clear image pair; the trained network model is used to process the fuzzy image, and the fuzzy image is restored. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a fuzzy image restoration flowchart based on an improved generative adversarial network.

[0029] Figure 2 is a generative adversarial network model schematic diagram.

[0030] Figure 3 is a generator network structure schematic diagram.

[0031] Figure 4 is a down-sampling convolution block and a dilated convolution block structure schematic diagram.

[0032] Figure 5 is a residual dense block structure schematic diagram.

[0033] Figure 6 is a dense block structure schematic diagram.

[0034] Figure 7 is a transposed convolution block structure schematic diagram.

[0035] Figure 8 is a discriminator network structure schematic diagram.

[0036] Figure 9 is a training image pair schematic diagram, wherein: (a) - clear image, (b) - corresponding fuzzy image.

[0037] Figure 10 is a test fuzzy image.

[0038] Figure 11 is a fuzzy image restoration result. DETAILED DESCRIPTION

[0039] The embodiments of the present application will be described in detail below in combination with the drawings and technical solutions.

[0040] (1) Establishing a Generative Adversarial Network (GAN) model. The GAN consists of a generator and a discriminator. The generator comprises four parts: a downsampling module, a hybrid dilated convolutional module, a feature enhancement and extraction module, and an upsampling module. The downsampling module consists of three downsampling convolutional blocks. The first downsampling convolutional block has a kernel size of 7*7 and 64 kernels, normalized using InstanceNorm and activated by ReLU. The second downsampling convolutional block has a kernel size of 3*3 and 128 kernels, normalized using InstanceNorm and activated by ReLU. The third downsampling convolutional block has a kernel size of 3*3 and 256 kernels, normalized using InstanceNorm and activated by ReLU. The hybrid dilated convolutional module consists of three dilated convolutional blocks with dilation rates of 1, 2, and 5. All three dilated convolutional blocks have a kernel size of 3*3 and 256 kernels, and all use InstanceNorm for normalization. The feature enhancement and extraction module consists of three residual dense blocks, such as... Figure 5 As shown, the residual scaling factor β is 0.2, all convolutional kernels in the dense block are 3*3 in size, and there are 256 kernels in total. The activation function is LeakyReLU. The upsampling module consists of two transposed convolutional blocks, one convolutional layer, and one activation layer. The first transposed convolutional block has 128 kernels, and the second transposed convolutional block has 64 kernels. A skip connection is added between the output position of the first transposed convolutional block and the output position of the second convolutional block in the downsampling module. The convolutional layer has 7*7 kernels in size, and there are 3 kernels in total. The activation function is tanh. The discriminator consists of four downsampled convolutional blocks and one convolutional layer. The first downsampled convolutional block has 64 kernels of size 3*3 and uses the LeakyReLU activation function. The second downsampled convolutional block has 128 kernels of size 3*3 and uses the LeakyReLU activation function. The third downsampled convolutional block has 256 kernels of size 3*3 and uses the LeakyReLU activation function. The fourth downsampled convolutional block has 512 kernels of size 3*3 and uses the LeakyReLU activation function. The final convolutional layer has 1 kernel of size 3*3.

[0041] (2) Design the network loss function. The network loss function consists of two parts: adversarial loss and content loss. The content loss has a weight of 100 and is composed of mean absolute loss (L1 loss) and perceptual loss, both with a weight of 0.5. The gradient penalty term in the adversarial loss function has a weight of 10. In the perceptual loss function, the feature map obtained by passing the input image through the third max pooling layer and the third convolution in the pre-trained VGG19 network model is used for calculation. The width and height of the feature map are both 28.

[0042] (3) Training network model. The representative line laser light strip image in the field of non-contact optical measurement is used as training and testing data. The training data set of the network is composed of 185 pairs of fuzzy light strip images and corresponding clear light strip images, as shown in Figure 9

[0043] (4) Fuzzy image restoration. The trained generator model is used to process the fuzzy image. The tested fuzzy image is as shown in Figure 10 Figure 11 The clear image generated is as shown in

[0044] The improved generative adversarial network-based fuzzy image restoration method provided by the application can restore fuzzy images by training a generative adversarial network model, is stable in training, has strong adaptability, and can meet the high-precision restoration requirements of fuzzy images.​​

Claims

1. A method for restoring blurred images based on an improved generative adversarial network, characterized in that, First, a Generative Adversarial Network (GAN) model is established, consisting of a generator and a discriminator. An improved generator network structure is designed by adding hybrid dilated convolutional modules and skip connections to the generator, expanding its receptive field and fusing image information from deep and shallow networks. Simultaneously, the original residual blocks are replaced with residual dense blocks to improve the stability of network training. A network loss function is designed by adding an L1 loss term to the original content loss to reduce the difference between the generated and target images. The GAN model is then trained using image pairs consisting of a blurred image and its corresponding sharp image. Finally, the blurred image is input into the trained GAN model, and the generator processes the blurred image to achieve image restoration. The specific steps are as follows: (1) Establishing a generative adversarial network model The original generative adversarial network generator model is improved by adding a hybrid dilated convolutional module and skip connections, and replacing residual blocks with residual dense blocks. The improved generator consists of four parts: a downsampling module, a hybrid dilated convolutional module, a feature enhancement and extraction module, and an upsampling module. The downsampling module consists of downsampling convolutional blocks, the hybrid dilated convolutional module consists of dilated convolutional blocks, the feature enhancement and extraction module consists of residual dense blocks, and the upsampling module consists of transposed convolutional blocks, convolutional layers, and activation layers. Skip connections are added between the downsampling and upsampling modules. The discriminator consists of downsampled convolutional blocks; (2) Design the network loss function The network loss function consists of two parts: adversarial loss and content loss. The content loss has a weight of λ1 and is composed of two parts: the average absolute loss (L1 loss) and the perceptual loss, with weights of λ2 and λ3, respectively. The adversarial loss is calculated using WGAN-GP with a gradient penalty term: Where E is the expected value, P g P is the set of clear images generated. r A collection of real, clear images. To generate an image set composed of interpolated real-world sharp images, For image The discrimination result obtained after passing through discriminator D, D(x) and Image x and The discrimination result obtained after passing through discriminator D x~P r and These represent the image data originating from set P. g P r and λ is the weight of the gradient penalty term; L1 loss is used to calculate the average deviation between the generated image and the sharp image: Where x and y are the pixel coordinates of the image, I S For a truly clear image, G(I) B (I) is a blurred image B The image generated after passing through generator G; Perceptual loss is defined as the real image I S and generated image G(I) B The Euclidean distance between feature maps obtained after processing by the pre-trained VGG19 network model: Among them, W i,j and H i,j These represent the width and height of the feature map, φ. i,j The feature map obtained by passing the input image through the i-th max pooling layer and the j-th convolution in the pre-trained VGG19 network model; (3) Training the network model The generative adversarial network model is trained using image pairs consisting of a blurred image of size 256*256 and its corresponding sharp image. (4) Blurred image restoration The trained generator is used to process the blurred image and restore it.

Images