Natural image de-lighting method based on cascaded feature attention pyramid network

Abstract

The present invention proposes an image de-lighting method based on a cascaded feature attention pyramid space network. The generative adversarial network architecture is adopted to construct a cascaded feature attention pyramid network, which overcomes the problem in the prior art that when repairing images with different noise intensities and types, the image repair effect is poor due to poor robustness of feature extraction, and improves the robustness of noise removal. The present invention introduces an attention mechanism into the cascaded feature attention pyramid network structure, which reduces the uncertainty of the focus area during image repair due to noise clutter, thereby reducing the blurring effect of the repair noise on the natural image and improving the repair quality of the natural image. In addition, the present invention fills the deficiency of the prior art that only focuses on common noises such as rain marks, haze blur and shadow occlusion, and expands the application scenarios of image denoising and repair applications.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and further relates to a natural image de-lighting method based on a cascaded feature attention pyramid network in the field of natural image restoration. The present invention can eliminate the astigmatism interference in photos or video images taken in lighting scenes, realize the restoration of natural images, and improve the quality of natural images. Background Art

[0002] In daily life and industrial production, with the widespread application of digital images, the requirements for image quality are becoming higher and higher. However, image imaging is often disturbed by various factors such as the external environment, resulting in interference noise in the image, reducing the visual quality of the image and information readability. Natural image restoration refers to the process of reconstructing lost or damaged parts of images and videos. By reconstructing the missing areas of the damaged image, the information contained in the image can be restored. At present, there are some related application technologies for the classic problems of image denoising in natural image restoration: traces of rain, haze blur, shadow occlusion, etc. However, there is no effective solution to the blur and occlusion problems caused by astigmatism and refraction, which are more common in natural images.

[0003] Liao Yongwei and other scholars proposed a method for repairing occluded targets in images based on generative adversarial networks in their paper "Multi-scale Generative Adversarial Network Image Restoration Algorithm Based on Multi-scale Fusion" (Computer Applications, 2023, 43(2):9). The specific implementation steps of this method are: the first step is to introduce a multi-feature fusion module in the generator to fuse the features of different dilated convolutions and effectively extract image features; the second step is to introduce the Euler distance based on feature variance as the perceptual feature matching loss function during the training process to facilitate the extraction of image edge information and enhance the structural consistency of the repaired image. The method proposed in this method has good performance under noise occlusion of simple-shaped images. However, the method still has the following shortcomings: the noise occlusion has a simple shape, but in the actual production and life process, the interference in the image is complex, and the occlusion size and shape are random and diverse. When the occlusion in the image is changed, the model trained by this method has poor generalization performance for noise, which leads to a decrease in the image restoration effect.

[0004] The University of Jinan proposed an image restoration method (image deraining) in a single scene in its patent application "Image Deraining Method and System Based on Densely Connected Deep Residual Network" (patent application number: CN202010705395.0, application publication number: CN111860003A). The specific implementation steps of this method are: the first step is to construct a densely connected deep residual network for rain mark extraction; the second step is to extract high-frequency information features from the acquired image to be processed; the third step is to input the extracted feature map into a pre-trained densely connected deep residual network to obtain an output rain mark image; the fourth step is to perform a difference process between the rain mark image and the image to be processed to obtain an image after removing the rain marks. This method achieves image deraining under simple conditions by combining high-pass filtering with deep learning, thereby improving image quality. However, this method still has some shortcomings: since this method only performs image restoration when the image contains a single noise (rain marks), in the actual image restoration work, the image will be affected by various noises such as haze, shadows, and astigmatism. If this method is used to overcome many problems such as haze, shadows, and astigmatism, the image restoration framework constructed by this method cannot eliminate different types of noise. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a natural image de-astigmatism method based on a cascaded feature attention pyramid network to solve the blur and occlusion problems caused by astigmatism refraction in natural images; and to repair other common noises in natural images, such as rain marks, haze blur, shadow occlusion and other problems.

[0006] The idea of ​​achieving the purpose of the present invention is that, in order to solve the interference problem caused by various noises such as astigmatism in natural images, the present invention adopts a generative adversarial network architecture, constructs a neural network that eliminates noises such as astigmatism in images, and realizes the restoration of natural images. Specifically, a cascaded feature attention pyramid is used as the backbone network of the generator, and more robust and rich semantic information and spatial information are extracted from images containing astigmatism noise, so as to retain as much image background detail information and high-frequency information as possible in the process of generating images without astigmatism, thereby improving the accuracy of natural image restoration and eliminating the interference of astigmatism noise. In addition, the adversarial training idea in the generative adversarial network is used to design a discriminator network to ensure that the generator model obtains stronger supervision, further improving the effect of natural image restoration.

[0007] The specific steps of the present invention include the following:

[0008] Step 1, build a cascaded feature attention pyramid space network consisting of a downsampling module, a mapping module, a first upsampling module, a two-branch attention module with two branches in parallel, and a second upsampling module connected in series in sequence;

[0009] The structure of the first branch in the dual-branch attention module is sequentially connected in series: the first convolution layer, the first pooling layer, the second pooling layer, the first addition layer, the first activation layer, the second convolution layer, and the second activation layer are connected in series to form the first branch; the number of convolution kernels of the first and second convolution layers is set to N / 8 and N respectively, the convolution kernel size is set to 1×1, the convolution kernel moving step is set to 1, the first activation layer adopts the ReLu activation function, and the second activation layer adopts the Sigmoid activation function, where N is the number of channels of the input feature map;

[0010] The structure of the second branch in the dual-branch attention module is sequentially connected in series: the first convolution layer, the second convolution layer, the third convolution layer, the first deconvolution layer, the second deconvolution layer, and the third deconvolution layer are connected in series to form the second branch; the number of convolution kernels of the first to third convolution layers is set to 64, 128, and 256 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1; the number of convolution kernels of the first to third deconvolution layers is set to 256, 128, and 64 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1;

[0011] Step 2, build a discriminator network consisting of two dual-scale spatial attention Markov discriminators and a decomposition discriminator in parallel;

[0012] Step 3: Connect the cascaded feature attention pyramid space network and the discriminator network in parallel to form a generative adversarial network;

[0013] Step 4: Generate training set:

[0014] Select at least 6000 three-channel color astigmatism images with a size of 256x 256 and their corresponding non-astigmatism images to form a training set;

[0015] Step 5: Train the Generative Adversarial Network:

[0016] The training set is input into the cascaded attention feature pyramid network, and the Adam optimizer is used to iteratively update all the weights of the network until the pyramid network loss function converges, and the predicted image without astigmatism and the predicted astigmatism spot image are output; the Adam optimizer is used to iteratively update all the weights of the discriminator network until the discriminator network loss function converges, and the trained cascaded attention feature pyramid network is obtained;

[0017] Step 6, repair the image:

[0018] The image to be de-astigmatized is input into the trained cascaded feature attention pyramid space network, and the result of image de-astigmatization is output using the saved network weights.

[0019] Compared with the prior art, the advantages of the present invention are as follows:

[0020] First, since the present invention proposes a technical solution for removing astigmatism from natural images, it fills the deficiency of the existing technology that only focuses on common noises such as rain traces, haze blur and shadow occlusion, so that the present invention expands the application scenarios of image denoising and restoration applications.

[0021] Second, since the present invention constructs a cascaded feature attention pyramid network, the network uses a cascaded feature pyramid to replace the traditional unidirectional convolution structure, thereby extracting more robust and rich semantic features and spatial features from multi-scale image information, overcoming the problem in the prior art that when repairing images with different noise intensities and types, the image restoration effect is poor due to poor feature extraction robustness. The present invention can reduce the influence of noise of different intensities and disturbance feature extraction effects, thereby improving the robustness of noise elimination.

[0022] Third, since the present invention introduces an attention mechanism into the cascaded feature attention pyramid network structure, it achieves more precise attention to the noise existing in natural images and captures the position information of different noises in the image in a complex background image. The present invention reduces the uncertainty of the focus area during image restoration due to noise clutter, thereby reducing the blurring effect of the restoration noise on the natural image and improving the restoration quality of the natural image.

[0023] Fourth, since the present invention uses a generative adversarial network including a generator and a discriminator, the generator and the discriminator are constantly learning through the game of competition between the two. Compared with the common one-way reasoning neural network architecture used in the prior art, the generative adversarial network is more suitable for image generation tasks. The neural network architecture selected by the present invention is more effective in generating noise-free images in natural image restoration tasks, further improving the quality of natural image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of the present invention;

[0025] Figure 2 This is a diagram of the cascaded feature attention pyramid space network architecture of the present invention;

[0026] Figure 3 It is a structural diagram of the dual-branch attention mechanism of the present invention;

[0027] Figure 4 It is a structural diagram of the dual-scale spatial attention Markov discriminator of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0029] Reference Figure 1 , the steps for implementing the present invention are further described.

[0030] This paper proposes a generative adversarial network to achieve the mapping from astigmatism images to non-astigmatism images. Like all generative adversarial networks, it consists of two parts, the generator and the discriminator. The generator module consists of a cascaded feature attention pyramid space network, and the generator is responsible for separating the astigmatism from the background image as much as possible. The discriminator module contains two dual-scale spatial attention Markov discriminators D 1 , D 2 and a factorized discriminator D C , responsible for judging whether the decomposed image is clean and whether the generated image is realistic enough.

[0031] Reference Figure 2 , the structure of the cascaded feature attention pyramid space network constructed by the present invention is further described.

[0032] Step 1: Build a cascaded feature attention pyramid space network consisting of a downsampling module, a mapping module, a first upsampling module, a dual-branch attention module with two branches in parallel, and a second upsampling module connected in series.

[0033] The downsampling module is composed of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, and the fifth convolution layer connected in series in sequence. The number of convolution kernels of the first to fifth convolution layers is set to 32, 64, 192, 1088, and 2080 respectively, the size of the convolution kernel is set to 3×3, and the convolution kernel moving step is set to 1. The outputs of the first to fifth convolution layers are used as features Figure 1 ,feature Figure 2 ,feature Figure 3 ,feature Figure 4 ,Feature Figure 5.

[0034] The mapping module is composed of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, and the fifth convolution layer in parallel. The number of convolution kernels of the first to fourth convolution layers is set to 256, the number of convolution kernels of the fifth convolution layer is set to 128, the size of the convolution kernel is set to 1×1, and the convolution kernel moving step is set to 1. Feature map 5 is convolved through the first layer to obtain feature map 6. Figure 4 After the second layer of convolution, we get feature map 7. Figure 3 After the third layer of convolution, we get feature map 8. Figure 2 After the fourth convolution layer, we get feature map 9. Figure 1 After the fifth layer of convolution, feature map 10 is obtained.

[0035] The first upsampling module is composed of a first upsampling layer, a second upsampling layer, and a third upsampling layer in parallel. The upsampling scale factors of the first to third upsampling layers are all set to 2, and the upsampling methods all adopt the nearest neighbor upsampling. Feature map 6 is updated through the first upsampling layer, feature map 7 is updated through the second upsampling layer, and feature map 8 is updated through the third upsampling layer.

[0036] Reference Figure 3 , the structure of the dual-branch attention module constructed by the present invention is further described.

[0037] The structure of the first branch in the dual-branch attention module is connected in series in sequence: the first convolution layer, the first pooling layer, the second pooling layer, the first addition layer, the first activation layer, the second convolution layer, and the second activation layer are connected in series to form the first branch; the number of convolution kernels of the first and second convolution layers are set to N / 8 and N respectively, the convolution kernel sizes are set to 1×1, the convolution kernel moving steps are set to 1, the first activation layer adopts the ReLu activation function, and the second activation layer adopts the Sigmoid activation function, where N is the number of channels of the input feature map.

[0038] The structure of the second branch in the dual-branch attention module is connected in series in sequence: the first convolution layer, the second convolution layer, the third convolution layer, the first deconvolution layer, the second deconvolution layer, and the third deconvolution layer are connected in series to form the second branch. The number of convolution kernels of the first to third convolution layers is set to 64, 128, and 256 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1. The number of convolution kernels of the first to third deconvolution layers is set to 256, 128, and 64 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1. Feature maps 6 to 10 are sequentially obtained through the dual-branch attention network. Figure 1 To 5.

[0039] The second upsampling module is composed of the first concatenation layer and the first addition layer in series. Figures 1 to 4 It is sent to the first splicing layer for splicing. The splicing result is the same as the attention Figure 4 Add it to attention map 5.

[0040] Step 2: Build a discriminator network consisting of two dual-scale spatial attention Markov discriminators in parallel with a decomposition discriminator.

[0041] Reference Figure 4 , the structure of the dual-scale spatial attention Markov discriminator constructed by the present invention is further described.

[0042] The dual-scale spatial attention Markov discriminator is composed of a first convolution layer, a first pooling layer, a second pooling layer, a first addition layer, a first activation layer, a second convolution layer, a second activation layer, a third convolution layer, a fourth convolution layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, and an output layer connected in series in sequence. The number of convolution kernels of the first to seventh convolution layers is set to N / 8, N, N, 64, 64, 64, 1 respectively, the size of the first and second convolution kernels is set to 1×1, the size of the third convolution kernel is set to 3×3, the size of the fourth to seventh convolution kernels is set to 4×4, the convolution kernel moving step is set to 1, the first activation layer adopts ReLu activation function, and the second activation layer adopts Sigmoid activation function, wherein N is the number of channels of the input feature map.

[0043] The decomposition discriminator is composed of a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, and an output layer connected in series in sequence. The number of convolutional kernels of the first to fourth convolutional layers is set to 64, 64, 64, and 1 respectively, the size of the convolutional kernel is set to 4×4, and the convolutional kernel moving step is set to 1.

[0044] Step 3: Connect the cascaded feature attention pyramid space network and the discriminator network in parallel to form a generative adversarial network.

[0045] Step 4: Generate training set.

[0046] At least 6000 three-channel color astigmatism images with a size of 256x 256 and their corresponding non-astigmatism images are selected to form a training set.

[0047] Step 5: Train the generative adversarial network.

[0048] The training set is input into the cascaded attention feature pyramid network, and the Adam optimizer is used to iteratively update all the weights of the network until the pyramid network loss function converges, and the predicted astigmatism-free image and the predicted astigmatism spot image are output; the Adam optimizer is used to iteratively update all the weights of the discriminator network until the discriminator network loss function converges, and the trained cascaded attention feature pyramid network is obtained.

[0049] The discriminator network is trained as follows.

[0050] In the first step, the denoised image obtained by the cascaded feature attention pyramid spatial network is sent to the first dual-scale spatial attention Markov discriminator to determine whether it is a non-astigmatism image, and the loss is calculated and updated iteratively.

[0051] In the second step, the noise-only image obtained by the cascaded feature attention pyramid spatial network is sent to the second dual-scale spatial attention Markov discriminator to determine whether it only contains astigmatism, and the loss is calculated and updated iteratively.

[0052] In the third step, the noise and de-astigmatism images obtained according to the cascaded feature attention pyramid space network are sent to the decomposition discriminator to determine whether the astigmatism and background images are separated cleanly, and the loss is calculated and updated iteratively.

[0053] The pyramid network loss function is as follows:

[0054]

[0055] Among them, x′ i represents the output image, x i represents the corresponding true value image, i represents the number of images, i=1,2,||·|| 1 Indicates the standard pixel level l 1 norm, ||x′ i -x i || 1 Denotes the output image x′ i and the corresponding ground truth image x i The distance between them, ||x 1 -x 2 || 1 Represents the real image x 1 and x 2 The distance between them, ||x′ 1 -x′ 2 || 1 Denotes the output image x′ 1 and x′ 2 The distance between ||x′ 1 -x 1 || 1 +||x′ 2 -x 2 || 1 Make the decomposed image similar to the corresponding real image by minimizing ||x 1 -x 2 || 1 and ||x′ 1 -x′ 2 || 1 The error between them, the generator makes the distance between the decomposed image pairs and the distance between the real image pairs as consistent as possible, so the generator G can be trained to minimize L on the entire dataset ME-Cross loss:

[0056] The discriminator network loss function includes a decomposition discriminator loss function and two identical dual-scale spatial attention Markov discriminator loss functions;

[0057]

[0058]

[0059] in, represents the two-scale spatial attention Markov discriminator loss function, represents the decomposition discriminator loss function, m represents the serial number of the dual-scale spatial attention Markov discriminator, m = 1, 2, E represents the expectation, p represents the distribution of the data, G represents the cascaded feature attention pyramid space network, D m represents the two-scale spatial attention Markov discriminator, D C represents the decomposed discriminator and y represents the true label.

[0060] Finally, the loss of the generator is minimized and the loss of the discriminator is maximized to achieve end-to-end training. The final objective function is:

[0061] Step 6: Repair the image.

[0062] The image to be de-astigmatized is input into the trained cascaded feature attention pyramid space network, and the result of image de-astigmatization is output using the saved network weights.

Claims

1. An image de-lightening method based on cascaded feature attention pyramid spatial network, It is characterized in that Constructing a cascaded feature attention pyramid space network; the steps of the method include the following: Step 1, build a cascaded feature attention pyramid space network consisting of a downsampling module, a mapping module, a first upsampling module, a two-branch attention module with two branches in parallel, and a second upsampling module connected in series in sequence; The structure of the first branch in the dual-branch attention module is sequentially connected in series: the first convolution layer, the first pooling layer, the second pooling layer, the first addition layer, the first activation layer, the second convolution layer, and the second activation layer are connected in series to form the first branch; the number of convolution kernels of the first and second convolution layers is set to N / 8 and N respectively, the convolution kernel size is set to 1×1, the convolution kernel moving step is set to 1, the first activation layer adopts the ReLu activation function, and the second activation layer adopts the Sigmoid activation function, where N is the number of channels of the input feature map; The structure of the second branch in the dual-branch attention module is sequentially connected in series: the first convolution layer, the second convolution layer, the third convolution layer, the first deconvolution layer, the second deconvolution layer, and the third deconvolution layer are connected in series to form the second branch; the number of convolution kernels of the first to third convolution layers is set to 64, 128, and 256 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1; the number of convolution kernels of the first to third deconvolution layers is set to 256, 128, and 64 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1; Step 2, build a discriminator network consisting of two dual-scale spatial attention Markov discriminators and a decomposition discriminator in parallel; Step 3: Connect the cascaded feature attention pyramid space network and the discriminator network in parallel to form a generative adversarial network; Step 4: Generate training set: Select at least 6000 three-channel color astigmatism images with a size of 256x 256 and their corresponding non-astigmatism images to form a training set; Step 5: Train the Generative Adversarial Network: The training set is input into the cascaded attention feature pyramid network, and the Adam optimizer is used to iteratively update all the weights of the network until the pyramid network loss function converges, and the predicted image without astigmatism and the predicted astigmatism spot image are output; the Adam optimizer is used to iteratively update all the weights of the discriminator network until the discriminator network loss function converges, and the trained cascaded attention feature pyramid network is obtained; Step 6, repair the image: The image to be de-astigmatized is input into the trained cascaded feature attention pyramid space network, and the result of image de-astigmatization is output using the saved network weights.

2. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The downsampling module described in step 1 is composed of the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the fifth convolutional layer connected in series in sequence. The number of convolution kernels of the first to fifth convolutional layers is set to 32, 64, 192, 1088, and 2080 respectively, the convolution kernel size is set to 3×3, and the convolution kernel moving step is set to 1.

3. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The mapping module described in step 1 is composed of the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, and the fifth convolution layer in parallel. The number of convolution kernels of the first to fourth convolution layers is set to 256, the number of convolution kernels of the fifth convolution layer is set to 128, the convolution kernel size is set to 1×1, and the convolution kernel moving step is set to 1.

4. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The first upsampling module described in step 1 is composed of a first upsampling layer, a second upsampling layer, and a third upsampling layer in parallel. The upsampling scale factors of the first to third upsampling layers are all set to 2, and the upsampling methods all adopt nearest neighbor upsampling.

5. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The second upsampling module described in step 1 is composed of a first concatenation layer and a first addition layer connected in series.

6. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The dual-scale spatial attention Markov discriminator described in step 2 is composed of a first convolutional layer, a first pooling layer, a second pooling layer, a first addition layer, a first activation layer, a second convolutional layer, a second activation layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and an output layer connected in series in sequence. The number of convolution kernels of the first to seventh convolutional layers is set to N / 8, N, N, 64, 64, 64, 1, respectively, the sizes of the first and second convolution kernels are set to 1×1, the size of the third convolution kernel is set to 3×3, the sizes of the fourth to seventh convolution kernels are set to 4×4, the convolution kernel moving step is set to 1, the first activation layer adopts ReLu activation function, and the second activation layer adopts Sigmoid activation function, where N is the number of channels of the input feature map.

7. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The decomposition discriminator described in step 2 is composed of the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer, and the output layer connected in series in sequence. The number of convolution kernels of the first to fourth convolutional layers is set to 64, 64, 64, and 1 respectively, the convolution kernel size is set to 4×4, and the convolution kernel moving step is set to 1.

8. The image de-lightening method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The specific steps for training the discriminator described in step 5 are as follows: In the first step, the denoised image obtained by the cascaded feature attention pyramid spatial network is sent to the first dual-scale spatial attention Markov discriminator to determine whether it is a non-astigmatism image, and the loss is calculated and updated iteratively; In the second step, the noise-only image obtained by the cascaded feature attention pyramid spatial network is sent to the second dual-scale spatial attention Markov discriminator to determine whether it only contains astigmatism, and the loss is calculated and updated iteratively; In the third step, the noise and de-astigmatism images obtained according to the cascaded feature attention pyramid space network are sent to the decomposition discriminator to determine whether the astigmatism and background images are separated cleanly, and the loss is calculated and updated iteratively.

9. The image de-lighting method of the cascaded feature attention pyramid spatial network according to claim 1, It is characterized in that The pyramid network loss function described in step 5 is as follows: Among them, x′ i represents the output image, x i represents the corresponding true value image, i represents the number of images, i=1,2,||·|| 1 Indicates the standard pixel level l 1 Norm.

10. The image de-lightening method of the cascaded feature attention pyramid spatial network according to claim 9, It is characterized in that The discriminator network loss function in step 5 includes a decomposition discriminator loss function and two identical dual-scale spatial attention Markov discriminator loss functions; in, represents the two-scale spatial attention Markov discriminator loss function, represents the decomposition discriminator loss function, m represents the serial number of the dual-scale spatial attention Markov discriminator, m = 1, 2, E represents the expectation, p(·) represents the distribution of the data, G represents the cascaded feature attention pyramid space network, D m represents the two-scale spatial attention Markov discriminator, D C represents the decomposed discriminator and y represents the true label.

Images