CycleGAN dehazing method based on attention mechanism

Abstract

The application discloses a CycleGAN defogging method based on an attention mechanism and belongs to the technical field of image defogging processing, solves the problems of image color distortion, target detail blur and the like, and the application firstly samples an input picture, extracts features through nine residual network blocks, connects each residual module through Senet, then gives weights according to the importance of channels, extracts features, restores through upsampling, and obtains a defogging image. Compared with other methods, after the image defogging processing through the method, the image has the advantages of color fidelity, target detail clarity and the like, meanwhile, the algorithm effectively alleviates the problems of image color distortion, target detail blur and the like, and has high scientific significance and practical value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a CycleGAN defogging method based on an attention mechanism. BACKGROUND

[0002] Fog and haze exist widely in nature. Water vapor near the ground in a humid environment is easy to combine with dust to become fine water droplets or condense into ice crystals under low temperature conditions. This fine water droplet and ice crystal and the air composition aerosol system is called fog. The aerosol system composed of nitric acid, sulfide and dust particles in the urban environment is called haze. In order to cope with the increasingly frequent fog and haze weather, the State Disaster Office identified the fog and haze weather as one of the natural disasters in 2014. Fog and haze disasters pose a severe challenge to the fields of traffic safety, aviation safety and even military safety in China. In the field of traffic safety, the visibility of the road monitoring and vehicle diversion system is reduced in fog and haze weather, and the low-definition image of fog will greatly interfere with the target detection effect. In the field of aviation safety, fog and haze will block the pilots' field of view during the take-off and landing stages of civil aviation aircraft, causing flight delays, diversion and even suspension of flights and other adverse conditions. In the field of military safety, fog and haze weather will cause occlusion in the visible light band, causing information loss, and affecting the accuracy of various observation satellites and guided anti-missile systems. Therefore, in order to ensure that the computer vision processing system meets the needs of traffic safety, aviation safety and military safety in China under fog and haze weather, it is of great practical significance to restore and defog the foggy image and improve the image definition and contrast.

[0003] With the expansion of the user base of mobile devices and portable computers and the substantial improvement of computer computing power, the fields of image processing, image style transfer, image content-based retrieval and classification, and image generation have gradually developed into a combination of multiple disciplines. The generative adversarial network (GAN) is an important technology in these fields. Currently, researchers have used GAN and attention mechanism (Senet) for defogging. Engin et al. used a recurrent neural adversarial network in the defogging problem to avoid the problem of few training samples. Xu Qin et al. combined channel attention and pixel attention in the defogging algorithm to give more weight to important features. Liang et al. proposed a strong baseline model SwinIR, which performs well in image super-resolution and denoising. Attention mechanism is a core technology widely used in natural language processing, statistical learning, image detection, speech recognition and other fields since the rapid development of deep learning. SUMMARY

[0004] Therefore, the patent of the application combines CycleGAN with an attention mechanism, and proposes a CycleGAN defogging method based on an attention mechanism. Compared with other methods, after image defogging processing using the method, the image has the advantages of color fidelity, clear target details, etc. At the same time, the algorithm effectively alleviates the problems of image color distortion and target detail blur, and has high scientific significance and practical value.

[0005] The technical scheme adopted by the patent of the application is as follows:

[0006] A CycleGAN defogging method based on an attention mechanism includes the following steps:

[0007] Step 1: Use a cycle generative adversarial network (CycleGAN) to generate high-quality paired images for network model training, combine loss function optimization, and generate a final network model;

[0008] Step 2: Based on the network model generated in step 1, input the real image and perform down-sampling operation, and extract the feature map of the real image after down-sampling operation through 9 residual network blocks;

[0009] Step 3: Use the attention mechanism (Squeeze-and-Excitation Networks, Senet) to connect between each residual network block, and give the channel weight according to the importance of the channel;

[0010] Step 4: Combine the channel weight to restore the feature map by up-sampling to obtain the defogging image.

[0011] The step 1 specifically includes the following steps:

[0012] 1.1: Use a cycle generative adversarial network (CycleGAN) to generate a high-quality paired image dataset, and use it for network model training, wherein the paired image dataset generation method is specifically:

[0013] The implementation method of single training data augmentation to paired dataset is equivalent to converting the image from the source domain X to the target domain Y to obtain the corresponding paired image, and the cycle generative adversarial network is to train a mapping G from the source domain X to the target domain Y, so that the image distribution of G(X) can be consistent with the image distribution Y using the adversarial loss.

[0014] 1.2: Improve and optimize the loss function, adjust the complete loss condition, and generate a final network model combined with the optimized loss function to realize image conversion.

[0015] The step 2 specifically includes the following steps:

[0016] 2.1: Based on the network model generated in step 1, input the real image and perform downsampling operation;

[0017] 2.2: The real image after the downsampling operation is extracted by 9 residual network blocks.

[0018] The step 3 specifically comprises the following steps:

[0019] 3.1: Connect each residual network block using attention mechanism;

[0020] 3.2: Perform Squeeze operation on the feature map extracted in step 2.2 to obtain the weight value that determines the importance of each channel feature, which has a global receptive field, and then perform global mean pooling processing for all channels to finally obtain a feature map with a global receptive field;

[0021] 3.3: Perform Excitation operation on the feature map with global receptive field, which reversely affects the feature map through fully connected layer, and obtains the importance of the channel by predicting the importance of the corresponding channel of the feature map, and gives weight to the channel.

[0022] The step 4 specifically comprises the following steps:

[0023] 4.1: Based on the channel with weight obtained in step 3.3, perform upsampling restoration operation on the feature map to output the obtained defogging image.

[0024] As described above, due to the adoption of the above technical solutions, the present application has the following advantages:

[0025] The present application combines CycleGAN defogging algorithm and attention mechanism, and proposes a CycleGAN defogging algorithm based on attention mechanism. After image defogging processing by the method, the image has the advantages of color fidelity, clear target details, etc. At the same time, the algorithm effectively alleviates the problems of image color distortion and target detail blur. BRIEF DESCRIPTION OF DRAWINGS

[0026] The present application will be illustrated by examples and with reference to the accompanying drawings, in which:

[0027] Figure 1 is the image defogging process schematic diagram of the present application;

[0028] Figure 2 is the Squeeze-and-Excitation module structure schematic diagram;

[0029] Figure 3 is the discriminant network structure schematic diagram;

[0030] Figure 4is a schematic diagram of a generation network after adding an attention mechanism;

[0031] Figure 5 is a comparison chart of dehazing experimental results of various algorithms; DETAILED DESCRIPTION

[0032] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0034] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0035] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0036] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0037] EMBODIMENT

[0038] As shown in the drawings, the CycleGAN dehazing algorithm based on the attention mechanism is disclosed in the embodiments of the present application, including the following steps: Figures 1-5 Step 1: using a cycle generative adversarial network (CycleGAN) to generate high-quality paired images for network model training, combining loss function optimization and generating a final network model;

[0039] The step 1 specifically includes the following steps:

[0040] 1.1: using a cycle generative adversarial network (CycleGAN) to generate a high-quality paired image dataset, and using it for network model training, wherein the paired image dataset generation method is specifically:

[0041]

[0042] ​The method of augmenting a single training data into a paired dataset is equivalent to transforming the image to be augmented from the source domain X to the target domain Y to obtain the corresponding paired images. The recurrent generative adversarial network is trained to transform the image from the source domain X to the target domain Y into a mapping G, so that the image distribution of G(X) can be consistent with the image distribution Y using adversarial loss.

[0043] 1.2: The loss function is improved and optimized, the complete loss condition is adjusted, and the final network model is generated by combining the optimized loss function to achieve image conversion. The optimization of the loss function is as follows:

[0044] (1) Countering losses:

[0045] During training, an adversarial loss is applied to two mapping functions. The generator G attempts to generate images G(x) similar to real images, while the discriminator DY aims to distinguish the generated sample images G(x) from real samples. The two compete with each other and grow together.

[0046]

[0047] (2) Cyclic consistency loss:

[0048] Adversarial loss alone cannot guarantee that the learned function can map a single input x to the desired output y. To further reduce the space of possible mapping functions, the learned mapping function should be cyclically consistent. For each image x from the domain X, the image translation loop should be able to bring X back to the original image. That is, x→G(x)→F(G(x))≈x represents forward consistency, and y→G(y)→F(G(y))≈y represents backward cyclic consistency.

[0049] The cycle consistency penalty is as follows:

[0050] L cyc (G, F) = E y～Pdata(y) [||G(F(y))-y||1]+E x～Pdata(x) [||F(G(x))-x||1] (2)

[0051] (3) Complete loss:

[0052] λ is the control weight, and the complete loss is as follows:

[0053] L(G, F, Dx, DY) = L GAN (G, D) Y (X, Y) + L GAN (F, D) x ,Y,X)+λL cyc (G, F) (3)

[0054] Step 2: Based on the network model generated in step 1, input the real image and perform down-sampling operation, and extract feature maps from the real image after down-sampling operation through 9 residual network blocks;

[0055] The step 2 specifically comprises the following steps:

[0056] 2.1: Based on the network model generated in step 1, input the real image and perform down-sampling operation;

[0057] 2.2: Extract feature maps from the real image after down-sampling operation through 9 residual network blocks.

[0058] Step 3: Use attention mechanism (Squeeze-and-Excitation Networks, Senet) to connect between each residual network block, and give weight to the channel according to the importance of the channel;

[0059] The step 3 specifically comprises the following steps:

[0060] 3.1: Use attention mechanism to connect each residual network block;

[0061] 3.2: Perform Squeeze operation on the feature maps extracted in step 2.2 to obtain the weight value that determines the importance of each channel feature, which has a global receptive field, and then perform global mean pooling processing on all channels to finally obtain a feature map with a global receptive field;

[0062] 3.3: Perform Excitation operation on the feature map with global receptive field, which reversely affects the feature map through fully connected layer, and obtains the importance weight value of the channel by predicting the importance of the corresponding channel, and then assigns the weight value to the corresponding channel.

[0063] Step 4: Combine the channel weight to restore the feature map by up-sampling to obtain the defogging image.

[0064] The step 4 specifically comprises the following steps:

[0065] 4.1: Based on the channel with weight obtained in step 3.3, perform up-sampling restoration operation on the feature map to output the obtained defogging image.

[0066] Example analysis:

[0067] The method herein is trained on the NTIRE 2018 dataset, which includes indoor and outdoor foggy images and corresponding real images in the training set, and randomly selects images in the dataset as test images. The CPU model is AMD EPYC 7642 48-Core Processor, the GPU model is GTX3090, and the GPU memory size is 24GB. The programming language is Python, version 3.7, and the framework uses Pytorch, version 1.7.0.

[0068] Comparative experiments are performed on the methods herein and CLANE, DCP, CycleGAN, and CycleGAN with added cbam dehazing algorithms, as shown in Figure 5 The comparison results are shown in (a) to (f) from left to right as the original image, the CLANE dehazing result, the DCP dehazing result, the CycleGAN dehazing result, and the CycleGAN with added cbam dehazing result. As can be seen from Figure 5 , the CLANE dehazing result has severe color distortion and poor detail recovery, and the fog in the upper half of the second image is still obvious. The DCP dehazing result has a dark overall color tone, which also severely affects the target details, and the fog in the upper half of the second image is still obvious. CycleGAN is better than the previous two in color tone and detail recovery, but still has a red color tone problem, such as the curtain part. CycleGAN with added cbam has severe color distortion, with an overall yellow color tone, and the fog in the second image is still obvious. The dehazing result of the method herein is color-true and close to the real situation, with good detail recovery, and the small bracket reflection part in the first image is clearly visible.

[0069] Comparative experiments are performed on the methods herein and CLANE, DCP, CycleGAN, and CycleGAN with added cbam dehazing algorithms, and two indicators, the peak signal-to-noise ratio (psnr) of the image and the structural similarity (ssim) of the image, are used to evaluate the processing results of each algorithm. The larger the psnr value, the smaller the difference between the clear image after dehazing and the real clear image without fog, and the better the dehazing effect of the algorithm. If the difference between the clear image after dehazing and the real clear image without fog is larger, the psnr value is smaller, and the dehazing effect of the algorithm is worse. The larger the ssim value, the smaller the distortion of the clear image after dehazing relative to the real clear image without fog, and the better the dehazing effect. If the difference is larger, the structural similarity value is smaller, and the dehazing effect is worse. From the comparative analysis in Table 1 and Table 2, the image dehazing effect using the algorithm herein is the best.

[0070] Table 1 psnr index

[0071]

[0072] Table 2 ssim index

[0073]

[0074] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0075] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An attention mechanism-based CycleGAN defogging method, characterized in that, The method comprises the following steps: Step 1: generating high-quality paired images for network model training by using a cycle generative adversarial network, combining loss function optimization, and generating a final network model; Step 2: inputting a real image and performing a down-sampling operation based on the network model generated in step 1, and extracting feature maps from the real image subjected to the down-sampling operation through nine residual network blocks; Step 3: connecting each residual network block using an attention mechanism, and assigning weights to the channels according to the importance of the channels; Step 4: restoring the feature maps by up-sampling in combination with the channel weights to obtain a defogged image.

2. The CycleGAN defogging method based on an attention mechanism according to claim 1, characterized in that, The step 3 specifically comprises the following steps: Step 3.1: connecting each residual network block using an attention mechanism; Step 3.2: performing a squeeze operation on the feature maps extracted in step 2.2 to obtain feature maps with a global receptive field; Step 3.3: performing an excitation operation on the feature maps with the global receptive field to obtain the importance weight values of the channels corresponding to the feature maps, and assigning weights to the channels.

Images