Image dehazing method and model based on prior generative adversarial network

By introducing a priori generation of adversarial network and HSV space loss functions in image defog removal, the long training time and poor effect in neural network defog removal method are solved, and high-quality image defog removal effect is achieved, with clear object contours. Accurate color and complete fog removal effect.

CN114998151BActive Publication Date: 2024-11-01TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210682052.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-15
Publication Date
2024-11-01
Estimated Expiration
2042-06-15

AI Technical Summary

Technical Problem

The existing neural network-based image defog removal method has problems such as long training time, unclear object contour recovery, distorted color, and incomplete fog removal, and lacks effective prior knowledge.

Method used

Using an image defogging method based on a priori generation adversarial network, the model is trained through an image generator and a discriminator, the structural features are extracted using the band-stop filtered image, and the priori loss is designed in the HSV space to achieve decoupling of the fogging image and compensation.

Benefits of technology

The image recovery quality is significantly improved, the object contour is clear and the color is not distorted, and the fog removal effect is close to the fog-free image, achieving the most advanced performance level.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114998151B_ABST
    Figure CN114998151B_ABST
Patent Text Reader

Abstract

The invention discloses an image defogging method based on a priori generative adversarial network, comprising: collecting paired foggy images and fog-free images to construct a training set; preprocessing the images of the training set to obtain a four-channel image, wherein the four channels include three channels of R, G, and B and a fourth channel retaining high-frequency and low-frequency information after filtering; taking shallow features of the four-channel foggy image and down-sampling to obtain deep features; up-sampling the deep features to obtain reconstructed up-sampled features, integrating the image features before up-sampling into the reconstructed up-sampled features through jump connections during the up-sampling process, and reconstructing a four-channel defogging image; training the image defogging model on the training set according to an image generator loss and a discriminator loss; the image generator loss includes a reconstruction loss between a foggy image and a defogging image, an adversarial loss, and a priori loss that characterizes the difference between the defogging image and the fog-free image in an HSV space.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to an image defogging method and model based on a priori generative adversarial network. Background Art

[0002] In weather conditions such as fog or haze, there are a large number of tiny suspended particles in the outdoor air that refract and scatter light, causing a significant decrease in the clarity and contrast of the images captured by the device, color shift, blurred edge contours, and obscuration of important information, causing serious visual interference. Dehazing the image can reduce these effects and optimize various downstream computer vision tasks.

[0003] Image defogging methods can be roughly divided into traditional methods based on the physical model of atmospheric scattering and methods based on neural networks. The defogging method based on the physical model mainly analyzes and constructs the physical model of haze degradation based on the atmospheric scattering formula, removes fog and haze without damaging the quality of the image through optimization, and also takes into account the enhancement of various depth of field images, that is, estimates the various parameters in the formula model to reversely derive the haze-free image. With the development of deep learning, neural networks have been introduced into image defogging algorithms. The defogging methods based on neural networks are mainly divided into physical model parameter estimation and end-to-end defogging. The former relies on the atmospheric scattering model and uses neural networks to estimate the parameters in the model; the end-to-end defogging method uses convolutional neural networks to generate feature images and generates haze-free images by up-sampling and down-sampling.

[0004] Traditional methods based on the physical model of atmospheric scattering can quickly remove fog from images and have a certain visual effect, but the color will change significantly and the image texture recovery is not obvious. In addition, this method is heavily dependent on the estimated parameters of the atmospheric scattering model, but many scene parameters cannot be accurately estimated, and it is difficult to achieve a good defogging effect overall.

[0005] However, there are still several problems with dehazing methods based on neural networks: (1) Most deep learning methods use convolutional neural networks to perform dehazing under supervision by upsampling and downsampling. Since the prior knowledge of traditional methods is completely abandoned, the object contours are not restored clearly, the colors are distorted, and the dehazing is not thorough. (2) Current methods based on convolutional neural networks rely too much on increasing the depth of the network, that is, enhancing the nonlinear ability of the network, which leads to high training costs, long training time, and limited dehazing effect.

[0006] In summary, the image dehazing method based on the atmospheric scattering model has weak adaptability and poor dehazing effect; the method based on deep learning lacks effective prior knowledge, resulting in slight flaws in the dehazing effect. Summary of the invention

[0007] The main purpose of the present invention is to overcome the shortcomings of the prior art and propose an image defogging method based on a priori generative adversarial networks to solve the problems of long training time, unclear object contour restoration, color distortion, and incomplete defogging in existing neural network-based defogging methods.

[0008] To achieve the above object, the present invention proposes the following technical solutions:

[0009] A method for image defogging based on a priori generative adversarial network, comprising the steps of training an image defogging model, wherein the image defogging model comprises an image generator and a discriminator, the image generator is used to process an input foggy image into a defogging image, and the discriminator is used to discriminate whether the defogging image is a fog-free image; the steps of training the image defogging model comprise: S1, collecting paired foggy images and fog-free images to form image pairs, using the image pairs to construct a training data set, and dividing the data set into a training set and a test set; S2, preprocessing the images of the training set to obtain a four-channel image, wherein the four channels include three channels of R, G, and B and a fourth channel retaining high-frequency and low-frequency information after filtering; S3, A training set is used to extract shallow features of the four-channel foggy image and perform downsampling to obtain deep features; S4, the deep features are upsampled to obtain reconstructed upsampled features, and the image features before upsampling are integrated into the reconstructed upsampled features through jump connections during the upsampling process to reconstruct a four-channel defogged image; S5, according to the image generator loss and the discriminator loss, the image defogging model is trained on the training set; wherein the image generator loss includes: the reconstruction loss between the foggy image and the defogged image, the adversarial loss, and the prior loss that characterizes the difference between the defogged image and the fog-free image in the HSV space; wherein the HSV space includes an H channel representing hue, an S channel representing saturation, and a V channel representing brightness.

[0010] Furthermore, the image defogging method also includes: using the test set to test the image defogging model that has been trained, and judging the defogging effect based on the test results, wherein the defogging effect is measured using two indicators: peak signal-to-noise ratio PSNR and structural similarity SSIM.

[0011] Further, step S2 specifically includes: S21, reading the grayscale image of the image in the training set and converting it to the frequency domain through Fourier transform; S22, filtering the grayscale image using a band-stop filter to obtain a fourth channel that retains the low-frequency and high-frequency information of the image; S23, splicing the fourth channel with the three channels of R, G, and B to form the four-channel image.

[0012] Furthermore, the band-stop filter has a bandwidth of 60 Hz and a distance from the center frequency of 25 Hz.

[0013] Furthermore, the reconstruction loss between the foggy image and the defogged image includes: a pixel-level reconstruction loss and a perceptual loss representing a semantic-level reconstruction loss, the pixel-level reconstruction loss is measured using an L1 loss, and the perceptual loss is measured using perceptual similarity and style similarity in a feature mapping space;

[0014] The pixel-level reconstruction loss is:

[0015]

[0016] Where N represents the number of samples processed by the image generator, is the four-channel foggy image of the input image generator, is the four-channel haze-free image, is the dehazed image of four channels output by the image generator;

[0017] The perceived loss is:

[0018]

[0019] Among them, φ(·) represents the mapping relationship of the 30th layer of the feature map vgg16 network, is the RGB input of the foggy image, is the RGB input of the haze-free image, x′ is the transpose of x, where x is a general symbol representing or

[0020] The adversarial loss is:

[0021]

[0022] Where M is the number of samples processed by the discriminator, is the output of the discriminator;

[0023] The prior loss is:

[0024]

[0025] in, and They represent the H channel and SV channel of the four-channel dehazed image output by the image generator in the HSV space. and They are the H channel and SV channel of the four-channel fog-free image in the HSV space. is the SV channel of the four-channel foggy image in the HSV space, and ε is a very small constant used to avoid the denominator being 0; S channels x representing input data x s and V channel x v Here x is a general symbol representing or

[0026] The image generator loss is where λ 1 ,λ 2 ,λ 3 ,λ 4 is the loss weight.

[0027] Furthermore, the discriminator loss is:

[0028]

[0029] Furthermore, the image defogging method also includes: S6, obtaining the image to be processed, and performing the preprocessing to obtain the corresponding four-channel image to be processed; S7, inputting the preprocessed image to be processed into the trained image defogging model, and outputting the corresponding four-channel defogging image.

[0030] On the other hand, the present invention also proposes an image defogging model based on a priori generative adversarial network, including: an image preprocessing module, used to preprocess the image into a four-channel image, the four channels including three channels of R, G, and B and a fourth channel that retains high-frequency and low-frequency information after filtering; an image generator, connected to the output end of the image preprocessing module, used to perform defogging on the input four-channel foggy image to output a four-channel defogged image; the defogging process includes: extracting shallow features of the four-channel foggy image and down-sampling it to obtain deep features; up-sampling the deep features to obtain reconstructed up-sampled features, and in the upsampling process, integrating the image features before upsampling into the reconstructed up-sampled features through jump connections to reconstruct a four-channel defogged image; a discriminator, connected to the output end of the image generator, used to discriminate whether the defogged image output by the image generator is a fog-free image.

[0031] Furthermore, the image generator includes: a first shallow feature extraction module, connected to the output end of the image preprocessing module, for extracting shallow features of the image; a downsampling module, connected to the output end of the shallow feature extraction module, for downsampling shallow features to obtain deep features; an upsampling module, connected to the output end of the downsampling module, for upsampling the deep features to obtain reconstructed upsampling features, and integrating the image features before upsampling into the reconstructed upsampling features through the jump connection; an output module, connected to the output end of the upsampling module, for outputting a four-channel dehazed image.

[0032] Furthermore, the discriminator receives the defogged image output by the image generator and generates a classification feature map of the defogged image, so as to determine whether the defogged image is a haze-free image according to the classification feature map.

[0033] Furthermore, the upsampling module, the downsampling module and the feature extraction module included in the discriminator all introduce a CBAM attention module based on the convolutional layer.

[0034] In addition, the present invention also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned image defogging method can be implemented.

[0035] The beneficial effects of the technical solution of the present invention are: on the one hand, the image after band-stop filtering is added to the network as the fourth channel of the image for training, so that the network model can better extract the structural features of the image and achieve decoupling at the structural level; on the other hand, the present invention utilizes the distortion of the foggy image in color, brightness, and saturation to convert the image from the RGB space to the HSV space, designs the prior loss in the HSV space, calculates the prior loss in the way of calculating the contrast loss, and uses the contrast regularization method to allow the image to achieve visual decoupling. The defogging method and model of the present invention, which decouples and compensates the foggy image based on the structural prior and the visual prior, can restore the foggy image to a high-quality fog-free image. It is proved through sufficient experiments and at multiple image evaluation standard levels that the designed four-channel input method can quickly improve the image restoration quality during training, the prior loss method can improve the final image performance, and achieves the most advanced performance (SOTA) on the public test data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a block diagram of an image defogging model based on a priori generative adversarial network according to an embodiment of the present invention;

[0037] Figure 2 It is a network structure diagram of an image generator in an image defogging model based on a priori generative adversarial network according to an embodiment of the present invention;

[0038] Figure 3 is a network structure diagram of a shallow feature extraction module in an image generator according to an embodiment of the present invention;

[0039] Figure 4 is a network structure diagram of a submodule 31 of a downsampling module in an image generator according to an embodiment of the present invention;

[0040] Figure 5 is a network structure diagram of a submodule 41 of an upsampling module in an image generator according to an embodiment of the present invention;

[0041] Figure 6 is a Dense Layer network structure diagram of a feature extraction module in an embodiment of the present invention;

[0042] Figure 7 is a network structure diagram of a discriminator in an embodiment of the present invention;

[0043] Figure 8 is a working principle diagram of a band-stop filter in an embodiment of the present invention;

[0044] Fig. 9 The defogging effect comparison between the image defogging method of the embodiment of the present invention and the existing image defogging method;

[0045] Fig.10 It is a grayscale image of a fog-free image and a foggy image in an embodiment of the present invention under RGB channels and a fourth channel retaining high frequencies and / or low frequencies;

[0046] Fig.11 It is a comparison diagram of the foggy image and the fog-free image in the HSV space and the RGB space according to the embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0048] At present, the defogging method based on convolutional neural network is too dependent on increasing the network depth, that is, enhancing the nonlinear ability of the network, and does not consider the coupling effect between the fog medium and the image background. It lacks effective compensation for the image, resulting in high training cost, long training time, and limited defogging effect. In view of this, the embodiment of the present invention proposes a method and model for decoupling and compensating foggy images based on structural priors and visual priors to restore high-quality fog-free images. The model is based on a lightweight PeleeNet (a lightweight network variant based on DenseNet) and is designed to use a priori knowledge decoupling generative adversarial network (GAN). The newly designed generative adversarial network eliminates the problem of numerous redundant structures of traditional GAN ​​networks.

[0049] Please refer to Figure 1, an embodiment of the present invention proposes an image defogging model based on a priori generative adversarial network, comprising: an image preprocessing module 1, an image generator and a discriminator 7. The image preprocessing module 1 can process the input image into a four-channel image, wherein the four channels refer to the three channels of R, G, and B and the fourth channel retaining high-frequency and low-frequency information after filtering. The image generator comprises a shallow feature extraction module 2, a downsampling module 3, an upsampling module 4 and an output module 5 connected in sequence. The shallow feature extraction module 2 is connected to the output end of the image preprocessing module 1, receives the preprocessed four-channel image and extracts shallow features from it. The downsampling module 3 is connected to the output end of the shallow feature extraction module 2, receives the shallow features and performs downsampling to generate deep features of the image. The upsampling module 4 is connected to the output end of the downsampling module 3, and is used to upsample the deep features to obtain reconstructed upsampling features. During this period, the output of the downsampling module is fused with the reconstructed upsampling features through multiple jump connections, and the four-channel defogging image is restored and output through the output module 5. The discriminator 7 is connected to the output end of the image generator, and is used to discriminate whether the defogging image output by the image generator is a fog-free image.

[0050] The image preprocessing module 1 processes the input image into a four-channel image by first reading the grayscale image, then converting the grayscale image into the frequency domain through Fourier transform, and setting the band-stop filter to a bandwidth of 60 Hz and a distance center frequency of 25 Hz. Figure 8 , use the above-set band-stop filter to filter the grayscale image to generate a fourth channel image, which retains the high-frequency and low-frequency information of the input image. Finally, the fourth channel image is spliced ​​with the three channels of R, G, and B to obtain a four-channel image RGBF HL .

[0051] refer to Figure 2 and Figure 3 The shallow feature extraction module 2 is used to perform preliminary feature extraction and extract shallow features of the image. In some embodiments, the network structure of the shallow feature extraction module 2 is as follows: Figure 3 As shown, it includes convolution unit 21, convolution unit 22, convolution unit 23, and convolution unit 27 connected in sequence from the input end to the output end, wherein the output of convolution unit 21 passes through pooling layer 24, and the output of pooling layer 24 is spliced ​​(merged) with the output of convolution unit 23 and then input into convolution unit 27. Among them, convolution units 21, 22, 23 and 27 have the same network structure, which are all composed of a convolution layer, a BN (regularization) layer and a Leaky ReLU layer connected in sequence.

[0052] refer to Figure 2, downsampling module 3 includes three submodules Group1, Group2, and Group3. Group1, Group2, and Group3 contain 4 layers of Dense Layer, 6 layers of Dense Layer, and 8 layers of Dense Layer respectively, and the convolution unit Bconv1 and the pooling layer Pooling1 are connected in sequence after the last layer of Dense Layer in each submodule. Figure 4 For example, submodule 31 (Group 1) includes n=4 Dense Layers D-1 to Dn connected in sequence, and Dense Layers D-1 to Dn have the same network structure, which together constitute a feature extraction module within a submodule. Dense Layer represents a layer of a densely connected convolutional network, such as Figure 6 As shown, a Dense Layer layer includes a convolution unit 61, a convolution unit 62, and a lightweight CBAM attention module 63. The input received by a Dense Layer layer passes through the convolution unit 62 on the one hand, and passes through the convolution unit 61 and the CBAM attention module on the other hand. The input received by a Dense Layer layer, the output of the convolution unit 62, and the output of the CBAM attention module 63 are spliced ​​into one data as the output 64 of the Dense Layer layer. The convolution unit 61 includes three identical sub-units, and the convolution unit 62 includes two identical sub-units. Each of the sub-units has the same network structure as the aforementioned convolution units 21, 22, 23 and 27, which will not be repeated here.

[0053] refer to Figure 2 , upsampling module 4 includes four submodules Group4, Group5, Group6, and Group7, and these four submodules have the same network structure, please refer to Figure 5 , taking the structure of one of the submodules 41 as an example, the submodule 41 is composed of a deconvolution layer 411 and five Dense Layers D-1 to D-5 connected in sequence, wherein the output of the deconvolution layer 411 is concatenated with the output of the submodule through a jump connection (cascade) to form a data 412, which is then input into the first Dense Layer D-1. Figure 2 and Figure 5The upsampling process of upsampling module 4 includes multiple jump connections. Specifically, these jump connections include: the output of the image preprocessing module is spliced ​​with the output of Group7, and then they enter the output module 5 together; the output of Group1 is spliced ​​with the output of Group6, and then they enter Group7 for convolution; the output of Group2 is spliced ​​with the output of Group5, and they enter Group6 for convolution together; the output of Group3 is spliced ​​with the output of Group4, and they enter Group5 for convolution together.

[0054] refer to Figure 2 , after the upsampling module 4 with skip connection and the output module 5, the reconstructed dehazed image 20 is output, wherein the output module 5 is composed of a convolution unit 51 and a convolution layer 52 connected in sequence. The convolution unit 5 has the same network structure as the aforementioned convolution units 21, 22, 23 and 27, which will not be described in detail here.

[0055] refer to Figure 1 and Figure 7 The discriminator 7 is connected to the output end of the image generator, receives the four-channel defogging image output by the generator, and generates a classification feature map of the defogging image to determine whether the defogging image is a non-fogging image according to the classification feature map. Figure 7 , the discriminator 7 includes a shallow feature extraction module 71 connected in sequence (which has the same network structure as the shallow feature extraction module 2, see Figure 3 ), pooling layer 72, feature extraction module 73, convolution layer 74, convolution layer 75, BN layer 76, convolution layer 77 and Sigmoid activation layer 78; wherein, convolution layer 74, convolution layer 75, BN layer 76 and convolution layer 77 are connected to form a module, which is used to restore the output of feature extraction module 73 to the size and size of the original image, and belongs to the fine-tuning at the end of deep upsampling. Among them, feature extraction module 73 is composed of 5 Dense Layers D-1 to D-5 connected, and these 5 DenseLayers have the same structure, and the structure is the same as the aforementioned Figure 4 (submodule 31 of the downsampling module), Figure 5 The Dense Layer structure in the submodule 41 of the upsampling module is the same, so in order to simplify the description, the same label is used. In addition, it can be seen that the lightweight CBAM attention module is introduced in the upsampling module, the downsampling module and the feature extraction module (composed of several Dense Layers connected) included in the discriminator. For details, please refer to Figure 6 , a CBAM attention module is introduced in each dense layer to enhance the model's collection of detailed features and the network's attention to channel information.

[0056] The embodiment of the present invention further provides an image defogging method based on a priori generative adversarial network, which is implemented based on the image defogging model of the aforementioned embodiment. First, the image defogging model is trained, and then the trained image defogging model is used to perform image defogging. By simply inputting the image to be processed into the trained image defogging model, a high-quality defogging image can be output.

[0057] Next, the training process of the image defogging model is described in detail, which includes steps S1 to S5:

[0058] S1. Dataset preparation: Collect pairs of foggy images and fog-free images to form image pairs and construct a dataset. The image content of the paired foggy images and fog-free images is the same, the difference is that at least part of the foggy image is blocked or blurred by the foggy medium, resulting in distortion in texture and color; the image file format is PNG, with the same resolution. For the dataset, it is divided into training set and test set in a ratio of 8:2.

[0059] S2. Dataset preprocessing: First, randomly crop the images of the training set (including foggy images and fog-free images) to a size of 224×224 and randomly flip them; then, read the grayscale image and convert it to the frequency domain using Fourier transform, setting the band-stop filter to a bandwidth of 60Hz and a distance of 25Hz from the center frequency. Use the band-stop filter set to perform band-stop filtering on the grayscale image, retain the high-frequency and low-frequency information of the image to generate a fourth channel image, and concatenate the RGB channel with the fourth channel to obtain a four-channel image RGBF HL .

[0060] S3. Input the four-channel foggy image in the training set processed by step S2 into the shallow feature extraction module, extract the shallow features of the four-channel foggy image, and then use the downsampling module to downsample to obtain the corresponding deep features.

[0061] S4. The deep features obtained in step S3 are then entered into the upsampling module for upsampling, during which the image features before upsampling are fused with the reconstructed upsampling features through multiple jump connections, and finally the four-channel dehazed image is restored. The multiple jump connections include: the output of the image preprocessing module is spliced ​​with the output of Group7, and then they are both entered into output module 5; the output of Group1 is spliced ​​with the output of Group6, and then they are both entered into Group7 for convolution; the output of Group2 is spliced ​​with the output of Group5, and they are both entered into Group6 for convolution, and the output of Group3 is spliced ​​with the output of Group4, and they are both entered into Group5 for convolution.

[0062] S5. Train the image defogging model on the training set according to the image generator loss and the discriminator loss. The image generator loss includes: the reconstruction loss between the foggy image and the defogged image, the adversarial loss, and the prior loss that characterizes the difference between the defogged image and the non-fogged image in the HSV space; the HSV space (H is hue, S is saturation, and V is value) includes an H channel representing hue, an S channel representing saturation, and a V channel representing value.

[0063] The reconstruction loss between the foggy image and the defogged image includes: pixel-level reconstruction loss and perceptual loss representing semantic-level reconstruction loss, the pixel-level reconstruction loss is measured using L1 loss, and the perceptual loss is measured using perceptual similarity and style similarity in feature mapping space; the details are as follows.

[0064] The pixel-level reconstruction loss is:

[0065]

[0066] Where N represents the number of samples processed by the image generator, is the four-channel foggy image of the input image generator, is the four-channel haze-free image, is the dehazed image of four channels output by the image generator;

[0067] The perceived loss is:

[0068]

[0069] Among them, φ(·) represents the mapping relationship of the 30th layer of the feature map vgg16 network, is the RGB input of the foggy image, is the RGB input of the haze-free image, x′ is the transpose of x, where x is a general symbol representing or

[0070] The adversarial loss is:

[0071]

[0072] Where M is the number of samples processed by the discriminator, is the output of the discriminator;

[0073] The prior loss is:

[0074]

[0075] in, and They represent the H channel and SV channel of the four-channel dehazed image output by the image generator in the HSV space. and They are the H channel and SV channel of the four-channel fog-free image in the HSV space. It is the SV channel of the four-channel foggy image in the HSV space. ε is a very small constant used to avoid the denominator being 0, and its value can be around e-9 to e-12. S channels x representing input data x s and V channel x v Here x is a general symbol representing or This loss function can compare the difference between the generator output image and the fog-free image in terms of hue, saturation and brightness. At the same time, contrast regularization is used to take the fog-free image as a positive example and the foggy image as a negative example, in order to increase the distance between the defogging image and the foggy image and reduce the gap between the defogging image and the fog-free image. Contrast regularization adopts the idea of ​​clustering method to enable the network to learn the coupling characteristics between the fog medium and the image background, and use negative examples to limit the feasible solution space.

[0076] The image generator loss is where λ 1 ,λ 2 ,λ 3 ,λ 4 is the loss weight.

[0077] The discriminator loss is:

[0078]

[0079] After obtaining the loss function of the image generator and the loss function of the discriminator, the image defogging model is trained on the training set by calculating the two losses. Adam is used as the optimizer, and the training parameters include: learning rate 0.0001, β 1 =0.9,β 2 =0.999, weight decay is 1×10 -4 ,(λ 1 ,λ 2 ,λ 3 ,λ 4 )=(1.0,0.5,1.0,0.5). When the loss of the image generator and the discriminator do not change, the training is completed, which means that the discriminator can no longer distinguish whether it is a reconstructed dehazed image or an original non-hazed image.

[0080] After training, the test set is used to test the image dehazing model, and the dehazing effect is judged according to the test results. The dehazing effect is measured by two indicators: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM). The value range of structural similarity (SSIM) is [-1, 1]. The larger the values ​​of PSNR and SSIM, the better the restoration effect of the dehazing image.

[0081] The effectiveness of the solution of the present invention is verified by two experiments below.

[0082] 1) Comparative experiment

[0083] The first experiment uses the RESIDE dataset, which includes three datasets (RESIDE-V0, RESIDE-Standard, RESIDE-β) with a total of 386,085 pairs of foggy and fog-free images as training sets, and tests are performed on the RESIDE SOTS dataset. The results are shown in Table 1. It can be seen that the image defogging method of the present invention has reached the highest level in both evaluation indicators and obtained the best performance (SOTA). Fig. 9 It can be seen that the defogging method of the present invention has a significant defogging effect compared with other existing defogging methods, and is close to a haze-free image.

[0084] Table 1 Performance comparison between the present invention and the existing method

[0085]

[0086]

[0087] 2) Ablation experiment

[0088] In order to better demonstrate that the structural prior and visual prior of the present invention have a decoupling effect in the dehazing level, two types of ablation experiments are used for illustration.

[0089] 2.1) Experimental demonstration of band-stop filter structure decoupling. The RESIDE dataset (ITS and OTS) contains 86,125 images in total, and each image is processed into four types of images: RGB image, RGBF H Image, RGBF L Image, RGBF HL Then, each image is trained for 20 epochs on the GAN (generative adversarial network) (i.e., image generator) designed by the present invention, and tested on the SOTS and NTIRE'18 datasets. The results are shown in Table 2.

[0090] Table 2 Comparison of the effects of different data processing

[0091]

[0092] Among them, RGB refers to RGB image, RGBF L It is the three channels of RGB plus the fourth channel F containing low-frequency information L The four-channel image, RGBF H It is the three channels of RGB plus the fourth channel F containing high-frequency information H The four-channel image, RGBF HL It is the three channels of RGB plus the fourth channel F containing low-frequency and high-frequency information HL The four-channel image is composed of high-frequency signals that retain the image's contour information, low-frequency signals that retain the image's texture and other internal information, and the band-stop filter signal is similar to the high-frequency signal, but retains more useful information. HL When , PSNR, SSIM, and FSIM (feature similarity) all achieved the highest level on the test set. Fig.10 , the data effect of adding high-frequency information or low-frequency information alone is also better than the original RGB input. Input is RGBF H The effect is better than that of RGBF input L The effect shows that the influence of fog on the image is concentrated on the contour and texture.

[0093] 2.2) Using the RESIDEe-V0 and RESIDE-β datasets, a total of 386,085 images were analyzed in RGBF format. HL The form is taken as input, and the loss functions are and (remove prior loss) for training:

[0094]

[0095] The test was conducted on the SOTS dataset, and the results are shown in Table 3:

[0096] Table 3 Comparison of the results of the two losses

[0097]

[0098] It can be seen that adding the prior loss to the loss function can improve the final dehazing and reconstruction effects of the model, with PSNR increased by 2.0db and SSIM increased by 0.031.

[0099] Fig.11The following is a comparison of foggy and fog-free images in HSV and RGB space. It can be seen that the visual effects of foggy and fog-free images in HSV space are more obvious than those in RGB space. The visual difference between foggy and fog-free images of the same content in RGB space is not as large as that in HSV space.

[0100] It can be seen that the image defogging method and model based on the prior generative adversarial network proposed in the embodiment of the present invention achieve decoupling at the structural level by adding the image after band-stop filtering as the fourth channel of the image to the network for training, and achieves decoupling at the visual level by converting the image from the RGB space to the HSV space to calculate the prior loss, which can restore the defogging image with clear object contours, undistorted colors and thorough defogging that is close to a haze-free image.

[0101] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. An image dehazing method based on prior generative adversarial network, It is characterized in that The method comprises the steps of training an image defogging model, wherein the image defogging model comprises an image generator and a discriminator, the image generator is used to process an input foggy image into a defogged image, and the discriminator is used to discriminate whether the defogged image is a fog-free image; The step of training the image defogging model comprises: S1. Collect pairs of foggy images and fog-free images to form image pairs, use the image pairs to construct a training data set, and divide the data set into a training set and a test set; S2, preprocessing the images of the training set to obtain a four-channel image, wherein the four channels include three channels of R, G, and B and a fourth channel that retains high-frequency and low-frequency information after filtering; S3, extracting shallow features of the four-channel foggy image from the preprocessed training set and performing downsampling to obtain deep features; S4, upsampling the deep features to obtain reconstructed upsampled features, and in the upsampling process, integrating the image features before upsampling into the reconstructed upsampled features through skip connections to reconstruct a four-channel dehazed image; S5. Train the image dehazing model on the training set according to the image generator loss and the discriminator loss; wherein the image generator loss includes: a reconstruction loss between the hazy image and the dehazed image, an adversarial loss, and a priori loss that characterizes the difference between the dehazed image and the haze-free image in the HSV space; wherein the HSV space includes an H channel representing hue, an S channel representing saturation, and a V channel representing brightness.

2. The image defogging method according to claim 1, It is characterized in that The method also includes: using the test set to test the image defogging model that has been trained, and judging the defogging effect according to the test result, wherein the defogging effect is measured by two indicators: peak signal-to-noise ratio PSNR and structural similarity SSIM.

3. The image defogging method according to claim 1, It is characterized in that Step S2 specifically includes: S21, read the grayscale image of the image in the training set and convert it to the frequency domain through Fourier transform; S22, filtering the grayscale image using a band-stop filter to obtain a fourth channel retaining low-frequency and high-frequency information of the image; S23, splicing the fourth channel with the three channels of R, G, and B to form the four-channel image.

4. The image defogging method according to claim 3, It is characterized in that The band-stop filter has a bandwidth of 60 Hz and a distance from the center frequency of 25 Hz.

5. The image defogging method according to claim 1, It is characterized in that The reconstruction loss between the foggy image and the defogged image includes: a pixel-level reconstruction loss and a perceptual loss representing a semantic-level reconstruction loss, wherein the pixel-level reconstruction loss is measured using an L1 loss, and the perceptual loss is measured using perceptual similarity and style similarity in a feature mapping space; The pixel-level reconstruction loss is: Where N represents the number of samples processed by the image generator, is the four-channel foggy image of the input image generator, is the four-channel haze-free image, is the dehazed image of four channels output by the image generator; The perceived loss is: Among them, φ(·) represents the mapping relationship of the 30th layer of the feature map vgg16 network, is the RGB input of the foggy image, is the RGB input of the haze-free image, x′ is the transpose of x, where x is a general symbol representing or The adversarial loss is: Where M is the number of samples processed by the discriminator, is the output of the discriminator; The prior loss is: in, and They represent the H channel and SV channel of the four-channel dehazed image output by the image generator in the HSV space. and They are the H channel and SV channel of the four-channel fog-free image in the HSV space. is the SV channel of the four-channel foggy image in the HSV space, and ε is a very small constant used to avoid the denominator being 0; S channels representing input data x s and V channel x v Here x is a general symbol representing or The image generator loss is where λ 1 ,λ 2 ,λ 3 ,λ 4 is the loss weight.

6. The image defogging method according to claim 5, It is characterized in that The discriminator loss is:

7. The image defogging method according to claim 1, It is characterized in that Also includes: S6, acquiring the image to be processed, and performing the preprocessing to obtain the corresponding four-channel image to be processed; S7: input the preprocessed image to be processed into the trained image defogging model, and output the corresponding four-channel defogging image.

8. An image dehazing model based on prior generative adversarial network, It is characterized in that include: An image preprocessing module, used for preprocessing an image into a four-channel image, wherein the four channels include three channels of R, G, and B and a fourth channel retaining high-frequency and low-frequency information after filtering; An image generator, connected to the output end of the image preprocessing module, for performing defogging processing on the input four-channel foggy image to output a four-channel defogging image; The defogging process includes: extracting shallow features of the four-channel foggy image and performing downsampling to obtain deep features; upsampling the deep features to obtain reconstructed upsampled features, and in the upsampling process, integrating the image features before upsampling into the reconstructed upsampled features through jump connections to reconstruct a four-channel defogging image; The discriminator is connected to the output end of the image generator and is used to discriminate whether the defogging image output by the image generator is a haze-free image.

9. The image defogging model as claimed in claim 8, It is characterized in that The image generator comprises: A first shallow feature extraction module, connected to the output end of the image preprocessing module, for extracting shallow features of the image; A downsampling module, connected to the output end of the shallow feature extraction module, for downsampling the shallow features to obtain deep features; An upsampling module, connected to the output end of the downsampling module, for upsampling the deep features to obtain reconstructed upsampling features, and integrating the image features before upsampling into the reconstructed upsampling features through the skip connection; The output module is connected to the output end of the upsampling module and is used to output a four-channel defogging image.

10. The image defogging model according to claim 9, It is characterized in that The discriminator receives the defogging image output by the image generator and generates a classification feature map of the defogging image, so as to determine whether the defogging image is a non-fog image according to the classification feature map.

11. The image defogging model according to claim 9, It is characterized in that The up-sampling module, the down-sampling module and the feature extraction module included in the discriminator all introduce the CBAM attention module based on the convolution layer.

12. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the image defogging method described in any one of claims 1 to 7 can be implemented.