Image defogging method and system based on edge attention and multi-order differential loss

Abstract

The application discloses an image defogging method and system based on edge attention and multi-order differential loss, and the method comprises the following steps: acquiring a foggy image dataset, preprocessing the foggy image dataset, dividing the foggy image dataset into a thick fog image dataset and a thin fog image dataset, and taking the thin fog image dataset as an input image dataset; constructing a restoration network and a degradation network; jointly training the restoration network and the degradation network to obtain a trained restoration network and a trained degradation network; and inputting a to-be-tested image into the trained restoration network to obtain a defogging result. The foggy image and the fog-free image are processed by using a multi-order convolution template, the consistency of the defogging image with the real fog-free image in contrast and brightness and edge information is constrained, the clear image four-class discriminator is subjected to adversarial training with the restoration network, the effects of the restoration network and the degradation network are more remarkable, the performance of the network is improved, and the defogging effect is improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image dehazing method and system based on edge attention and multi-order differential loss. Background Technology

[0002] Currently, the image quality obtained by shooting in foggy environments is greatly reduced. The image contrast is reduced, the color fidelity is reduced, and edge information is missing. The overall visual effect of the image is poor. For widely used computer vision tasks, such degraded images are extremely disruptive and destructive.

[0003] The problem of single-image dehazing studies how to recover the original fog-free image given an input image that has been degraded by fog. Currently, dehazing algorithms are mainly divided into dehazing methods based on physical priors and dehazing methods based on deep learning.

[0004] He discovered from a large number of outdoor fog-free images that at least one color channel had very low pixel values ​​in non-sky areas, which he termed the dark channel. Based on this prior knowledge of the dark channel, he estimated the transfer rate map and global atmospheric light intensity from foggy images to recover fog-free, clear images. However, since the gray values ​​of the sky and fog are similar, the dark channel prior dehazing method cannot handle foggy images containing large areas of sky well. Zhu et al. found that the brightness and saturation of pixels in foggy images change drastically with fog concentration. Images in fog-free areas exhibit low brightness and high saturation, while in dense fog areas they exhibit high brightness and low saturation. As fog concentration increases, brightness gradually increases and saturation gradually decreases. Based on a color attenuation prior, they built a model to estimate depth of field using brightness and saturation, reconstructing a clear image. The above-mentioned physics-based prior dehazing methods rely on prior knowledge of the image, which is related to the actual application scenario. When the prior does not conform, the dehazing quality of the image is poor.

[0005] With the development of deep learning, dehazing methods based on deep learning have been widely studied. However, some dehazing methods do not pay attention to the edge information of the image during the dehazing process, resulting in poor image quality after dehazing and problems such as color cast and incomplete dehazing. At the same time, some dehazing methods rely on paired datasets, which are complex and difficult to obtain.

[0006] Deep learning-based dehazing methods are mainly divided into two categories. One category relies on atmospheric scattering models, using convolutional neural networks to learn the transmission map, and then using the model formula to dehaze a single image. DehazeNet inputs a hazy image into a pre-convolutional network and outputs the corresponding transmission rate map. It directly estimates the mapping relationship between the hazy image and the transmission rate end-to-end, reconstructs the clear image using the atmospheric scattering model, and proposes a new activation function, BReLU, to improve the dehazing quality. Boyun Li et al. proposed three joint sub-networks: a clear image estimation network, a transmission map estimation network, and an atmospheric light intensity estimation network. Inputting a hazy image, these three sub-networks produce the clear image, transmission rate map, and atmospheric light intensity, respectively. This method does not rely on paired datasets and uses an unsupervised approach to achieve image dehazing.

[0007] Another type of deep learning-based dehazing algorithm does not rely on atmospheric scattering models. It mainly extracts features through convolutional neural networks, learns the mapping between foggy images and clear, fog-free images from a large dataset, and builds an end-to-end dehazing network. It takes a foggy image as input and outputs a fog-free image, thus achieving dehazing of a single image.

[0008] GridDehazeNet proposes three modules: a preprocessing module, a backbone module, and a post-processing module. The preprocessing module can specifically enhance the input image, while the backbone network extracts multi-scale information from the image based on an attention mechanism, performing effective information exchange at different scales. The post-processing module is used to improve the quality of the dehazed image. The entire network performs well for uniform fog, but performs poorly for non-uniform fog removal. GCANet uses smoothed dilated convolution to reduce the grid effect caused by general dilated convolution, eliminates artifacts, and uses an additional gated subnetwork to fuse information from different scales of the image for image restoration. Dong et al. proposed an enhanced decoder that uses only reconstruction L1 loss to supervise the restoration of fog-free images. Hong et al. introduced an additional teacher network, using the information from positive samples / fog-free images extracted by the teacher network to guide the student network / dehazing network, but did not make good use of the information from negative samples / fog images.

[0009] Current deep learning-based dehazing methods are ineffective for non-uniform fog. They do not pay attention to the edge information of the image during the dehazing process, resulting in blurred image edges and difficulty in reconstructing details obscured by fog. In addition, they rely on paired datasets, which are complex and difficult to obtain. Summary of the Invention

[0010] To overcome the shortcomings and deficiencies of existing technologies, this invention provides an image dehazing method and system based on edge attention and multi-order differential loss. This invention reduces dependence on paired datasets through a semi-supervised approach, processes dehazed and haze-free images using multi-order convolutional templates, and constrains the consistency of the dehazed image with the real haze-free image in contrast, brightness, and edge information, thereby achieving clearer images. Figure 4 The classifier and the restoration network are trained adversarially to make the effects of the restoration and degradation networks more significant, improve the network performance, and enhance the dehazing effect.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] This invention provides an image dehazing method based on edge attention and multi-order differential loss, comprising the following steps:

[0013] Obtain the foggy image dataset, preprocess the foggy image dataset, divide the foggy image dataset into a dense fog image dataset and a light fog image dataset, and use the light fog image dataset as the input image dataset;

[0014] Construct restoration and degradation networks;

[0015] The restored network and the degenerate network are jointly trained to obtain the trained restored network and the degenerate network, specifically including:

[0016] The first stage of training: the hazy image is processed by the restoration network to obtain the first clear image, and the first clear image is processed by the degradation network to obtain the hazy image;

[0017] The second stage of training: the hazy image is processed by a degradation network to obtain a dense fog image, and the dense fog image is processed by a restoration network to obtain a second clear image;

[0018] Content loss is calculated for the first clear image, the second clear image, and the real haze-free image. Multi-order differential loss is calculated for the first clear image, the second clear image, and the real haze-free image. Cycle consistency loss is calculated for the hazy image and the hazy image. The first clear image is then input into the clear image... Figure 4 Adversarial loss is calculated in the classification discriminator, clearly... Figure 4 The classification discriminator and the restoration network are trained adversarially by inputting dense fog images into the dense fog network. Figure 4 Adversarial loss is calculated in the classification discriminator, in dense fog. Figure 4 The classification discriminator is trained adversarially against the degenerate network;

[0019] The total loss is obtained by weighted summation of content loss, adversarial loss, cycle consistency loss, and multi-order differential loss, and the network parameters are updated using the gradient descent algorithm.

[0020] The image to be tested is input into the trained restoration network to obtain the dehazing result.

[0021] As a preferred technical solution, preprocessing of the hazy image dataset includes:

[0022] Measure the FADE fog density index of all fog images in the fog image dataset, calculate the mean of all FADE fog density indices, classify images with values ​​greater than the mean as dense fog images, and classify images with values ​​less than the mean as light fog images.

[0023] As a preferred technical solution, the restoration network includes an edge attention branch and a super-resolution restoration branch;

[0024] The edge attention branch adopts an encoding-decoding structure. In the encoding structure, a pre-trained network Res2Net is used to extract features. In the decoding structure, multiple edge attention modules are included, and each attention module includes two edge attention layers.

[0025] The super-resolution restoration branch adopts an encoding and decoding structure. In the encoding structure, a pre-trained network Res2Net is used to extract features, and the decoding structure includes multiple residual blocks for extracting image features and restoring lost image information.

[0026] The image features output by the edge attention branch and the super-resolution restoration branch are sequentially processed through progressive compression blocks, mirror padding, convolutional layers, and the Tanh activation function to obtain a restored clear image;

[0027] The network structure of the degraded network is the same as that of the edge attention branch in the restored network.

[0028] As a preferred technical solution, the first and second phases of training are carried out simultaneously, with the first phase being supervised learning and the second phase being unsupervised learning.

[0029] As a preferred technical solution, content loss is calculated by comparing the first clear image, the second clear image, and the real haze-free image, specifically as follows:

[0030]

[0031] Among them, L con For content loss, N is the total number of images, and J is the content loss. gt b represents the true, fog-free image, and e represents the first clear image.

[0032] As a preferred technical solution, a multi-order differential loss calculation is performed on the first clear image, the second clear image, and the real fog-free image, specifically expressed as follows:

[0033]

[0034] Among them, L frac For multi-order differential loss, N is the total number of images, v is the order, and F... v Let J be a differential convolution of order v, where V is a list of orders, and J is a differential convolution of order v. gt b represents the true, fog-free image, and e represents the first clear image.

[0035] As a preferred technical solution, the cycle consistency loss is calculated for the foggy image and the hazy image, specifically expressed as follows:

[0036]

[0037] Where N is the total number of images, a is the input image, and c is the foggy image.

[0038] As a preferred technical solution, the first clear image is input into the clear image. Figure 4 Adversarial loss is calculated in the classification discriminator, clearly... Figure 4 The classification discriminator and the restoration network are trained adversarially by inputting dense fog images into the dense fog network. Figure 4 Adversarial loss is calculated in the classification discriminator, in dense fog. Figure 4 The classifier and the degenerate network are trained adversarially, specifically as follows:

[0039]

[0040]

[0041] L gan (RNet,D1)=CE(D1(b),C gt )

[0042]

[0043] in, For clear training Figure 4 The loss of the classification discriminator, For training in dense fog Figure 4 The loss of the classifier, L gan (RNet, D1) represents the restored network RNet and the clear network D1. Figure 4 Adversarial loss of classifier D1, L gan (DNet, D2) represents the degenerate network DNet and the dense fog network. Figure 4 Adversarial loss of classifier D2, L adv Let J be the total adversarial loss, CE be the cross-entropy loss, and J be the total adversarial loss. gt For a truly fog-free image, I dense-gtThe images are real fog images, where a is the input fog image, b is the first clear image, d is the fog image, and C is the real fog image. gt C represents the category of a truly fog-free image. a C represents the category of the input hazy image. dense-gt C represents the category of a real fog image. RNet C represents the category of sharp images generated by the restoration network RNet. DNet This indicates the category of the fog map generated by the degenerate network DNet.

[0044] The present invention also provides an image dehazing system based on edge attention and multi-order differential loss, comprising: an image acquisition module, an image preprocessing module, a network construction module, a network training module, and a dehazing module;

[0045] The image acquisition module is used to acquire a dataset of foggy images;

[0046] The image preprocessing module is used to preprocess the foggy image dataset, dividing the foggy image dataset into a dense fog image dataset and a light fog image dataset, and using the light fog image dataset as the input image dataset.

[0047] The network construction module is used to construct restoration networks and degradation networks;

[0048] The network training module is used to jointly train the restoration network and the degradation network to obtain the trained restoration network and degradation network, specifically including:

[0049] The first stage of training: the hazy image is processed by the restoration network to obtain the first clear image, and the first clear image is processed by the degradation network to obtain the hazy image;

[0050] The second stage of training: the hazy image is processed by a degradation network to obtain a dense fog image, and the dense fog image is processed by a restoration network to obtain a second clear image;

[0051] Content loss is calculated for the first clear image, the second clear image, and the real haze-free image. Multi-order differential loss is calculated for the first clear image, the second clear image, and the real haze-free image. Cycle consistency loss is calculated for the hazy image and the hazy image. The first clear image is then input into the clear image... Figure 4 Adversarial loss is calculated in the classification discriminator, clearly... Figure 4 The classification discriminator and the restoration network are trained adversarially by inputting dense fog images into the dense fog network. Figure 4 Adversarial loss is calculated in the classification discriminator, in dense fog. Figure 4 The classification discriminator is trained adversarially against the degenerate network;

[0052] The total loss is obtained by weighted summation of content loss, adversarial loss, cycle consistency loss, and multi-order differential loss, and the network parameters are updated using the gradient descent algorithm.

[0053] The dehazing module is used to input the image to be tested into the trained restoration network to obtain the dehazing result.

[0054] As a preferred technical solution, content loss is calculated by comparing the first clear image, the second clear image, and the real haze-free image, specifically as follows:

[0055]

[0056] Among them, L con For content loss, N is the total number of images, and J is the content loss. gt b represents the true, fog-free image, and e represents the first clear image.

[0057] The first clear image, the second clear image, and the true haze-free image are subjected to multi-order differential loss calculation, specifically expressed as follows:

[0058]

[0059] Among them, L frac For multi-order differential loss, v is the order, F v For v-order differential convolution, where V is a list of orders;

[0060] The cycle consistency loss is calculated for both foggy and lightly foggy images, specifically as follows:

[0061]

[0062] Where a is the input image and c is the foggy image;

[0063] Input the first clear image into the clear image. Figure 4 Adversarial loss is calculated in the classification discriminator, clearly... Figure 4 The classification discriminator and the restoration network are trained adversarially by inputting dense fog images into the dense fog network. Figure 4 Adversarial loss is calculated in the classification discriminator, in dense fog. Figure 4 The classifier and the degenerate network are trained adversarially, specifically as follows:

[0064]

[0065]

[0066] L gan (RNet,D1)=CE(D1(b),C gt )

[0067]

[0068] in, For clear training Figure 4 The loss of the classification discriminator, For training in dense fog Figure 4 The loss of the classifier, L gan (RNet, D1) represents the restored network RNet and the clear network D1. Figure 4 Adversarial loss of classifier D1, L gan (DNet, D2) represents the degenerate network DNet and the dense fog network. Figure 4 Adversarial loss of classifier D2, L adv Let J be the total adversarial loss, CE be the cross-entropy loss, and J be the total adversarial loss. gt For a truly fog-free image, I dense-gt d represents a real fog image, and C represents the fog image. gt C represents the category of a truly fog-free image. a C represents the category of the input hazy image. dense-gt C represents the category of a real fog image. RNet C represents the category of sharp images generated by the restoration network RNet. DNet This indicates the category of the fog map generated by the degenerate network DNet.

[0069] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0070] (1) This invention constructs a recurrent dual model, in which the dehazing network, i.e. the restoration network, adopts a dual-branch structure design. An edge attention module is designed in the edge attention branch. By learning the weights of image edges, the network's perception of image edge information is enhanced, and the network's ability to restore edge information is improved. The super-resolution restoration branch is used to reconstruct lost details. The fogging network, i.e. the degradation network, learns images with different degradation levels, and ultimately improves the robustness and performance of the restoration network. During training, the restoration-degradation process is supervised learning, and the degradation-restoration process is unsupervised learning. The dependence on paired datasets is reduced through semi-supervised learning.

[0071] (2) Existing loss functions often use L1 loss to constrain the dehazed image and the real fog-free image. However, L1 loss only calculates the difference between the two images at the pixel level and does not capture the contrast difference and edge information difference between the dehazed image and the real fog-free image. In order to solve the whole problem, this invention proposes multi-order differential loss based on fractional derivative. By processing the dehazed image and the fog-free image through multi-order convolution templates, the dehazed image is constrained to be consistent with the real fog-free image in terms of contrast, brightness and edge information.

[0072] (2) This invention will clearly Figure 4The classification discriminator and the reconstruction network are trained adversarially to handle dense fog. Figure 4 The classification discriminator is trained adversarially against the degradation network. Compared with conventional binary classification discriminators, the two four-class classification discriminators of this invention are used to constrain the clear image in the restoration-degradation process and the foggy image in the degradation-restoration process, making the effects of the restoration network and the degradation network more significant and improving the network performance. Attached Figure Description

[0073] Figure 1 This is a flowchart illustrating the image dehazing method based on edge attention and multi-order differential loss of the present invention.

[0074] Figure 2 This is a schematic diagram of the process for jointly training the restoration network and the degradation network according to the present invention;

[0075] Figure 3 This is a schematic diagram of the network structure of the restored network of the present invention;

[0076] Figure 4 This is a schematic diagram of the network structure of the edge attention layer of the present invention;

[0077] Figure 5 For clarity of this invention Figure 4 Structure diagram of a classification discriminator;

[0078] Figure 6 A schematic diagram of the fractional convolution template used in the multi-order differential loss calculation of this invention;

[0079] Figure 7 This is a schematic diagram of the visualization results of the edge attention map of the present invention;

[0080] Figure 8 This is a schematic diagram comparing the defogging effect of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0082] Example 1

[0083] like Figure 1 As shown, this embodiment provides an image dehazing method based on edge attention and multi-order differential loss, including the following steps:

[0084] S1: Obtain the foggy image dataset, preprocess the foggy image dataset, divide the foggy image dataset into a dense fog image dataset and a light fog image dataset, and use the light fog image dataset as the input image dataset;

[0085] In this embodiment, the image preprocessing steps specifically include:

[0086] The FADE fog density index of all foggy images in the dataset is measured. The fog density of an image is evaluated by the FADE index. The mean of all FADE indices is calculated. Images with a value greater than the mean are classified as dense fog images, and images with a value lower than the mean are classified as light fog images. The light fog images are used as input images a. The real fog-free images corresponding to the light fog images are used for supervised training, and the dense fog images are used for unsupervised training, i.e., for calculating the adversarial loss in the defogging process.

[0087] S2: Construct restoration and degradation networks;

[0088] like Figure 3 As shown, the restoration network includes an edge attention branch and a super-resolution restoration branch. The edge attention branch is used to improve the network's perception of edge information, and the super-resolution restoration branch is used to reconstruct lost details and textures. The edge attention branch includes multiple edge attention modules, and each attention module contains two edge attention layers.

[0089] In this embodiment, the edge attention branch is based on the encoding / decoding structure. A pre-trained Res2Net network is used in the encoding structure to extract features, resulting in features at multiple scales, namely E... 1 / 2 E 1 / 4 E 1 / 8 The features representing 1 / 2, 1 / 4, and 1 / 8 of the original input image size accelerate network convergence. During decoding at different scales, the decoding network includes multiple edge attention modules, each containing two edge attention layers. During decoding, the Pixelshuffle module is used for upsampling. The Pixelshuffle module converts low-resolution images to high-resolution images by blending pixels in the low-resolution image and reconstructing large pixel blocks in the high-resolution image.

[0090] like Figure 4 As shown, the structure of each layer of the edge attention layer is constructed. Each layer consists of three branches: X branch (XLayer), Y branch (YLayer), and information extraction branch (InfoLayer). The X and Y branches are used to learn the edge weights of the image, respectively. The final edge attention EA is obtained by adding them pixel by pixel. The information extraction branch (InfoLayer) further extracts the feature information of the input F. The output is multiplied pixel by pixel with the edge attention map, and then added to F to obtain the output feature F. out ;

[0091] This invention utilizes visualization technology to visualize edge attention maps. First, the edge attention map is normalized, and then converted into a Jet color map, such as... Figure 7 As shown, the fog distribution in the input foggy image is uneven and almost fills the entire image, resulting in a significant loss of texture and detail. In the edge attention map, the edges of the wood have higher weights, while the right side of the wood, the ground, and parts of the sky are smoother areas with lower weights, consistent with the Sobel edge map of the fog-free image. This verifies that edge attention blocks can make the network pay more attention to the edge information of the image in foggy regions, thereby improving the network's ability to reconstruct edge information in foggy regions.

[0092] In this embodiment, the encoding network in the super-resolution recovery branch uses a pre-trained network Res2Net for feature extraction and shares parameters with the edge attention branch. The decoding structure contains multiple residual blocks to extract features and recover lost information. It also includes multiple pixelshuffle modules and convolutional layers, where the pixelshuffle modules are used for upsampling.

[0093] In this embodiment, the input image passes through the edge attention branch and the super-resolution restoration branch to obtain the corresponding feature outputs. These outputs are then fused together and input into a progressive compression block, a mirror-filled ReflectionPad, a 7x7 convolution, and a Tanh activation function to obtain a restored clear image. The progressive compression block consists of multiple consecutive convolutions that compress the number of features from 80 to 20, reducing feature loss.

[0094] In this embodiment, the network structure of the degradation network is the same as that of the edge attention branch in the restoration network. The degradation network is used to simulate the generation of blurry, foggy images. By learning the degradation factors in images with different degrees of degradation, the performance of the restoration network is progressively enhanced.

[0095] In this embodiment, the restoration-degradation process is supervised training, while the degradation-restoration process is unsupervised training.

[0096] like Figure 5 As shown, a clear structure is obtained. Figure 4 The classification discriminator structure is clear. Figure 4 The classifier and the restoration network are trained adversarially. The restoration network generates a clear image, and the inputs are a clear, fog-free image, a foggy image, and a clear image generated by the restoration network RNet. After passing through a convolutional network, four values ​​are obtained: the probability of the input image corresponding to the class, the clarity, and the probability of the image being clear. Figure 4 The classification discriminator determines whether the generated image is a real, clear, fog-free image, a hazy image, a densely fogged image, or an image generated by the restoration network.

[0097] In this embodiment, the structure of the dense fog four-class discriminator is clear. Figure 4 Similar to the classification discriminator, there are slight differences in input and output. The dense fog four-class discriminator takes as input a clear, fog-free image, a hazy image, a dense fog image, and a dense fog image generated by the degraded network DNet, and outputs the corresponding image probability. In the adversarial process against the degraded network, the degraded network generates blurry, foggy images, and the dense fog four-class discriminator determines whether the generated image belongs to the real dense fog image, hazy image, clear image, or image generated by the degraded network. This adversarial process improves the mapping ability of the degraded network.

[0098] In this embodiment, using clear Figure 4 The classification discriminator and the fog four-class discriminator constrain the restoration network and the degradation network, thereby enhancing the network's mapping ability.

[0099] S3: As Figure 2 As shown, the restoration network and the degradation network are jointly trained to obtain the trained restoration network and degradation network; the training process is as follows:

[0100] In the first training phase, the hazy image is used as input image a. Input image a is processed by the restoration network to obtain the first clear image b, and then processed by the degradation network to obtain the hazy image c.

[0101] In the second training phase, the input image a is processed by a degradation network to obtain a foggy image d, and then processed by a restoration network to obtain a second clear image e.

[0102] The first and second training phases are conducted simultaneously, with the intermediate process of the first training phase being supervised learning. Loss calculations are performed on the first sharp image b and the real fog-free image, and then input into the sharp image... Figure 4 Adversarial loss is calculated in the classification discriminator. The intermediate process in the second training phase is unsupervised learning, where the dense fog image d is input into the dense fog... Figure 4 Adversarial loss is calculated in the classification discriminator.

[0103] The specific loss calculation process is as follows: Figure 1 The loss calculation involved includes L pointed to by b and e. con L frac b points to L gan c points to L gan Based on the properties of the loss function, b and e point to L. con This is called content loss, and b and e point to L. frac This is called multi-order differential loss. b points to L. gan and the L pointed to by c gan This is called adversarial loss, and the loss between c and a is called cycle consistency loss. Specifically, the first clear image b, the second clear image e, and the real fog-free image are compared to calculate content loss and multi-order differential loss. The first clear image b and the foggy image d are then input into the clear image and the foggy image d respectively. Figure 4 Adversarial loss is calculated in the classification discriminator and the fog four-class discriminator. Cycle consistency loss is calculated between the foggy image c and the foggy image a. Finally, the total loss is obtained based on the content loss, adversarial loss, cycle consistency loss and multi-order differential loss. The network parameters are then updated by the gradient descent algorithm. After the loss calculation, the loss will continue to decrease during the training process, thus improving the network performance.

[0104] In this embodiment, the content loss is specifically represented as:

[0105]

[0106] Among them, L con For content loss, N is the total number of images, and J is the content loss. gt 1 is the real, fog-free image; b is the clear image generated by the restoration network from the input image; and e is the clear image obtained by the degradation-restoration process from the input image.

[0107] In this embodiment, the multi-order differential loss function is specifically expressed as:

[0108]

[0109] Among them, L frac For multi-order differential loss, N is the total number of images, v is the order, and F... v For a differential convolution of order v, where V is a list of orders, the list of orders can be composed of any different orders. This example uses the classic Tiansi template, and the preferred order list is 0.5, 0.6 and 1.

[0110] like Figure 6 As shown, v represents the corresponding order, the numbers within the template are the parameters of the convolution kernel, and the convolution size of this template is 5x5; J gt denoted as 'b', where 'b' is the real, fog-free image, 'a' is the clear image generated from the input image through the restoration network, and 'e' is the clear image obtained from the input image after a degradation-restoration process. In the multi-order differential loss function of this embodiment, loss calculations are performed on template-processed images of different orders, and then weighted and applied... Figure 6 The three convolutional templates of order 0.5, 0.6, and 1.0 are used to process the image respectively.

[0111] In this embodiment, the cycle consistency loss is specifically expressed as:

[0112]

[0113] Where N is the total number of images, a is the input image, and c is the foggy image generated after the input image a is restored by the restoration network and degraded by the degradation network.

[0114] In this embodiment, it is clear Figure 4 Classifier D1 and dense fog Figure 4 The training loss and adversarial loss of the D2 classifier are as follows:

[0115]

[0116]

[0117] L gan (RNet,D1)=CE(D1(b),C gt )

[0118] L gan (DNet,D2)=CE(D2(d),C dense-gt )

[0119]

[0120] in, For clear training Figure 4 The loss of the classification discriminator, For training in dense fog Figure 4 The loss of the classifier, L gan (RNet,D1) represents the adversarial loss of RNet and D1, L gan (DNet, D2) represents the adversarial loss of DNet and D2, L adv J represents the total adversarial loss; CE represents the cross-entropy loss; J represents the total adversarial loss. gt For a true haze-free image, and for a hazy image, for a corresponding true haze-free image, i.e., in pairs, I dense-gt Given real fog images, a is the input fog image, b is the clear image generated by the restoration network from the input image, and d is the fog image obtained by the degradation network from the input image. gt C represents the category of a truly fog-free image. a C represents the category of the input hazy image. dense-gt C represents the category of a real fog image. RNet C represents the category of sharp images generated by the restoration network RNet. DNet Indicates the category of the fog map generated by the degenerate network DNet;

[0121] The total loss function is as follows:

[0122] L total =αL con +βL cycle +γL adv +λL frac

[0123] Wherein, α, β, γ, and λ represent weighting parameters, which are preferably 1.0, 0.1, 0.1, and 0.1, respectively;

[0124] The network parameters are updated by backpropagation using stochastic gradient descent until the network converges, resulting in a well-trained restored network.

[0125] S4: After training, a restoration network is obtained. The image to be tested is processed through the restoration network to obtain the dehazing result.

[0126] In this embodiment, only the restoration network is needed for dehazing. The purpose of the degradation network is to generate images with a high degree of degradation during training and train them together with the restoration network to improve the restoration performance of the restoration network.

[0127] In this embodiment, the Adam optimizer is used for optimization, with beta1 set to 0.9 and beta2 set to 0.999. The number of edge attention layers is set to 2, and the order of the multi-order differential loss is selected as 0.5, 0.6, and 1.0. In the total loss function, α is set to 1.0 and β is set to 0.2. The training epochs are set to 500, the batch size is set to 1 for each iteration, and the learning rate is initialized to 1e-4, decaying by half every 50 epochs.

[0128] This invention was tested on the NH-HAZE dataset, such as... Figure 8 As shown, the experimental comparison results are obtained. The first row of the figure shows the fogged image, the second row shows the dehazing result of FFA, the third row shows the dehazing result of the method of this invention, and the fourth row shows the fog-free image. The method of this invention dehazes more thoroughly and is closer to the real fog-free image. By comparing the dehazing results of FFA, the effectiveness and rationality of the dehazing method proposed in this invention are demonstrated.

[0129] The comparison of the indicators is shown in Table 1 below. As can be seen from the table, the method proposed in this invention is superior to the FFA network in both PSNR and SSIM indicators.

[0130] Table 1 Comparison of Indicators

[0131]

[0132] Example 2:

[0133] Except for the following technical contents, the technical contents of this embodiment are the same as those of Embodiment 1;

[0134] This embodiment provides an image dehazing system based on edge attention and multi-order differential loss, including: an image acquisition module, an image preprocessing module, a network construction module, a network training module, and a dehazing module;

[0135] In this embodiment, the image acquisition module is used to acquire a dataset of foggy images;

[0136] In this embodiment, the image preprocessing module is used to preprocess the foggy image dataset, dividing the foggy image dataset into a dense fog image dataset and a light fog image dataset, and using the light fog image dataset as the input image dataset.

[0137] In this embodiment, the network construction module is used to construct the restoration network and the degradation network;

[0138] In this embodiment, the network training module is used to jointly train the restoration network and the degradation network to obtain the trained restoration network and degradation network, specifically including:

[0139] The first stage of training: the hazy image is processed by the restoration network to obtain the first clear image, and the first clear image is processed by the degradation network to obtain the hazy image;

[0140] The second stage of training: the hazy image is processed by a degradation network to obtain a dense fog image, and the dense fog image is processed by a restoration network to obtain a second clear image;

[0141] Content loss is calculated for the first clear image, the second clear image, and the real haze-free image. Multi-order differential loss is calculated for the first clear image, the second clear image, and the real haze-free image. Cycle consistency loss is calculated for the hazy image and the hazy image. The first clear image is then input into the clear image... Figure 4 Adversarial loss is calculated in the classification discriminator, clearly... Figure 4 The classification discriminator and the restoration network are trained adversarially by inputting dense fog images into the dense fog network. Figure 4 Adversarial loss is calculated in the classification discriminator, in dense fog. Figure 4 The classification discriminator is trained adversarially against the degenerate network;

[0142] The total loss is obtained by weighted summation of content loss, adversarial loss, cycle consistency loss, and multi-order differential loss, and the network parameters are updated using the gradient descent algorithm.

[0143] In this embodiment, the dehazing module is used to input the image to be tested into the trained restoration network to obtain the dehazing result.

[0144] In this embodiment, the restoration network includes an edge attention branch and a super-resolution restoration branch;

[0145] The edge attention branch adopts an encoding-decoding structure. The encoding structure uses a pre-trained network Res2Net for feature extraction, and the decoding structure includes multiple edge attention modules, each of which includes two edge attention layers.

[0146] The super-resolution restoration branch adopts an encoding-decoding structure. In the encoding structure, a pre-trained network Res2Net is used for feature extraction. In the decoding structure, multiple residual blocks are included to extract image features and restore lost image information.

[0147] The image features output from the edge attention branch and the super-resolution restoration branch are sequentially processed through progressive compression blocks, mirror padding, convolutional layers, and the Tanh activation function to obtain a restored sharp image;

[0148] The network structure of the degraded network is the same as that of the edge attention branch in the restored network.

[0149] In this embodiment, the first and second training phases are performed simultaneously, with the first phase being supervised learning and the second phase being unsupervised learning.

[0150] In this embodiment, content loss is calculated by comparing the first clear image, the second clear image, and the actual haze-free image, specifically as follows:

[0151]

[0152] Among them, L con For content loss, N is the total number of images, and J is the content loss. gt b represents the true, fog-free image, and e represents the first clear image.

[0153] The first clear image, the second clear image, and the true haze-free image are subjected to multi-order differential loss calculation, specifically expressed as follows:

[0154]

[0155] Among them, L frac For multi-order differential loss, v is the order, F v For v-order differential convolution, where V is a list of orders;

[0156] The cycle consistency loss is calculated for both foggy and lightly foggy images, specifically as follows:

[0157]

[0158] Where a is the input image and c is the foggy image;

[0159] Input the first clear image into the clear image. Figure 4 Adversarial loss is calculated in the classification discriminator, clearly... Figure 4 The classification discriminator and the restoration network are trained adversarially by inputting dense fog images into the dense fog network. Figure 4 Adversarial loss is calculated in the classification discriminator, in dense fog. Figure 4The classifier and the degenerate network are trained adversarially, specifically as follows:

[0160]

[0161]

[0162] L gan (RNet,D1)=CE(D1(b),C gt )

[0163]

[0164] in, For clear training Figure 4 The loss of the classification discriminator, For training in dense fog Figure 4 The loss of the classifier, L gan (RNet, D1) represents the restored network RNet and the clear network D1. Figure 4 Adversarial loss of classifier D1, L gan (DNet, D2) represents the degenerate network DNet and the dense fog network. Figure 4 Adversarial loss of classifier D2, L adv Let J be the total adversarial loss, CE be the cross-entropy loss, and J be the total adversarial loss. gt For a truly fog-free image, I dense-gt d represents a real fog image, and C represents the fog image. gt C represents the category of a truly fog-free image. a C represents the category of the input hazy image. dense-gt C represents the category of a real fog image. RNet C represents the category of sharp images generated by the restoration network RNet. DNet This indicates the category of the fog map generated by the degenerate network DNet.

[0165] In this embodiment, a multi-order differential loss is introduced into the calculation of the loss function to constrain the consistency of the dehazed image and the real haze-free image in terms of contrast and edge information, thereby improving the performance of the network.

[0166] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An image dehazing method based on edge attention and multi-order differential loss, characterized in that, Includes the following steps: Obtain the foggy image dataset, preprocess the foggy image dataset, divide the foggy image dataset into a dense fog image dataset and a light fog image dataset, and use the light fog image dataset as the input image dataset; Construct restoration and degradation networks; The restored network and the degenerate network are jointly trained to obtain the trained restored network and the degenerate network, specifically including: The first stage of training: the hazy image is processed by the restoration network to obtain the first clear image, and the first clear image is processed by the degradation network to obtain the hazy image; The second stage of training: the hazy image is processed by a degradation network to obtain a dense fog image, and the dense fog image is processed by a restoration network to obtain a second clear image; Content loss is calculated for the first clear image, the second clear image, and the real fog-free image; multi-order differential loss is calculated for the first clear image, the second clear image, and the real fog-free image; and cycle consistency loss is calculated for the foggy image and the hazy image. The first clear image is input into the clear image four-class discriminator for adversarial loss calculation. The clear image four-class discriminator is then trained adversarially against the restoration network. Similarly, the foggy image is input into the foggy image four-class discriminator for adversarial loss calculation. The foggy image four-class discriminator is then trained adversarially against the degradation network. Specifically: ； ； ； ； in, The loss for training a clear image four-class discriminator, The loss function used to train the four-class classifier for fog maps. To recover the adversarial loss of the RNet network and the clear graph four-class discriminator D1, For the adversarial loss of the degenerate network DNet and the fog map four-class discriminator D2, Let J be the total adversarial loss, CE be the cross-entropy loss, and J be the total adversarial loss. gt For a truly fog-free image, I dense-gt The images are real fog images, where a is the input fog image, b is the first clear image, d is the fog image, and C is the real fog image. gt C represents the category of a truly fog-free image. a C represents the category of the input hazy image. dense-gt C represents the category of a real fog image. RNet C represents the category of sharp images generated by the restoration network RNet. DNet Indicates the category of the fog map generated by the degenerate network DNet; The total loss is obtained by weighted summation of content loss, adversarial loss, cycle consistency loss, and multi-order differential loss, and the network parameters are updated using the gradient descent algorithm. The image to be tested is input into the trained restoration network to obtain the dehazing result.

2. The image dehazing method based on edge attention and multi-order differential loss according to claim 1, characterized in that, Preprocessing of the hazy image dataset includes: Measure the FADE fog density index of all fog images in the fog image dataset, calculate the mean of all FADE fog density indices, classify images with values ​​greater than the mean as dense fog images, and classify images with values ​​less than the mean as light fog images.

3. The image dehazing method based on edge attention and multi-order differential loss according to claim 1, characterized in that, The restoration network includes an edge attention branch and a super-resolution restoration branch; The edge attention branch adopts an encoding-decoding structure. In the encoding structure, a pre-trained network Res2Net is used to extract features. In the decoding structure, multiple edge attention modules are included, and each attention module includes two edge attention layers. The super-resolution restoration branch adopts an encoding and decoding structure. In the encoding structure, a pre-trained network Res2Net is used to extract features, and the decoding structure includes multiple residual blocks for extracting image features and restoring lost image information. The image features output by the edge attention branch and the super-resolution restoration branch are sequentially processed through progressive compression blocks, mirror padding, convolutional layers, and the Tanh activation function to obtain a restored clear image; The network structure of the degraded network is the same as that of the edge attention branch in the restored network.

4. The image dehazing method based on edge attention and multi-order differential loss according to claim 1, characterized in that, The first and second phases of training are conducted simultaneously, with the first phase being supervised learning and the second phase being unsupervised learning.

5. The image dehazing method based on edge attention and multi-order differential loss according to claim 1, characterized in that, Content loss is calculated by comparing the first clear image, the second clear image, and the true haze-free image, specifically as follows: ； Among them, L con For content loss, N is the total number of images, and J is the content loss. gt b represents the true, fog-free image, and e represents the first clear image.

6. The image dehazing method based on edge attention and multi-order differential loss according to claim 1, characterized in that, The first clear image, the second clear image, and the true haze-free image are subjected to multi-order differential loss calculation, specifically expressed as follows: ； Among them, L frac For multi-order differential loss, N is the total number of images, v is the order, and F... v Let J be a differential convolution of order v, where V is a list of orders, and J is a differential convolution of order v. gt b represents the true, fog-free image, and e represents the first clear image.

7. The image dehazing method based on edge attention and multi-order differential loss according to claim 1, characterized in that, The cycle consistency loss is calculated for both foggy and lightly foggy images, specifically as follows: ； Where N is the total number of images, a is the input image, and c is the foggy image generated after the input image a is restored by the restoration network and degraded by the degradation network.

8. An image dehazing system based on edge attention and multi-order differential loss, characterized in that, include: Image acquisition module, image preprocessing module, network construction module, network training module, and dehazing module; The image acquisition module is used to acquire a dataset of foggy images; The image preprocessing module is used to preprocess the foggy image dataset, dividing the foggy image dataset into a dense fog image dataset and a light fog image dataset, and using the light fog image dataset as the input image dataset. The network construction module is used to construct restoration networks and degradation networks; The network training module is used to jointly train the restoration network and the degradation network to obtain the trained restoration network and degradation network, specifically including: The first stage of training: the hazy image is processed by the restoration network to obtain the first clear image, and the first clear image is processed by the degradation network to obtain the hazy image; The second stage of training: the hazy image is processed by a degradation network to obtain a dense fog image, and the dense fog image is processed by a restoration network to obtain a second clear image; Content loss is calculated for the first clear image, the second clear image, and the real fog-free image; multi-order differential loss is calculated for the first clear image, the second clear image, and the real fog-free image; and cycle consistency loss is calculated for the foggy image and the hazy image. The first clear image is input into the clear image four-class discriminator for adversarial loss calculation. The clear image four-class discriminator is then trained adversarially against the restoration network. Similarly, the foggy image is input into the foggy image four-class discriminator for adversarial loss calculation. The foggy image four-class discriminator is then trained adversarially against the degradation network. Specifically: ； ； ； ； in, The loss for training a clear image four-class discriminator, The loss function used to train the four-class classifier for fog maps. To recover the adversarial loss of the RNet network and the clear graph four-class discriminator D1, For the adversarial loss of the degenerate network DNet and the fog map four-class discriminator D2, Let J be the total adversarial loss, CE be the cross-entropy loss, and J be the total adversarial loss. gt For a truly fog-free image, I dense-gt The images are real fog images, where a is the input fog image, b is the first clear image, d is the fog image, and C is the real fog image. gt C represents the category of a truly fog-free image. a C represents the category of the input hazy image. dense-gt C represents the category of a real fog image. RNet C represents the category of sharp images generated by the restoration network RNet. DNet Indicates the category of the fog map generated by the degenerate network DNet; The total loss is obtained by weighted summation of content loss, adversarial loss, cycle consistency loss, and multi-order differential loss, and the network parameters are updated using the gradient descent algorithm. The dehazing module is used to input the image to be tested into the trained restoration network to obtain the dehazing result.

9. The image dehazing system based on edge attention and multi-order differential loss according to claim 8, characterized in that, Content loss is calculated by comparing the first clear image, the second clear image, and the true haze-free image, specifically as follows: ； Among them, L con For content loss, N is the total number of images, and J is the content loss. gt b represents the true, fog-free image, and e represents the first clear image. The first clear image, the second clear image, and the true haze-free image are subjected to multi-order differential loss calculation, specifically expressed as follows: ； Among them, L frac For multi-order differential loss, v is the order, F v For v-order differential convolution, where V is a list of orders; The cycle consistency loss is calculated for both foggy and lightly foggy images, specifically as follows: ； Where a is the input image, and c is the foggy image generated after the input image a is restored by the restoration network and degraded by the degradation network.

Images