A non-paired image defogging method based on a cyclic diffusion model

Abstract

The application discloses a non-paired image defogging method based on a cyclic diffusion model, which comprises the following steps: 1, obtaining training set images; 2, constructing a defogging network based on a cyclic diffusion model; 3, extracting features of real foggy images and real clear images; 4, establishing a total loss function; 5, training the defogging network based on the cyclic diffusion model; and 6, defogging a single image by using the trained defogging network based on the cyclic diffusion model. The method has simple steps and reasonable design, and the defogging network based on the cyclic diffusion model is trained based on real foggy images and real clear images which are not paired, so that the prior information contained in the stable cyclic diffusion model in the defogging network based on the cyclic diffusion model is fully utilized while the defogging capability is ensured, the generated defogging image has better visual quality, and the defogging network has better generalization capability.

Description

Technical Field

[0001] This invention belongs to the field of image dehazing technology, specifically relating to an unpaired image dehazing method based on a cyclic diffusion model. Background Technology

[0002] Due to the impact of smog, the image quality acquired by intelligent video surveillance systems is severely degraded, exhibiting reduced clarity, decreased contrast, and blurred details. This negatively impacts subsequent image processing tasks, such as object recognition and scene understanding. Image dehazing aims to restore images acquired under smoggy conditions to fog-free images, thereby improving the visual quality of the images.

[0003] Existing image dehazing methods are mainly divided into image enhancement-based methods, prior information-based methods, and deep learning-based methods. Prior information-based methods only process image contrast and color information, achieving some dehazing effect, but their effectiveness is limited. Prior information-based methods treat the image itself as the research object, using the information inherent in the image as prior knowledge to obtain the transmission map and atmospheric light intensity, then substituting them into an atmospheric scattering model to obtain the dehazed image. These methods consider the causes of haze formation in real-world scenes and can effectively restore the texture information of haze-free images. However, since artificially set prior information for certain scenes is not applicable to all scenes, these methods often over-dehaze, resulting in a decrease in the quality of the dehazed image. Deep learning-based image dehazing methods have become the mainstream at present. Image dehazing is achieved by learning the mapping between hazy and clear images using a data-driven approach. This requires a large amount of paired real-world image data for training. However, collecting paired real-world hazy training datasets is difficult and resource-intensive in the real world. Therefore, most existing research uses synthetic hazy data for training and achieves good dehazing results on synthetic test data. However, due to the differences between synthetic and real-world hazy data, these methods often fail to effectively dehaze real-world hazy images. To address this issue, a method relies on real-world hazy and clear images and uses an unpaired training approach. This method, trained on real-world data, exhibits good generalization ability on real-world image data.

[0004] As the recurrent diffusion model has demonstrated powerful generative capabilities in the field of image generation, fine-tuning its training can both leverage its strong feature representation capabilities and significantly reduce training costs.

[0005] Therefore, a well-designed unpaired image dehazing method based on a cyclic diffusion model is needed. Unpaired training-based image dehazing methods can effectively utilize training data from real-world scenes, thereby enabling the trained dehazing model to have better generalization ability and resulting in dehazed images with better visual quality. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing an unpaired image dehazing method based on a cyclic diffusion model. The method has simple steps and a reasonable design. It trains a dehazing network based on a cyclic diffusion model on unpaired real foggy images and real clear images. While ensuring the dehazing ability, it makes full use of the prior information contained in the stable cyclic diffusion model in the dehazing network, so that the generated dehazed images have better visual quality and the dehazing network has better generalization ability.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an unpaired image dehazing method based on a cyclic diffusion model, characterized in that the method includes the following steps:

[0008] Step 1: Obtaining the training set images:

[0009] Real foggy images were collected from the RTTS and URHI datasets in the RESIDE foggy image database, and real clear images were collected from the ADE20K database and the OTS dataset in the RESIDE foggy image database as training data; one real foggy image and one real clear image were used as a set of training data.

[0010] Step 2: Construct a dehazing network based on a cyclic diffusion model:

[0011] The defogging network based on the cyclic diffusion model includes a defogging branch network and a fogging branch network. The defogging branch network includes a first defogging network module, a first fogging network module, and a defogging discriminator. The fogging branch network includes a second fogging network module, a second defogging network module, and a fogging discriminator. The first defogging network module, the second defogging network module, the first fogging network module, and the second fogging network module have the same structure, and the defogging discriminator and the fogging discriminator have the same structure.

[0012] Both the first and second defogging network modules include a defogging backbone network, a text perception guidance module, and a physical perception guidance module;

[0013] Both the first and second fogging network modules include a fogging backbone network and a text perception guidance module; both the defogging backbone network and the fogging backbone network are stable cyclic diffusion models.

[0014] The defogging backbone network and the fogging backbone network have the same structure, and both include a VAE encoder, a UNet and a VAE decoder. Four zero convolutional layers are added between the VAE encoder and the VAE decoder.

[0015] Step 3: Feature extraction from real foggy images and real clear images:

[0016] Step 301: Process the real foggy image x through the text-aware guidance module in the first dehazing network module to obtain a foggy image with added text control conditions. Then, process the foggy image with added text control conditions through the dehazing backbone network in the first dehazing network module to obtain a clear image after dehazing.

[0017] The real foggy image x is processed by the text perception guidance module in the second fogging network module to obtain the input foggy image with added text control conditions. The input foggy image with added text control conditions is then passed through the fogging backbone network in the second fogging network module to obtain the output foggy image F(x).

[0018] The real foggy image x is passed through the physical perception guidance module in the first dehazing network module to obtain the first reconstructed foggy image I. phy ;

[0019] Clear image after dehazing After passing through the text-aware guidance module in the first fogging network module, a dehazed image with added text control conditions is obtained. This dehazed image with added text control conditions is then processed by the fogging backbone network in the first fogging network module to obtain a synthesized fogged image.

[0020] Step 302: The real, clear image y is processed by the text-aware guidance module in the second fogging network module to obtain a clear image with added text control conditions. The clear image with added text control conditions is then processed by the fogging backbone network in the second fogging network module to obtain the fogged image.

[0021] The real clear image y is processed by the text perception guidance module in the first dehazing network module to obtain the input clear image with added text control conditions. The input clear image with added text control conditions is then passed through the dehazing backbone network of the first dehazing network module to obtain the output clear image G(y).

[0022] Image after fogging After passing through the text-aware guidance module in the second dehazing network module, a hazy image with added text control conditions is obtained. This hazy image with added text control conditions is then processed by the dehazing backbone network in the second dehazing network module to obtain a synthesized dehazing image.

[0023] Image after fogging After passing through the physical perception guidance module in the second dehazing network module, the second reconstructed hazy image I′ is obtained. phy ;

[0024] Step 4: Establishing the total loss function:

[0025] Step 401, according to The generative adversarial loss L is obtained GAN ; where D Y (y) represents the value output by the dehazing discriminator for the true, clear image y. This indicates a clear image after dehazing. The value output by the defogging discriminator, D X (x) represents the value output by the fog discriminator for the real foggy image x. Image after fogging The value output by the fogging discriminator;

[0026] Step 402, according to Obtain the cycle-consistent loss L cyc ;in, Represents a real foggy image x and a synthetic foggy image. Manhattan distance between them Represents a real foggy image x and a synthetic foggy image. LPIPS distance between Represents the true sharp image y and the synthetic dehazed image. Manhattan distance between them Represents the true sharp image y and the synthetic dehazed image. LPIPS distance between them;

[0027] Step 403, according to L idt =L rec1 (G(y),y)+L rec2 (G(y),y)+L rec1 (F(x),x)+L rec2 (F(x),x), thus obtaining the identity loss L idt ; among which, L rec1 (G(y),y) represents the Manhattan distance between the true sharp image y and the output sharp image G(y), L rec2 (G(y),y) represents the LPIPS distance between the true sharp image y and the output sharp image G(y), L rec1 (F(x),x) represents the Manhattan distance between the real foggy image x and the output foggy image F(x), L rec2(F(x),x) represents the LPIPS distance between the real foggy image x and the output foggy image F(x);

[0028] Step 404, according to L phy (1) = L phy +L p ′ hy The reconstruction loss L is obtained. phy (1); where L phy Let L represent the reconstruction loss of the first branch, and L phy =L rec1 (x,I phy )+L rec2 (x,I phy ), L rec1 (x,I phy ) represents the real foggy image x and the first reconstructed foggy image I. phy The Manhattan distance between them, L rec2 (x,I phy ) represents the real foggy image x and the first reconstructed foggy image I. phy LPIPS distance between; L p ′ hy This represents the loss from reconstructing the second branch, and Image after fogging Second reconstructed foggy image I′ phy Manhattan distance between them Image after fogging Second reconstructed foggy image I′ phy LPIPS distance between them;

[0029] Step 405, according to L loss =L GAN +L cyc +L idt +0.5L phy (1) , thus obtaining the total loss function L loss ;

[0030] Step 5: Training the dehazing network based on the cyclic diffusion model:

[0031] Step 501: The computer uses the Adam optimization algorithm. It inputs a set of training data and first utilizes the generative adversarial loss L... GAN The fog discriminator and the defogging discriminator are trained until the generative adversarial loss L is obtained. GAN Fog discrimination and defogging discrimination at maximum value;

[0032] Step 502: The computer uses the Adam optimization algorithm, inputs the training data, and, under the fog discriminator and defogging discriminator determined in step 501, uses the total loss function L...loss The dehazing network based on the recurrent diffusion model is trained until the total loss function L is reached. loss Minimum, complete training on this set of training data;

[0033] Step 503: Following the methods in steps 501 to 502, continue training until the entire training set is trained, completing one iteration of training;

[0034] Step 504: Repeat steps 501 to 503, iteratively training until the preset number of iterations is met, to obtain the trained dehazing network based on the cyclic diffusion model;

[0035] Step 6: Dehaze a single image using the trained dehazing network based on the recurrent diffusion model:

[0036] A computer is used to input any foggy image into the first defogging network module of the defogging branch network of a pre-trained defogging network based on a cyclic diffusion model, and the resulting clear image is obtained after defogging.

[0037] The above-mentioned unpaired image dehazing method based on a cyclic diffusion model is characterized in that: in step two, the four zero convolutional layers added between the VAE encoder and the VAE decoder are the first zero convolutional layer, the second zero convolutional layer, the third zero convolutional layer, and the fourth zero convolutional layer, respectively.

[0038] The VAE encoder includes an input convolutional layer, a first DownEncoderBlock2D, a second DownEncoderBlock2D, a third DownEncoderBlock2D, and a fourth DownEncoderBlock2D;

[0039] The VAE decoder includes an output convolutional layer plus UNetMidBlock2D, a first UpDecoderBlock2D, a second UpDecoderBlock2D, a third UpDecoderBlock2D, and a fourth UpDecoderBlock2D;

[0040] A first zero-convolutional layer is set between the output of the input convolutional layer of the VAE encoder and the input of the fourth UpDecoderBlock2D; a second zero-convolutional layer is set between the output of the first DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a third zero-convolutional layer is set between the output of the second DownEncoderBlock2D and the input of the second UpDecoderBlock2D; and a fourth zero-convolutional layer is set between the output of the third DownEncoderBlock2D and the input of the first UpDecoderBlock2D.

[0041] The above-mentioned unpaired image dehazing method based on a cyclic diffusion model is characterized by the following: the process of processing the hazy image with added text control conditions through the dehazing backbone network of the first dehazing network module, the hazy image with added text control conditions through the dehazing backbone network of the second dehazing network module, the input clear image with added text control conditions through the dehazing backbone network of the first dehazing network module, the dehazing image with added text control conditions through the hazing backbone network of the first hazing network module, the clear image with added text control conditions through the hazing backbone network of the second hazing network module, and the input hazy image with added text control conditions through the hazing backbone network of the second hazing network module is the same, as detailed below:

[0042] Step 3011: The foggy image with added text control conditions, the fogged image with added text control conditions, the clear input image with added text control conditions, the defogging image with added text control conditions, the clear image with added text control conditions, and the foggy input image with added text control conditions are all recorded as the input image and control text.

[0043] Step 3012: The input image is passed through a VAE encoder to obtain an encoded output feature map; wherein, the input image is input into the input convolutional layer of the VAE encoder to obtain a first intermediate feature map, and the first intermediate feature map is passed through the first DownEncoderBlock2D, the second DownEncoderBlock2D, the third DownEncoderBlock2D and the fourth DownEncoderBlock2D in sequence to obtain a second intermediate feature map, a third intermediate feature map, a fourth intermediate feature map and a fifth intermediate feature map;

[0044] Step 3013: The encoded output feature map and control text are processed by UNet to obtain the sixth intermediate feature map;

[0045] Step 3014: The first intermediate feature map is passed through the first zero convolutional layer to output the first transition feature map; the second intermediate feature map is passed through the second zero convolutional layer to output the second transition feature map; the third intermediate feature map is passed through the third zero convolutional layer to output the third transition feature map; and the fourth intermediate feature map is passed through the fourth zero convolutional layer to output the fourth transition feature map.

[0046] Step 3015: Pass the sixth intermediate feature map, the first transition feature map, the second transition feature map, the third transition feature map, and the fourth transition feature map through the VAE decoder to obtain the decoded output feature map; wherein, the sixth intermediate feature map is passed through the output convolutional layer of the VAE decoder and then processed by UNetMidBlock2D to obtain the seventh intermediate feature map; the seventh intermediate feature map and the fourth transition feature map are added together and then processed by the first UpDecoderBlock2D to obtain the eighth intermediate feature map; the eighth intermediate feature map and the third transition feature map are added together and then processed by the second UpDecoderBlock2D to obtain the ninth intermediate feature map; the ninth intermediate feature map and the second transition feature map are added together and then processed by the third UpDecoderBlock2D to obtain the tenth intermediate feature map; the tenth intermediate feature map and the first transition feature map are added together and then processed by the fourth UpDecoderBlock2D to obtain the eleventh intermediate feature map;

[0047] Step 3016: Record the decoded output feature map corresponding to the foggy image with added text control conditions as the clear image after dehazing. The decoded output feature map corresponding to the hazy image with added text control conditions is denoted as the synthesized dehazed image. The decoded output feature map corresponding to the input sharp image with added text control conditions is denoted as the output sharp image G(y);

[0048] The decoded output feature map corresponding to the dehazed image with added text control conditions is denoted as the synthesized hazy image. The decoded output feature map corresponding to the clear image with added text control conditions is denoted as the fogged image. The decoded output feature map corresponding to the output of the input foggy image with added text control conditions is denoted as the output foggy image F(x).

[0049] The aforementioned unpaired image dehazing method based on a cyclic diffusion model is characterized by: the real hazy image x being processed by a text-aware guidance module in the first dehazing network module, the real clear image y being processed by a text-aware guidance module in the first dehazing network module, and the hazy image... The methods used in the text-aware guidance module of the second dehazing network module are all the same, as follows:

[0050] Step 3021: Combine the real foggy image x, the real clear image y, and the image after fogging. All are referred to as the first initial image;

[0051] Step 3022: Input the first initial image into the BLIP2 module to obtain the first text title;

[0052] Step 3023: Process the real foggy images in the training data using the text inversion method to obtain negative cue features representing fog;

[0053] Step 3024: After deleting haze-related words from the first text title, feed it into the CLIP model text encoder to obtain the text embedding representation features;

[0054] Step 3025: The text embedding representation features and the negative cue features representing fog are obtained by classifying the text without a classifier;

[0055] Step 3026: The text control obtained by processing the real foggy image x through the text perception guidance module in the first defogging network module and the real foggy image x are denoted as the foggy image with added text control conditions;

[0056] The text control and the true clear image y obtained by the text perception guidance module in the first dehazing network module are denoted as the input clear image with added text control conditions.

[0057] Image after fogging The image obtained after text-aware guidance by the text perception module in the second dehazing network module is both controlled and fogged. This is recorded as a fogged image with added text control conditions;

[0058] Clear image after dehazing The processing methods for the text-aware guidance module in the first fogging network module, the text-aware guidance module in the second fogging network module for the real fogged image x, and the text-aware guidance module in the real clear image y for the second fogging network module are all the same, as detailed below:

[0059] Step 30A: Obtain the clear image after dehazing. The real foggy image x and the real clear image y are denoted as the second initial image;

[0060] Step 30B: Input the second initial image into the BLIP2 module to obtain the second text title;

[0061] Step 30C: Add haze-related words to the second text title and then feed it into the CLIP model text encoder to obtain the second text embedding representation features;

[0062] Step 30D: The second text embedding representation features are guided by a classifier-free method to obtain the second text control;

[0063] Step 30E: Obtain the clear image after dehazing. The second text control and the clear image after defogging are obtained through the text-aware guidance module in the first fogging network module. This is denoted as a dehazed image with added text control conditions;

[0064] The real clear image y, after passing through the text perception guidance module in the second fogging network module, is used to obtain the second text control and the real clear image y, which are denoted as the input clear image with added text control conditions;

[0065] The second text control obtained by passing the real foggy image x through the text perception guidance module in the second fogging network module and the real foggy image x are denoted as the input foggy image with added text control conditions.

[0066] The above-mentioned unpaired image dehazing method based on a cyclic diffusion model is characterized in that: in step 301, the real foggy image x passes through the physical perception guidance module in the first dehazing network module to obtain the first reconstructed foggy image I. phy And the image after fogging in step 302 After passing through the physical perception guidance module in the second dehazing network module, the second reconstructed hazy image I′ is obtained. phy The method is the same, and the specific process is as follows:

[0067] Step A: The dark channel prior dehazing (DCP) algorithm is used to process the real hazy image x to obtain the first dehazing feature map J. dcp First transmission image t dcp and atmospheric light A;

[0068] The boundary constraint prior (BCCR) algorithm is used to process the real hazy image x to obtain the second dehazed feature map J. bccr Second transmission diagram t bccr ;

[0069] Step B: Use the computer to call the concatenate function to combine the real foggy image x and the first transmission image t. dcp Second transmission diagram t bccr After stitching, the transmission pattern is input into the transmission pattern optimization network to obtain the optimized transmission pattern t. ref ;

[0070] Step C: Combine the real foggy image x and the first dehazing feature map J. dcp Second defogging feature map J bccr After passing through the Physical Fusion Module (PFM), the fused feature map F is obtained. fuse ;

[0071] Step D: Fuse the feature map F fuse Inputting the dehazing backbone network yields the optimized dehazed image J. ref ;

[0072] Step F: Apply the Atmospheric Scattering Model (ASM) to the optimized dehazed image J. refOptimized transmission map t ref After processing with atmospheric light A, a reconstructed foggy image I is obtained. phy ;

[0073] Step G: Following the methods in steps A to F, process the image after fogging. After processing, the second reconstructed foggy image I′ is obtained. phy .

[0074] The above-mentioned unpaired image dehazing method based on a cyclic diffusion model is characterized by the following step C:

[0075] Step C01: Transfer the first dehazing feature map J dcp After passing through the first convolutional + ReLU activation function layer, the first feature map f1 is obtained;

[0076] The real foggy image x is passed through the first double-layer convolution module to obtain the second feature map f2;

[0077] The second defogging feature map J bccr After passing through the second convolutional + ReLU activation function layer, the third feature map f3 is obtained;

[0078] Step C02: Add the second feature map f2 to the first feature map f1 to obtain the fourth feature map f4; add the second feature map f2 to the third feature map f3 to obtain the fifth feature map f5.

[0079] Step C03: Pass the fourth feature map f4 through the third convolution + ReLU activation function layer to obtain the sixth feature map f6;

[0080] The fifth feature map f5 is passed through the fourth convolutional + ReLU activation function layer to obtain the seventh feature map f7;

[0081] Step C04: Perform a Hadamard product operation between the output of the sixth feature map f6 after passing through the first Sigmoid function layer and the fourth feature map f4 to obtain the eighth feature map f8;

[0082] The output of the seventh feature map f7 after passing through the second Sigmoid function layer is combined with the fifth feature map f5 through the Hadamard product to obtain the ninth feature map f9;

[0083] Step C05: Use the concatenate function to concatenate the eighth feature map f8 and the ninth feature map f9 to obtain the tenth feature map f. 10 And the tenth feature map f 10 After passing through a global adaptive pooling layer, a second double-layer convolutional module, and a Softmax layer, the eleventh feature map f is obtained. 11 ;

[0084] Step C06: Transfer the eleventh feature map f 11 With the tenth feature map f 10 Features after Hadamard product operation and the tenth feature image f 10 Adding them together, we get the twelfth feature map f. 12 ;

[0085] Step C07: Use the concatenate function to concatenate the first dehazed feature map J. dcp Second defogging feature map J bccr and the twelfth feature diagram f 12 After concatenation, the resulting data is fed into an inductively coupled convolutional network to obtain the fused feature weights f. w ;

[0086] Step C08: Use a computer to calculate the formula. The fused feature map F is obtained. fuse ;in, This represents the Hadamard product operation between feature map matrices. f represents the addition operation between feature map matrices; w1 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the first channel, f w2 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the second channel, f w3 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the third channel, f w4 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the fourth channel; f 12-1 Represents the twelfth feature map f 12 Feature maps of the first three channels; f 12-2 Represents the twelfth feature map f 12 Feature maps of the middle and rear three channels.

[0087] Compared with the prior art, the present invention has the following advantages:

[0088] 1. The method of this invention has simple steps and reasonable design. First, the training set of images is acquired; second, a dehazing network based on the cyclic diffusion model is constructed; then, features of real foggy images and real clear images are extracted; then, the total loss function is established; then, the dehazing network based on the cyclic diffusion model is trained; finally, the trained dehazing network based on the cyclic diffusion model is used to dehaze a single image, thereby improving the image dehazing effect.

[0089] 2. The dehazing network module of this invention includes a dehazing backbone network, a text perception guidance module, and a physical perception guidance module. The fogging network module includes a fogging backbone network, a text perception guidance module, and a physical perception guidance module. The text perception guidance module is introduced to improve the quality of dehazed images by using text-modal dehazing. The physical perception guidance module is introduced to comprehensively utilize the advantages of image dehazing algorithms based on prior information and atmospheric scattering models, making dehazing more effective, especially for images with high haze concentration. Both the dehazing backbone network and the fogging backbone network are stable cyclic diffusion models, thus fully utilizing the prior information contained in the stable cyclic diffusion model in the dehazing network based on the cyclic diffusion model, resulting in dehazed images with better visual quality. The dehazing network has good generalization ability.

[0090] 3. The addition of four zero-convolutional layers between the VAE encoder and VAE decoder in this invention is to ensure that the generated image retains the basic content and details of the original image, and to ensure that the details of the dehazed or hazy image are the same as those of the original image without distortion.

[0091] 4. This invention utilizes unpaired real foggy images and real clear images to train a dehazing network based on a cyclic diffusion model, enabling the unpaired method to process both real foggy and real clear images. This enhances the method's ability to process real foggy images and solves the current problem of difficulty in collecting large-scale paired real foggy and clear images, as well as the inability of dehazing real foggy images using synthesized paired foggy and clear images to effectively process real foggy images due to differences between real foggy images and synthetic foggy images.

[0092] In summary, the method of this invention is simple in steps and reasonable in design. It trains the dehazing network based on the cyclic diffusion model on unpaired real foggy images and real clear images. While ensuring the dehazing ability, it makes full use of the prior information contained in the stable cyclic diffusion model in the dehazing network, so that the generated dehazed images have better visual quality and the dehazing network has better generalization ability.

[0093] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0094] Figure 1 This is a flowchart of the method of the present invention.

[0095] Figure 2 This is a schematic diagram of the defogging network based on the cyclic diffusion model of the present invention.

[0096] Figure 3 This is a schematic diagram of the cyclic diffusion model of the present invention.

[0097] Figure 4 This is a schematic diagram of the physical perception guidance module of the present invention.

[0098] Figure 5 This is a schematic diagram of the physical fusion module of the present invention.

[0099] Figure 6 Images obtained by testing using the method of this invention. Detailed Implementation

[0100] like Figures 1 to 5 As shown, an unpaired image dehazing method based on a cyclic diffusion model according to the present invention includes the following steps:

[0101] Step 1: Obtaining the training set images:

[0102] Real foggy images were collected from the RTTS and URHI datasets in the RESIDE foggy image database, and real clear images were collected from the ADE20K database and the OTS dataset in the RESIDE foggy image database as training data; one real foggy image and one real clear image were used as a set of training data.

[0103] Step 2: Construct a dehazing network based on a cyclic diffusion model:

[0104] The defogging network based on the cyclic diffusion model includes a defogging branch network and a fogging branch network. The defogging branch network includes a first defogging network module, a first fogging network module, and a defogging discriminator. The fogging branch network includes a second fogging network module, a second defogging network module, and a fogging discriminator. The first defogging network module, the second defogging network module, the first fogging network module, and the second fogging network module have the same structure, and the defogging discriminator and the fogging discriminator have the same structure.

[0105] Both the first and second defogging network modules include a defogging backbone network, a text perception guidance module, and a physical perception guidance module;

[0106] Both the first and second fogging network modules include a fogging backbone network and a text perception guidance module; both the defogging backbone network and the fogging backbone network are stable cyclic diffusion models.

[0107] The defogging backbone network and the fogging backbone network have the same structure, and both include a VAE encoder, a UNet and a VAE decoder. Four zero convolutional layers are added between the VAE encoder and the VAE decoder.

[0108] Step 3: Feature extraction from real foggy images and real clear images:

[0109] Step 301: Process the real foggy image x through the text-aware guidance module in the first dehazing network module to obtain a foggy image with added text control conditions. Then, process the foggy image with added text control conditions through the dehazing backbone network in the first dehazing network module to obtain a clear image after dehazing.

[0110] The real foggy image x is processed by the text perception guidance module in the second fogging network module to obtain the input foggy image with added text control conditions. The input foggy image with added text control conditions is then passed through the fogging backbone network in the second fogging network module to obtain the output foggy image F(x).

[0111] The real foggy image x is passed through the physical perception guidance module in the first dehazing network module to obtain the first reconstructed foggy image I. phy ;

[0112] Clear image after dehazing After passing through the text-aware guidance module in the first fogging network module, a dehazed image with added text control conditions is obtained. This dehazed image with added text control conditions is then processed by the fogging backbone network in the first fogging network module to obtain a synthesized fogged image.

[0113] Step 302: The real, clear image y is processed by the text-aware guidance module in the second fogging network module to obtain a clear image with added text control conditions. The clear image with added text control conditions is then processed by the fogging backbone network in the second fogging network module to obtain the fogged image.

[0114] The real clear image y is processed by the text perception guidance module in the first dehazing network module to obtain the input clear image with added text control conditions. The input clear image with added text control conditions is then passed through the dehazing backbone network of the first dehazing network module to obtain the output clear image G(y).

[0115] Image after fogging After passing through the text-aware guidance module in the second dehazing network module, a hazy image with added text control conditions is obtained. This hazy image with added text control conditions is then processed by the dehazing backbone network in the second dehazing network module to obtain a synthesized dehazing image.

[0116] Image after fogging After passing through the physical perception guidance module in the second dehazing network module, the second reconstructed hazy image I′ is obtained. phy ;

[0117] Step 4: Establishing the total loss function:

[0118] Step 401, according to The generative adversarial loss L is obtained GAN ; where D Y (y) represents the value output by the dehazing discriminator for the true, clear image y. This indicates a clear image after dehazing. The value output by the defogging discriminator, D X (x) represents the value output by the fog discriminator for the real foggy image x. Image after fogging The value output by the fogging discriminator;

[0119] Step 402, according to Obtain the cycle-consistent loss L cyc ;in, Represents a real foggy image x and a synthetic foggy image. Manhattan distance between them Represents a real foggy image x and a synthetic foggy image. LPIPS distance between Represents the true sharp image y and the synthetic dehazed image. Manhattan distance between them Represents the true sharp image y and the synthetic dehazed image. LPIPS distance between them;

[0120] Step 403, according to L idt =L rec1 (G(y),y)+L rec2 (G(y),y)+L rec1 (F(x),x)+L rec2 (F(x),x), thus obtaining the identity loss L idt ; among which, L rec1 (G(y),y) represents the Manhattan distance between the true sharp image y and the output sharp image G(y), L rec2 (G(y),y) represents the LPIPS distance between the true sharp image y and the output sharp image G(y), L rec1 (F(x),x) represents the Manhattan distance between the real foggy image x and the output foggy image F(x), L rec2 (F(x),x) represents the LPIPS distance between the real foggy image x and the output foggy image F(x);

[0121] Step 404, according to L phy (1) = L phy +L p ′ hy The reconstruction loss L is obtained. phy(1); where L phy Let L represent the reconstruction loss of the first branch, and L phy =L rec1 (x,I phy )+L rec2 (x,I phy ), L rec1 (x,I phy ) represents the real foggy image x and the first reconstructed foggy image I. phy The Manhattan distance between them, L rec2 (x,I phy ) represents the real foggy image x and the first reconstructed foggy image I. phy LPIPS distance between; L p ′ hy This represents the loss from reconstructing the second branch, and Image after fogging Second reconstructed foggy image I′ phy Manhattan distance between them Image after fogging Second reconstructed foggy image I′ phy LPIPS distance between them;

[0122] Step 405, according to L loss =L GAN +L cyc +L idt +0.5L phy (1) , thus obtaining the total loss function L loss ;

[0123] Step 5: Training the dehazing network based on the cyclic diffusion model:

[0124] Step 501: The computer uses the Adam optimization algorithm. It inputs a set of training data and first utilizes the generative adversarial loss L... GAN The fog discriminator and the defogging discriminator are trained until the generative adversarial loss L is obtained. GAN Fog discrimination and defogging discrimination at maximum value;

[0125] Step 502: The computer uses the Adam optimization algorithm, inputs the training data, and, under the fog discriminator and defogging discriminator determined in step 501, uses the total loss function L... loss The dehazing network based on the recurrent diffusion model is trained until the total loss function L is reached. loss Minimum, complete training on this set of training data;

[0126] Step 503: Following the methods in steps 501 to 502, continue training until the entire training set is trained, completing one iteration of training;

[0127] Step 504: Repeat steps 501 to 503, iteratively training until the preset number of iterations is met, to obtain the trained dehazing network based on the cyclic diffusion model;

[0128] Step 6: Dehaze a single image using the trained dehazing network based on the recurrent diffusion model:

[0129] A computer is used to input any foggy image into the first defogging network module of the defogging branch network of a pre-trained defogging network based on a cyclic diffusion model, and the resulting clear image is obtained after defogging.

[0130] In this embodiment, in step two, the four zero convolutional layers added between the VAE encoder and the VAE decoder are the first zero convolutional layer, the second zero convolutional layer, the third zero convolutional layer, and the fourth zero convolutional layer.

[0131] The VAE encoder includes an input convolutional layer, a first DownEncoderBlock2D, a second DownEncoderBlock2D, a third DownEncoderBlock2D, and a fourth DownEncoderBlock2D;

[0132] The VAE decoder includes an output convolutional layer plus UNetMidBlock2D, a first UpDecoderBlock2D, a second UpDecoderBlock2D, a third UpDecoderBlock2D, and a fourth UpDecoderBlock2D;

[0133] A first zero-convolutional layer is set between the output of the input convolutional layer of the VAE encoder and the input of the fourth UpDecoderBlock2D; a second zero-convolutional layer is set between the output of the first DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a third zero-convolutional layer is set between the output of the second DownEncoderBlock2D and the input of the second UpDecoderBlock2D; and a fourth zero-convolutional layer is set between the output of the third DownEncoderBlock2D and the input of the first UpDecoderBlock2D.

[0134] In this embodiment, the processing methods for the following are the same: passing a foggy image with added text control conditions through the dehazing backbone network of the first dehazing network module; passing a foggy image with added text control conditions through the dehazing backbone network of the second dehazing network module; passing a clear input image with added text control conditions through the dehazing backbone network of the first dehazing network module; passing a dehazing image with added text control conditions through the fogging backbone network of the first fogging network module; passing a clear image with added text control conditions through the fogging backbone network of the second fogging network module; and passing a foggy input image with added text control conditions through the fogging backbone network of the second fogging network module. Specifically, the methods are as follows:

[0135] Step 3011: The foggy image with added text control conditions, the fogged image with added text control conditions, the clear input image with added text control conditions, the defogging image with added text control conditions, the clear image with added text control conditions, and the foggy input image with added text control conditions are all recorded as the input image and control text.

[0136] Step 3012: The input image is passed through a VAE encoder to obtain an encoded output feature map; wherein, the input image is input into the input convolutional layer of the VAE encoder to obtain a first intermediate feature map, and the first intermediate feature map is passed through the first DownEncoderBlock2D, the second DownEncoderBlock2D, the third DownEncoderBlock2D and the fourth DownEncoderBlock2D in sequence to obtain a second intermediate feature map, a third intermediate feature map, a fourth intermediate feature map and a fifth intermediate feature map;

[0137] Step 3013: The encoded output feature map and control text are processed by UNet to obtain the sixth intermediate feature map;

[0138] Step 3014: The first intermediate feature map is passed through the first zero convolutional layer to output the first transition feature map; the second intermediate feature map is passed through the second zero convolutional layer to output the second transition feature map; the third intermediate feature map is passed through the third zero convolutional layer to output the third transition feature map; and the fourth intermediate feature map is passed through the fourth zero convolutional layer to output the fourth transition feature map.

[0139] Step 3015: Pass the sixth intermediate feature map, the first transition feature map, the second transition feature map, the third transition feature map, and the fourth transition feature map through the VAE decoder to obtain the decoded output feature map; wherein, the sixth intermediate feature map is passed through the output convolutional layer of the VAE decoder and then processed by UNetMidBlock2D to obtain the seventh intermediate feature map; the seventh intermediate feature map and the fourth transition feature map are added together and then processed by the first UpDecoderBlock2D to obtain the eighth intermediate feature map; the eighth intermediate feature map and the third transition feature map are added together and then processed by the second UpDecoderBlock2D to obtain the ninth intermediate feature map; the ninth intermediate feature map and the second transition feature map are added together and then processed by the third UpDecoderBlock2D to obtain the tenth intermediate feature map; the tenth intermediate feature map and the first transition feature map are added together and then processed by the fourth UpDecoderBlock2D to obtain the eleventh intermediate feature map;

[0140] Step 3016: Record the decoded output feature map corresponding to the foggy image with added text control conditions as the clear image after dehazing. The decoded output feature map corresponding to the hazy image with added text control conditions is denoted as the synthesized dehazed image. The decoded output feature map corresponding to the input sharp image with added text control conditions is denoted as the output sharp image G(y);

[0141] The decoded output feature map corresponding to the dehazed image with added text control conditions is denoted as the synthesized hazy image. The decoded output feature map corresponding to the clear image with added text control conditions is denoted as the fogged image. The decoded output feature map corresponding to the output of the input foggy image with added text control conditions is denoted as the output foggy image F(x).

[0142] In this embodiment, the real foggy image x is processed by the text perception guidance module in the first dehazing network module, the real clear image y is processed by the text perception guidance module in the first dehazing network module, and the image after fogging is... The methods used in the text-aware guidance module of the second dehazing network module are all the same, as follows:

[0143] Step 3021: Combine the real foggy image x, the real clear image y, and the image after fogging. All are referred to as the first initial image;

[0144] Step 3022: Input the first initial image into the BLIP2 module to obtain the first text title;

[0145] Step 3023: Process the real foggy images in the training data using the text inversion method to obtain negative cue features representing fog;

[0146] Step 3024: After deleting haze-related words from the first text title, feed it into the CLIP model text encoder to obtain the text embedding representation features;

[0147] Step 3025: The text embedding representation features and the negative cue features representing fog are obtained by classifying the text without a classifier;

[0148] Step 3026: The text control obtained by processing the real foggy image x through the text perception guidance module in the first defogging network module and the real foggy image x are denoted as the foggy image with added text control conditions;

[0149] The text control and the true clear image y obtained by the text perception guidance module in the first dehazing network module are denoted as the input clear image with added text control conditions.

[0150] Image after fogging The image obtained after text-aware guidance by the text perception module in the second dehazing network module is both controlled and fogged. This is recorded as a fogged image with added text control conditions;

[0151] Clear image after dehazing The processing methods for the text-aware guidance module in the first fogging network module, the text-aware guidance module in the second fogging network module for the real fogged image x, and the text-aware guidance module in the real clear image y for the second fogging network module are all the same, as detailed below:

[0152] Step 30A: Obtain the clear image after dehazing. The real foggy image x and the real clear image y are denoted as the second initial image;

[0153] Step 30B: Input the second initial image into the BLIP2 module to obtain the second text title;

[0154] Step 30C: Add haze-related words to the second text title and then feed it into the CLIP model text encoder to obtain the second text embedding representation features;

[0155] Step 30D: The second text embedding representation features are guided by a classifier-free method to obtain the second text control;

[0156] Step 30E: Obtain the clear image after dehazing. The second text control and the clear image after defogging are obtained through the text-aware guidance module in the first fogging network module. This is denoted as a dehazed image with added text control conditions;

[0157] The real clear image y, after passing through the text perception guidance module in the second fogging network module, is used to obtain the second text control and the real clear image y, which are denoted as the input clear image with added text control conditions;

[0158] The second text control obtained by passing the real foggy image x through the text perception guidance module in the second fogging network module and the real foggy image x are denoted as the input foggy image with added text control conditions.

[0159] In this embodiment, the real foggy image x in step 301 is passed through the physical perception guidance module in the first dehazing network module to obtain the first reconstructed foggy image I. phy And the image after fogging in step 302 After passing through the physical perception guidance module in the second dehazing network module, the second reconstructed hazy image I′ is obtained. phy The method is the same, and the specific process is as follows:

[0160] Step A: The dark channel prior dehazing (DCP) algorithm is used to process the real hazy image x to obtain the first dehazing feature map J. dcp First transmission image t dcp and atmospheric light A;

[0161] The boundary constraint prior (BCCR) algorithm is used to process the real hazy image x to obtain the second dehazed feature map J. bccr Second transmission diagram t bccr ;

[0162] Step B: Use the computer to call the concatenate function to combine the real foggy image x and the first transmission image t. dcp Second transmission diagram t bccr After stitching, the transmission pattern is input into the transmission pattern optimization network to obtain the optimized transmission pattern t. ref ;

[0163] Step C: Combine the real foggy image x and the first dehazing feature map J. dcp Second defogging feature map J bccr After passing through the Physical Fusion Module (PFM), the fused feature map F is obtained. fuse ;

[0164] Step D: Fuse the feature map F fuse Inputting the dehazing backbone network yields the optimized dehazed image J. ref ;

[0165] Step F: Apply the Atmospheric Scattering Model (ASM) to the optimized dehazed image J. ref Optimized transmission map t refAfter processing with atmospheric light A, a reconstructed foggy image I is obtained. phy ;

[0166] Step G: Following the methods in steps A to F, process the image after fogging. After processing, the second reconstructed foggy image I′ is obtained. phy .

[0167] In this embodiment, step C is specifically performed as follows:

[0168] Step C01: Transfer the first dehazing feature map J dcp After passing through the first convolutional + ReLU activation function layer, the first feature map f1 is obtained;

[0169] The real foggy image x is passed through the first double-layer convolution module to obtain the second feature map f2;

[0170] The second defogging feature map J bccr After passing through the second convolutional + ReLU activation function layer, the third feature map f3 is obtained;

[0171] Step C02: Add the second feature map f2 to the first feature map f1 to obtain the fourth feature map f4; add the second feature map f2 to the third feature map f3 to obtain the fifth feature map f5.

[0172] Step C03: Pass the fourth feature map f4 through the third convolution + ReLU activation function layer to obtain the sixth feature map f6;

[0173] The fifth feature map f5 is passed through the fourth convolutional + ReLU activation function layer to obtain the seventh feature map f7;

[0174] Step C04: Perform a Hadamard product operation between the output of the sixth feature map f6 after passing through the first Sigmoid function layer and the fourth feature map f4 to obtain the eighth feature map f8;

[0175] The output of the seventh feature map f7 after passing through the second Sigmoid function layer is combined with the fifth feature map f5 through the Hadamard product to obtain the ninth feature map f9;

[0176] Step C05: Use the concatenate function to concatenate the eighth feature map f8 and the ninth feature map f9 to obtain the tenth feature map f. 10 And the tenth feature map f 10 After passing through a global adaptive pooling layer, a second double-layer convolutional module, and a Softmax layer, the eleventh feature map f is obtained. 11 ;

[0177] Step C06: Transfer the eleventh feature map f 11 With the tenth feature map f 10Features after Hadamard product operation and the tenth feature image f 10 Adding them together, we get the twelfth feature map f. 12 ;

[0178] Step C07: Use the concatenate function to concatenate the first dehazed feature map J. dcp Second defogging feature map J bccr and the twelfth feature diagram f 12 After concatenation, the resulting data is fed into an inductively coupled convolutional network to obtain the fused feature weights f. w ;

[0179] Step C08: Use a computer to calculate the formula. The fused feature map F is obtained. fuse ;in, This represents the Hadamard product operation between feature map matrices. f represents the addition operation between feature map matrices; w1 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the first channel, f w2 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the second channel, f w3 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the third channel, f w4 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the fourth channel; f 12-1 Represents the twelfth feature map f 12 Feature maps of the first three channels; f 12-2 Represents the twelfth feature map f 12 Feature maps of the middle and rear three channels.

[0180] In this embodiment, over 7,000 real foggy images were collected as training data from the Real-world Task-Driven Testing Set (RTTS) and Unannotated Real-world Hazy Images (URHI) datasets in the RESIDE foggy image database. Simultaneously, over 10,000 real, clear images were collected as training data from the ADE20K database and the Outdoor Training Set (OTS) dataset in the RESIDE foggy image database.

[0181] In this embodiment, Figure 2 The TAG on the top is for text-aware guidance modules, and the PAG is for physical-aware guidance modules.

[0182] In this embodiment, it should be noted that the defogging discriminator and the fogging discriminator have the same structure and are both based on the Vision-aided-gan network in the paper Ensembling Off-the-shelf Models for GAN Training, where cv_type is set to clip.

[0183] In this embodiment, it should be noted that the VAE encoder, UNet, and VAE decoder in the defogging backbone network and the fogging backbone network can refer to the conventional stable diffusion model 2.1 Turbo version.

[0184] In this embodiment, it should be noted that four zero-convolutional layers are added between the VAE encoder and the VAE decoder in order to ensure that the generated image retains the basic content and details of the original image, and to ensure that the details of the dehazed or hazy image are the same as those of the original image without distortion.

[0185] In this embodiment, the kernel size in the first zero convolutional layer, the second zero convolutional layer, the third zero convolutional layer, and the fourth zero convolutional layer is 1×1, the stride is 1, the padding is 0, and a zero initialization strategy is adopted; the number of kernels in the first zero convolutional layer is 256, and the number of kernels in the second zero convolutional layer, the third zero convolutional layer, and the fourth zero convolutional layer is 512 each;

[0186] The input convolutional layer has 128 kernels, a kernel size of 3×3, a stride of 1, and padding of 1.

[0187] The output convolutional layer has 512 kernels, a kernel size of 3×3, a stride of 1, and padding of 1.

[0188] In this embodiment, it should be noted that during training in step five, the fine-tuned layers in the VAE encoder and VAE decoder include "conv1", "conv2", "conv_in", "conv_shortcut", "conv", "conv_out", "skip_conv_1", "skip_conv_2", "skip_conv_3", "skip_conv_4", "to_k", "to_q", "to_v", and "to_out.0", with their rank set to 4 and the initial weight parameters set to "gaussian". For the UNet part, the fine-tuned layers include "to_k", "to_q", "to_v", "to_out.0", "conv", "conv1", "conv2", "conv_in", "conv_shortcut", "conv_out", "proj_out", "proj_in", "ff.net.2", and "ff.net.0.proj", all with a rank of 8 and initialized weights to "gaussian". Among these, "skip_conv_1", "skip_conv_2", "skip_conv_3", and "skip_conv_4" are the first, second, third, and fourth zero-convolutional layers, respectively.

[0189] In this embodiment, it should be noted that the BLIP2 module can be referred to in the paper Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.

[0190] For methods on text inversion, please refer to the paper "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion".

[0191] The CLIP model text encoder can be found in the paper Learning Transferable Visual Models From Natural Language Supervision.

[0192] For guidance on classifier-free Diffusion, please refer to the paper "Classifier-Free Diffusion Guidance".

[0193] In this embodiment, the size of the feature map is represented by the number of channels × length × width. In step 3011, the size of the input image is 3 × 256 × 256, the size of the first intermediate feature map is 128 × 256 × 256, the size of the second intermediate feature map is 128 × 128 × 128, the size of the third intermediate feature map is 256 × 64 × 64, the size of the fourth intermediate feature map is 512 × 32 × 32, and the size of the fifth intermediate feature map is 512 × 32 × 32. The encoded output feature map of the fifth intermediate feature map after passing through the subsequent modules of the VAE encoder is 4 × 32 × 32.

[0194] The size of the sixth intermediate feature map is 4×32×32, the size of the seventh intermediate feature map is 512×32×32, the size of the eighth intermediate feature map is 512×64×64, the size of the ninth intermediate feature map is 512×128×128, the size of the tenth intermediate feature map is 256×256×256, the size of the eleventh intermediate feature map is 128×256×256, and the size of the decoded output feature map of the eleventh intermediate feature map after passing through the subsequent modules of the VAE decoder is 3×256×256.

[0195] The size of the first transition feature map is 256×256×256, the size of the second transition feature map is 512×128×128, the size of the third transition feature map is 512×64×64, and the size of the fourth transition feature map is 512×32×32.

[0196] In this embodiment, it should be noted that the transillumination optimization network has the same structure as the transillumination optimization network in the paper RefineDNet: A Weakly Supervised Refinement Framework for Single Image Dehazing, except that the input channels of this invention are 5.

[0197] In this embodiment, it should be noted that in step D, the fused feature map F fuse The method for inputting the dehazing backbone network is the same as in steps 3012 to 3017; no control text needs to be input.

[0198] In this embodiment, the size of the feature map is represented by the number of channels × length × width. The size of the real foggy image x is 3 × 256 × 256, and the first dehazing feature map J... dcp Second defogging feature map J bccr The size is 3×256×256; the first transmission pattern t dcp Second transmission diagram t bccr The size is 1×256×256; the optimized transmission pattern t ref Its size is 1×256×256;

[0199] Optimized dehazed image J ref and reconstructed foggy image I phy Its size is 3×256×256;

[0200] In this embodiment, the size of the first feature map f1 to the ninth feature map f9 is 3×256×256, and the tenth feature map f... 10 The size of each feature map is 6×256×256, and the eleventh feature map is f. 11 The size of each feature is 6×1×1, and the twelfth feature map f 12 The size of each feature is 6×256×256, and the feature weights f after fusion are... w The size of each feature map is 4×256×256, and the fused feature map F fuse Its size is 3×256×256.

[0201] In this embodiment, the convolutions in the first convolution + ReLU activation function layer, the second convolution + ReLU activation function layer, the third convolution + ReLU activation function layer, and the fourth convolution + ReLU activation function layer are point convolutions with a kernel size of 1×1, a number of kernels of 3, a stride of 1, and padding of 0.

[0202] The first two-layer convolutional module includes convolutional layer 1, ReLU activation function and convolutional layer 2. The kernel size of convolutional layer 1 is 1×1, the number of kernels is 6, the stride is 1, and the padding is 0.

[0203] The kernel size of convolutional layer 2 is 1×1, the number of kernels is 3, the stride is 1, and the padding is 0;

[0204] The second double-layer convolutional module includes convolutional layer-1, ReLU activation function and convolutional layer-2. The kernel size of convolutional layer-1 is 1×1, the number of kernels is 6, the stride is 1, and the padding is 0.

[0205] The kernel size of convolutional layer-2 is 1×1, the number of kernels is 6, the stride is 1, and the padding is 0.

[0206] In a gated convolutional network, the kernel size is 3×3, the number of kernels is 4, the stride is 1, and the padding is 1.

[0207] In this embodiment, it should be noted that the preset number of iterations in step 502 is 10.

[0208] In this embodiment, as Figure 6The top row is a foggy image. Using the method of this invention, a clear image after defogging is obtained in the bottom row. Therefore, this invention can significantly improve the defogging effect of foggy images and also has high defogging quality in real foggy images.

[0209] In summary, the method of the present invention is simple in steps and reasonable in design. The image dehazing method based on unpaired training can effectively utilize training data from real scenes, thereby enabling the trained dehazing model to have good generalization ability and resulting in better visual quality of the generated dehazed images.

[0210] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for dehazing unpaired images based on a cyclic diffusion model, characterized in that, The method includes the following steps: Step 1: Obtaining the training set images: Real foggy images were collected from the RTTS and URHI datasets in the RESIDE foggy image database, and real clear images were collected from the ADE20K database and the OTS dataset in the RESIDE foggy image database as training data; one real foggy image and one real clear image were used as a set of training data. Step 2: Construct a dehazing network based on a cyclic diffusion model: The defogging network based on the cyclic diffusion model includes a defogging branch network and a fogging branch network. The defogging branch network includes a first defogging network module, a first fogging network module, and a defogging discriminator. The fogging branch network includes a second fogging network module, a second defogging network module, and a fogging discriminator. The first defogging network module, the second defogging network module, the first fogging network module, and the second fogging network module have the same structure, and the defogging discriminator and the fogging discriminator have the same structure. Both the first and second defogging network modules include a defogging backbone network, a text perception guidance module, and a physical perception guidance module; Both the first and second fogging network modules include a fogging backbone network and a text perception guidance module; both the defogging backbone network and the fogging backbone network are stable cyclic diffusion models. The defogging backbone network and the fogging backbone network have the same structure, and both include a VAE encoder, a UNet and a VAE decoder. Four zero convolutional layers are added between the VAE encoder and the VAE decoder. Step 3: Feature extraction from real foggy images and real clear images: Step 301: Process the real foggy image x through the text-aware guidance module in the first dehazing network module to obtain a foggy image with added text control conditions. Then, process the foggy image with added text control conditions through the dehazing backbone network in the first dehazing network module to obtain a clear image after dehazing. The real foggy image x is processed by the text perception guidance module in the second fogging network module to obtain the input foggy image with added text control conditions. The input foggy image with added text control conditions is then passed through the fogging backbone network in the second fogging network module to obtain the output foggy image F(x). The real foggy image x is passed through the physical perception guidance module in the first dehazing network module to obtain the first reconstructed foggy image I. phy ; Clear image after dehazing After passing through the text-aware guidance module in the first fogging network module, a dehazed image with added text control conditions is obtained. This dehazed image with added text control conditions is then processed by the fogging backbone network in the first fogging network module to obtain a synthesized fogged image. Step 302: The real, clear image y is processed by the text-aware guidance module in the second fogging network module to obtain a clear image with added text control conditions. The clear image with added text control conditions is then processed by the fogging backbone network in the second fogging network module to obtain the fogged image. The real clear image y is processed by the text perception guidance module in the first dehazing network module to obtain the input clear image with added text control conditions. The input clear image with added text control conditions is then passed through the dehazing backbone network of the first dehazing network module to obtain the output clear image G(y). Image after fogging After passing through the text-aware guidance module in the second dehazing network module, a hazy image with added text control conditions is obtained. This hazy image with added text control conditions is then processed by the dehazing backbone network in the second dehazing network module to obtain a synthesized dehazing image. Image after fogging After passing through the physical perception guidance module in the second dehazing network module, the second reconstructed hazy image I′ is obtained. phy ; Step 4: Establishing the total loss function: Step 401, according to The generative adversarial loss L is obtained GAN ; where D Y (y) represents the value output by the dehazing discriminator for the true, clear image y. This indicates a clear image after dehazing. The value output by the defogging discriminator, D X (x) represents the value output by the fog discriminator for the real foggy image x. Image after fogging The value output by the fogging discriminator; Step 402, according to Obtain the cycle-consistent loss L cyc ;in, Represents a real foggy image x and a synthetic foggy image. Manhattan distance between them Represents a real foggy image x and a synthetic foggy image. LPIPS distance between Represents the true sharp image y and the synthetic dehazed image. Manhattan distance between them Represents the true sharp image y and the synthetic dehazed image. LPIPS distance between them; Step 403, according to L idt =L rec1 (G(y),y)+L rec2 (G(y),y)+L rec1 (F(x),x)+L rec2 (F(x),x), thus obtaining the identity loss L idt Among them, L rec1 (G(y),y) represents the Manhattan distance between the true sharp image y and the output sharp image G(y), L rec2 (G(y),y) represents the LPIPS distance between the true sharp image y and the output sharp image G(y), L rec1 (F(x),x) represents the Manhattan distance between the real foggy image x and the output foggy image F(x), L rec2 (F(x),x) represents the LPIPS distance between the real foggy image x and the output foggy image F(x); Step 404, according to L phy (1) = L phy +L′ phy The reconstruction loss L is obtained. phy (1); where L phy Let L represent the reconstruction loss of the first branch, and L phy =L rec1 (x,I phy )+L rec2 (x,I phy ), L rec1 (x,I phy ) represents the real foggy image x and the first reconstructed foggy image I. phy The Manhattan distance between them, L rec2 (x,I phy ) represents the real foggy image x and the first reconstructed foggy image I. phy LPIPS distance between; L p ′ hy This represents the loss from reconstructing the second branch, and Image after fogging Second reconstructed foggy image I′ phy Manhattan distance between them Image after fogging Second reconstructed foggy image I′ phy LPIPS distance between them; Step 405, according to L loss =L GAN +L cyc +L idt +0.5L phy (1) , thus obtaining the total loss function L loss ; Step 5: Training the dehazing network based on the cyclic diffusion model: Step 501: The computer uses the Adam optimization algorithm. It inputs a set of training data and first utilizes the generative adversarial loss L... GAN The fog discriminator and the defogging discriminator are trained until the generative adversarial loss L is obtained. GAN Fog discrimination and defogging discrimination at maximum value; Step 502: The computer uses the Adam optimization algorithm, inputs the training data, and, under the fog discriminator and defogging discriminator determined in step 501, uses the total loss function L... loss The dehazing network based on the recurrent diffusion model is trained until the total loss function L is reached. loss Minimum, complete training on this set of training data; Step 503: Following the methods in steps 501 to 502, continue training until the entire training set is trained, completing one iteration of training; Step 504: Repeat steps 501 to 503, iteratively training until the preset number of iterations is met, to obtain the trained dehazing network based on the cyclic diffusion model; Step 6: Dehaze a single image using the trained dehazing network based on the recurrent diffusion model: A computer is used to input any foggy image into the first defogging network module of the defogging branch network of a pre-trained defogging network based on a cyclic diffusion model, and the resulting clear image is obtained after defogging.

2. The unpaired image dehazing method based on a cyclic diffusion model according to claim 1, characterized in that: In step two, the four zero-convolutional layers added between the VAE encoder and the VAE decoder are the first zero-convolutional layer, the second zero-convolutional layer, the third zero-convolutional layer, and the fourth zero-convolutional layer. The VAE encoder includes an input convolutional layer, a first DownEncoderBlock2D, a second DownEncoderBlock2D, a third DownEncoderBlock2D, and a fourth DownEncoderBlock2D; The VAE decoder includes an output convolutional layer plus UNetMidBlock2D, a first UpDecoderBlock2D, a second UpDecoderBlock2D, a third UpDecoderBlock2D, and a fourth UpDecoderBlock2D; A first zero-convolutional layer is set between the output of the input convolutional layer of the VAE encoder and the input of the fourth UpDecoderBlock2D; a second zero-convolutional layer is set between the output of the first DownEncoderBlock2D and the input of the third UpDecoderBlock2D; a third zero-convolutional layer is set between the output of the second DownEncoderBlock2D and the input of the second UpDecoderBlock2D; and a fourth zero-convolutional layer is set between the output of the third DownEncoderBlock2D and the input of the first UpDecoderBlock2D.

3. The unpaired image dehazing method based on a cyclic diffusion model according to claim 1, characterized in that: The processing methods for the following images are the same: a foggy image with added text control conditions passes through the dehazing backbone network in the first dehazing network module; a fogged image with added text control conditions passes through the dehazing backbone network in the second dehazing network module; a clear input image with added text control conditions passes through the dehazing backbone network in the first dehazing network module; a dehazed image with added text control conditions passes through the fogging backbone network in the first fogging network module; a clear image with added text control conditions passes through the fogging backbone network in the second fogging network module; and a foggy input image with added text control conditions passes through the fogging backbone network in the second fogging network module. Specifically, the following methods are used: Step 3011: The foggy image with added text control conditions, the fogged image with added text control conditions, the clear input image with added text control conditions, the defogging image with added text control conditions, the clear image with added text control conditions, and the foggy input image with added text control conditions are all recorded as the input image and control text. Step 3012: The input image is passed through a VAE encoder to obtain an encoded output feature map; wherein, the input image is input into the input convolutional layer of the VAE encoder to obtain a first intermediate feature map, and the first intermediate feature map is passed through the first DownEncoderBlock2D, the second DownEncoderBlock2D, the third DownEncoderBlock2D and the fourth DownEncoderBlock2D in sequence to obtain a second intermediate feature map, a third intermediate feature map, a fourth intermediate feature map and a fifth intermediate feature map; Step 3013: The encoded output feature map and control text are processed by UNet to obtain the sixth intermediate feature map; Step 3014: The first intermediate feature map is passed through the first zero convolutional layer to output the first transition feature map; the second intermediate feature map is passed through the second zero convolutional layer to output the second transition feature map; the third intermediate feature map is passed through the third zero convolutional layer to output the third transition feature map; and the fourth intermediate feature map is passed through the fourth zero convolutional layer to output the fourth transition feature map. Step 3015: Pass the sixth intermediate feature map, the first transition feature map, the second transition feature map, the third transition feature map, and the fourth transition feature map through the VAE decoder to obtain the decoded output feature map; wherein, the sixth intermediate feature map is passed through the output convolutional layer of the VAE decoder and then processed by UNetMidBlock2D to obtain the seventh intermediate feature map; the seventh intermediate feature map and the fourth transition feature map are added together and then processed by the first UpDecoderBlock2D to obtain the eighth intermediate feature map; the eighth intermediate feature map and the third transition feature map are added together and then processed by the second UpDecoderBlock2D to obtain the ninth intermediate feature map; the ninth intermediate feature map and the second transition feature map are added together and then processed by the third UpDecoderBlock2D to obtain the tenth intermediate feature map; the tenth intermediate feature map and the first transition feature map are added together and then processed by the fourth UpDecoderBlock2D to obtain the eleventh intermediate feature map; Step 3016: Record the decoded output feature map corresponding to the foggy image with added text control conditions as the clear image after dehazing. The decoded output feature map corresponding to the hazy image with added text control conditions is denoted as the synthesized dehazed image. The decoded output feature map corresponding to the input sharp image with added text control conditions is denoted as the output sharp image G(y); The decoded output feature map corresponding to the dehazed image with added text control conditions is denoted as the synthesized hazy image. The decoded output feature map corresponding to the clear image with added text control conditions is denoted as the fogged image. The decoded output feature map corresponding to the output of the input foggy image with added text control conditions is denoted as the output foggy image F(x).

4. The unpaired image dehazing method based on a cyclic diffusion model according to claim 1, characterized in that: A real, foggy image x is processed by the text-aware guidance module in the first dehazing network module; a real, clear image y is processed by the text-aware guidance module in the first dehazing network module; and the image after fogging is formed. The methods used in the text-aware guidance module of the second dehazing network module are all the same, as follows: Step 3021: Combine the real foggy image x, the real clear image y, and the image after fogging. All are referred to as the first initial image; Step 3022: Input the first initial image into the BLIP2 module to obtain the first text title; Step 3023: Process the real foggy images in the training data using the text inversion method to obtain negative cue features representing fog; Step 3024: After deleting haze-related words from the first text title, feed it into the CLIP model text encoder to obtain the text embedding representation features; Step 3025: The text embedding representation features and the negative cue features representing fog are obtained by classifying the text without a classifier; Step 3026: The text control obtained by processing the real foggy image x through the text perception guidance module in the first defogging network module and the real foggy image x are denoted as the foggy image with added text control conditions; The text control and the true clear image y obtained by the text perception guidance module in the first dehazing network module are denoted as the input clear image with added text control conditions. Image after fogging The image obtained after text-aware guidance by the text perception module in the second dehazing network module is both controlled and fogged. This is recorded as a fogged image with added text control conditions; Clear image after dehazing The processing methods for the text-aware guidance module in the first fogging network module, the text-aware guidance module in the second fogging network module for the real fogged image x, and the text-aware guidance module in the real clear image y for the second fogging network module are all the same, as detailed below: Step 30A: Obtain the clear image after dehazing. The real foggy image x and the real clear image y are denoted as the second initial image; Step 30B: Input the second initial image into the BLIP2 module to obtain the second text title; Step 30C: Add haze-related words to the second text title and then feed it into the CLIP model text encoder to obtain the second text embedding representation features; Step 30D: The second text embedding representation features are guided by a classifier-free method to obtain the second text control; Step 30E: Obtain the clear image after dehazing. The second text control and the clear image after defogging are obtained through the text-aware guidance module in the first fogging network module. This is denoted as a dehazed image with added text control conditions; The real clear image y, after passing through the text perception guidance module in the second fogging network module, is used to obtain the second text control and the real clear image y, which are denoted as the input clear image with added text control conditions; The second text control obtained by passing the real foggy image x through the text perception guidance module in the second fogging network module and the real foggy image x are denoted as the input foggy image with added text control conditions.

5. The unpaired image dehazing method based on a cyclic diffusion model according to claim 1, characterized in that: In step 301, the real foggy image x is processed by the physical perception guidance module in the first dehazing network module to obtain the first reconstructed foggy image I. phy And the image after fogging in step 302 After passing through the physical perception guidance module in the second dehazing network module, the second reconstructed hazy image I′ is obtained. phy The method is the same, and the specific process is as follows: Step A: The dark channel prior dehazing (DCP) algorithm is used to process the real hazy image x to obtain the first dehazing feature map J. dcp First transmission image t dcp and atmospheric light A; The boundary constraint prior (BCCR) algorithm is used to process the real hazy image x to obtain the second dehazing feature map J. bccr Second transmission diagram t bccr ; Step B: Use the computer to call the concatenate function to combine the real foggy image x and the first transmission image t. dcp Second transmission diagram t bccr After stitching, the transmission pattern is input into the transmission pattern optimization network to obtain the optimized transmission pattern t. ref ; Step C: Combine the real foggy image x and the first dehazing feature map J. dcp Second defogging feature map J bccr After passing through the Physical Fusion Module (PFM), the fused feature map F is obtained. fuse ; Step D: Fuse the feature map F fuse Inputting the dehazing backbone network yields the optimized dehazed image J. ref ; Step F: Apply the Atmospheric Scattering Model (ASM) to the optimized dehazed image J. ref Optimized transmission map t ref After processing with atmospheric light A, a reconstructed foggy image I is obtained. phy ; Step G: Following the methods in steps A to F, process the image after fogging. After processing, the second reconstructed foggy image I′ is obtained. phy .

6. The unpaired image dehazing method based on a cyclic diffusion model according to claim 5, characterized in that: Step C, the specific process is as follows: Step C01: Transfer the first dehazing feature map J dcp After passing through the first convolutional + ReLU activation function layer, the first feature map f1 is obtained; The real foggy image x is passed through the first double-layer convolution module to obtain the second feature map f2; The second defogging feature map J bccr After passing through the second convolutional + ReLU activation function layer, the third feature map f3 is obtained; Step C02: Add the second feature map f2 to the first feature map f1 to obtain the fourth feature map f4; add the second feature map f2 to the third feature map f3 to obtain the fifth feature map f5. Step C03: Pass the fourth feature map f4 through the third convolution + ReLU activation function layer to obtain the sixth feature map f6; The fifth feature map f5 is passed through the fourth convolutional + ReLU activation function layer to obtain the seventh feature map f7; Step C04: Perform a Hadamard product operation between the output of the sixth feature map f6 after passing through the first Sigmoid function layer and the fourth feature map f4 to obtain the eighth feature map f8; The output of the seventh feature map f7 after passing through the second Sigmoid function layer is combined with the fifth feature map f5 through the Hadamard product to obtain the ninth feature map f9; Step C05: Use the concatenate function to concatenate the eighth feature map f8 and the ninth feature map f9 to obtain the tenth feature map f. 10 And the tenth feature map f 10 After passing through a global adaptive pooling layer, a second double-layer convolutional module, and a Softmax layer, the eleventh feature map f is obtained. 11 ; Step C06: Transfer the eleventh feature map f 11 With the tenth feature map f 10 Features after Hadamard product operation and the tenth feature image f 10 Adding them together, we get the twelfth feature map f. 12 ; Step C07: Use the concatenate function to concatenate the first dehazed feature map J. dcp Second defogging feature map J bccr and the twelfth feature diagram f 12 After concatenation, the resulting data is fed into an inductively coupled convolutional network to obtain the fused feature weights f. w ; Step C08: Use a computer to calculate the formula. The fused feature map F is obtained. fuse ;in, This represents the Hadamard product operation between feature map matrices. f represents the addition operation between feature map matrices; w1 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the first channel, f w2 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the second channel, f w3 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the third channel, f w4 The fused feature weights f represent the weights f of the fused features. w The eigenvalue of the fourth channel; f 12-1 Represents the twelfth feature map f 12 Feature maps of the first three channels; f 12-2 Represents the twelfth feature map f 12 Feature maps of the middle and rear three channels.

Images