Image defogging method, device and equipment based on diffusion model

Abstract

The invention discloses an image defogging method, device and equipment based on a diffusion model, and the method comprises the steps: inputting a to-be-processed foggy image into a VAE image encoder, and obtaining an original image latent code; inputting the foggy image into a depth feature extractor to obtain depth features; inputting the description text into a text encoder to obtain text features; inputting the original image latent code, the depth feature and the text feature into a UNet de-noising device to obtain a de-noised latent code; and inputting the de-noised latent codes and the depth features into a VAE image decoder, and outputting a defogged fogless image. Visual information of an image, depth-of-field structure information of depth and semantic information of a text can be fully utilized, so that the defogging process is guided more accurately, and the defogging effect and color contrast reality of image defogging are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image defogging method, device and equipment based on a diffusion model. Background Art

[0002] In low-visibility conditions, such as fog, the contrast and color characteristics of target objects in an image are significantly attenuated by the fog, resulting in a decrease in image quality and difficulty in accurately extracting image features required in image processing and computer vision tasks. Image dehazing technology aims to remove the effects of fog from images through algorithmic means, restoring clear image details and colors. This technology is of great significance for improving image quality, enhancing visual effects, and satisfying applications such as image restoration and beautification.

[0003] Existing image dehazing methods, such as the traditional dark channel dehazing algorithm, can remove haze but are prone to incomplete removal. Deep learning methods based on CNN (Convolutional Neural Network) greatly improve dehazing through data-driven methods, but still struggle to effectively remove haze in complex scenes. Methods based on GAN (Generative Adversarial Network) can leverage the powerful generation capabilities of GAN to improve image dehazing results, but are prone to color cast, lack of realism, and fail to meet user needs. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to propose an image defogging method, device and equipment based on a diffusion model, aiming to solve the problems of existing image defogging methods such as poor defogging effect in complex scenes, easy color cast and lack of realism.

[0005] To achieve the above objectives, the present invention provides an image dehazing method based on a diffusion model. The method is implemented based on an image dehazing model constructed based on Stable Diffusion. The network structure of the image dehazing model includes a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser and a VAE image decoder. The method includes:

[0006] Input the foggy image to be processed into the VAE image encoder to obtain the original image latent code;

[0007] Input the foggy image into the depth feature extractor to obtain the depth feature;

[0008] Input the description text into the text encoder to obtain text features;

[0009] Input the original image latent code, depth features, and text features into the UNet denoiser to obtain the denoised latent code;

[0010] The denoising latent code and deep features are input into the VAE image decoder, and the dehazed image is output.

[0011] Preferably, the training process of the image defogging model includes:

[0012] Constructing image pairing data, the image pairing data including a non-fog original image, a foggy sample image generated by fogging and degrading the non-fog original image, and text corresponding to the non-fog original image;

[0013] The image pairing data is input into the network structure of the image dehazing model for joint training, and the model parameters are optimized through a preset loss function to obtain the image dehazing model, wherein the loss function includes noise loss, L1 loss and depth loss. The noise loss is the L2 loss of the noise predicted by the UNet denoiser and the preset noise, the L1 loss is the loss between the generated image and the haze-free original image, and the depth loss is the L2 loss of the depth map corresponding to the generated image and the haze-free original image.

[0014] Preferably, the VAE image encoder is constructed based on the VAE encoder structure of Stable Diffusion and is fine-tuned during the training process.

[0015] Preferably, the text encoder is constructed using Alpha-CLIP.

[0016] Preferably, the deep feature extractor is constructed using a pre-trained DepthPro model.

[0017] Preferably, the UNet denoiser is constructed based on UNet with pre-trained weights in Stable Diffusion and fine-tuned using LoRA.

[0018] Preferably, the VAE image decoder is constructed using the VAE decoder structure with pre-trained weights in Stable Diffusion.

[0019] To achieve the above objectives, the present invention provides an image defogging device based on a diffusion model. The device is implemented based on an image defogging model constructed based on Stable Diffusion. The network structure of the image defogging model includes a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser and a VAE image decoder. The device includes:

[0020] The encoding unit is used to input the foggy image to be processed into the VAE image encoder to obtain the original image latent code;

[0021] A first extraction unit is used to input the foggy image into a depth feature extractor to obtain depth features;

[0022] The second extraction unit is used to input the description text into a text encoder to obtain text features;

[0023] The denoising unit is used to input the original image latent code, deep features and text features into the UNet denoiser to obtain the denoised latent code;

[0024] The decoding unit is used to input the denoising latent code and deep features into the VAE image decoder and output the defogging fog-free image.

[0025] In order to achieve the above-mentioned objectives, the present invention also proposes an image defogging device based on a diffusion model, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of an image defogging method based on a diffusion model as described in the above-mentioned embodiment.

[0026] In order to achieve the above objectives, the present invention also proposes a computer-readable storage medium, on which a computer program is stored. The computer program is executed by a processor to implement the steps of an image defogging method based on a diffusion model as described in the above embodiment.

[0027] To achieve the above objectives, the present invention further proposes a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the image defogging method based on a diffusion model as described in the above embodiment.

[0028] Beneficial effects:

[0029] The above solution uses the powerful diffusion prior of the Diffusion Model to design the network, adding text semantic guidance while also incorporating depth maps for depth of field structure control. It also uses the multimodal prior knowledge of the fusion of text, image, and depth maps to help build a powerful generative network that can fully utilize the visual information of the image, the depth of field structure information, and the semantic information of the text, thereby more accurately guiding the dehazing process and improving the dehazing effect and color contrast realism of the image.

[0030] By constructing image pairings containing fog-free original images, foggy sample images, and corresponding text, and feeding this data into the network for joint training, the model can simultaneously learn the image's visual features, depth structure, and text semantic information during training, thereby better performing dehazing. Furthermore, optimizing the model using a comprehensive loss function comprising noise loss, L1 loss, and depth loss constrains the model's learning process from multiple dimensions, achieving a better balance between dehazing effectiveness, image detail preservation, and depth structure consistency.

[0031] The VAE image encoder built based on the Stable Diffusion VAE encoder structure can fully utilize the feature representation capabilities learned by the pre-trained model on large-scale data. It can quickly obtain better feature extraction performance with less training data and computing resources. At the same time, it can further adapt to specific dehazing tasks through fine-tuning, thereby improving the accuracy and efficiency of the model.

[0032] The text encoder built using Alpha-CLIP can effectively encode the input text into a feature vector with rich semantic information. The text features can be effectively integrated and interacted with the image features, thereby better guiding the UNet denoiser for image generation, making the dehazed image more consistent with the semantic requirements of the text description, and improving the semantic consistency and realism of the image.

[0033] The deep feature extractor built based on the DepthPro model can accurately extract the depth of field features of foggy images. These depth features can be fused with the UNet denoiser to effectively control the depth of field structure of the generated image, making it consistent with the original foggy image, helping to preserve the three-dimensional and spatial sense of the image, and making the defogged image more visually natural and realistic.

[0034] The UNet denoiser, built on Stable Diffusion's UNet, is suitable for image denoising due to its multi-scale network structure. Through LoRA fine-tuning, the model's denoising performance and detail retention capabilities can be further improved while keeping the number of model parameters low, allowing the dehazed image to better retain the image's texture and detail information while removing noise.

[0035] The VAE image decoder is built using the VAE decoder structure with pre-trained weights in Stable Diffusion. It can fully utilize the powerful image generation capabilities of the pre-trained model and effectively decode the denoised latent codes and deep features into high-quality haze-free images. After defogging, the image is not only visually clearer, but also closer to the real haze-free scene in terms of color and contrast. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of an image defogging method based on a diffusion model is provided in accordance with an embodiment of the present invention.

[0038] Figure 2 A schematic diagram of the overall processing of an image defogging model provided by one embodiment of the present invention.

[0039] Figure 3 A structural schematic diagram of an image defogging device based on a diffusion model provided by one embodiment of the present invention.

[0040] The realization of the objectives of the invention, the functional features and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention for which protection is sought, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] The present invention is described in detail below with reference to the embodiments.

[0043] Reference Figure 1 FIG2 is a flow chart of an image defogging method based on a diffusion model provided by an embodiment of the present invention.

[0044] In this embodiment, the method is implemented based on an image dehazing model constructed based on Stable Diffusion. The network structure of the image dehazing model includes a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser, and a VAE image decoder. The method includes:

[0045] S11, input the foggy image to be processed into the VAE image encoder to obtain the original image latent code;

[0046] S12, inputting the foggy image into a depth feature extractor to obtain depth features;

[0047] S13, inputting the description text into a text encoder to obtain text features;

[0048] S14, inputting the original image latent code, deep features, and text features into the UNet denoiser to obtain a denoised latent code;

[0049] S15, input the denoising latent code and deep features into the VAE image decoder, and output the defogging fog-free image.

[0050] In this embodiment, a foggy image to be processed is obtained, and then the foggy image is input into the image defogging model, and the defogging fog-free image is output. The network structure of the image defogging model includes a VAE (Variational Auto-Encoder) image encoder, a text encoder, a deep feature extractor, a UNet (U-shaped network, based on the convolutional neural network (CNN) architecture) denoiser, and a VAE image decoder. Figure 2 As shown in the figure, the foggy image is first input into the VAE image encoder to obtain the original image latent code corresponding to the foggy image; then the original image latent code is input into the UNet denoiser for iterative diffusion denoising; at the same time, the foggy image is input into the deep feature extractor in parallel with the UNet denoiser to obtain deep features for controlling the deep structure of the UNet denoiser; in addition, the descriptive text is input into the text encoder to obtain text features for text semantic control of the UNet denoiser; finally, the original image latent code output by the UNet denoiser and the deep features corresponding to the foggy image are fused and input into the VAE image decoder to output the final fog-free image.

[0051] Furthermore, the VAE image encoder is constructed based on the VAE encoder structure of Stable Diffusion and is fine-tuned during the training process.

[0052] Furthermore, the text encoder is constructed using Alpha-CLIP.

[0053] Furthermore, the deep feature extractor is constructed using a pre-trained DepthPro model.

[0054] Furthermore, the UNet denoiser is built based on the UNet with pre-trained weights in Stable Diffusion and fine-tuned using LoRA.

[0055] Furthermore, the VAE image decoder is constructed using the VAE decoder structure with pre-trained weights in Stable Diffusion.

[0056] In step S14, it further includes: inputting the original image latent code, deep features and text features into the UNet denoiser, and interacting and fusing them through the cross-attention mechanism.

[0057] In this embodiment, the VAE image encoder is constructed using the VAE encoder structure in the Stable Diffusion model. The pre-trained model is used for initialization and fine-tuned during training. The formula can be expressed as:

[0058] Z hazy =E(X hazy );

[0059] Among them, X hazy represents a foggy image, Z hazy Represents the original image latent code extracted by the VAE image encoder E.

[0060] The text encoder is built using Alpha-CLIP (ACLIP Model Focusing on Wherever You Want, a multimodal alignment model). Compared to CLIP, Alpha-CLIP has more precise text-image alignment capabilities and can identify the relationship between fog concentration and scene depth at a finer granularity, improving dehazing accuracy. Its formula can be expressed as:

[0061] F text =Alpha-CLIP(text);

[0062] Among them, text represents the description text corresponding to the image, F text Represents the text features obtained through Alpha-CLIP. The text features are fused with the UNet denoiser through a cross-attention mechanism to guide the denoising process.

[0063] The depth feature extractor is constructed using the pre-trained DepthPro model (Sharp Monocular Metric Depthin Less Than a Second, depth estimation model) to extract the depth field features of the foggy image and improve the consistency of the depth field structure between the generated fog-free image and the original foggy image. Its formula can be expressed as:

[0064] X depth =DepthPro(X hazy );

[0065] Among them, X depth Represents the deep features of the foggy image obtained by the deep feature extractor. The deep features are fused with the UNet denoiser through a cross-attention mechanism to guide the denoising process.

[0066] The UNet denoiser is built using the pre-trained UNet from Stable Diffusion and fine-tuned using LoRA (Low-Rank Adaptation of Large Language Models). UNet's multi-scale network structure makes it a natural denoiser, facilitating the iterative diffusion of noise in the latent space. Specifically, at the input, the initialisation noise is fused with the latent code of the foggy image, preserving the texture information of the foggy image. This can be expressed as:

[0067] Z=Concat(Z hazy ,Z noise );

[0068] Among them, Z noise Represents the noise of the initial setting, Concat represents the channel-level stacking operation, and Z represents the denoised latent code obtained by the Concat operation. The formula for the latent space diffusion iteration can be expressed as:

[0069] Z t-1 =Z t -∈ θ (Z t ,t,X hazy ,F text ,X depth );

[0070] Among them, ∈ θ (·) represents the noise predicted by the UNet denoiser, Z t is the latent code at the current step length t, Z t-1 is the latent code at the previous step length t-1. After the iteration is completed, the latent code Z0 at the step length 0 can be obtained.

[0071] LoRA, or residual low-rank decomposition, is a low-parameter training tool that fine-tunes the original model weights through a low-rank spatial matrix. The formula for LoRA fine-tuning can be expressed as:

[0072] W new =W+ΔW=W+AB;

[0073] Among them, W new Represents the weight parameter after fine-tuning, W represents the weight parameter of the original pre-training model, ΔW represents the weight parameter used for fine-tuning, and A and B represent the low-rank matrices used for fine-tuning.

[0074] In particular, the module branches of the text encoder and deep feature extractor serve as auxiliary branches of the UNet denoiser in the diffusion model. Through the cross-attention mechanism, the text features and deep features of different modalities interact and further fuse with the latent features of the foggy image in the intermediate layer of the UNet denoiser, thereby improving the final image defogging effect. The cross-attention feature fusion formula of the i-th intermediate layer of the UNet denoiser can be specifically expressed as:

[0075]

[0076] Among them, Z (i) represents the latent features obtained by fusion of the i-th intermediate layer of the UNet denoiser. Specifically, Special and represents the weight matrix in the cross attention layer, where d k is the set scaling factor, and T is the transposition operation.

[0077] and Special, and Represents the weight matrix in the crisscross attention layer.

[0078] In the feature fusion formula of the above cross attention mechanism, we can see that Atten text and Atten depth The two cross-attention layers share Q, thereby increasing the reuse of latent features, improving processing efficiency while ensuring the interaction of latent features of multiple modalities. Furthermore, the weight parameter λ is set empirically to 0.6.

[0079] The VAE image decoder is constructed using the VAE decoder structure with pre-trained weights in Stable Diffusion, and its formula can be expressed as:

[0080]

[0081] in, represents the haze-free image decoded by the VAE image decoder module D.

[0082] Furthermore, the training process of the image defogging model includes:

[0083] Constructing image pairing data, the image pairing data including a non-fog original image, a foggy sample image generated by fogging and degrading the non-fog original image, and text corresponding to the non-fog original image;

[0084] The image pairing data is input into the network structure of the image dehazing model for joint training, and the model parameters are optimized through a preset loss function to obtain the image dehazing model, wherein the loss function includes noise loss, L1 loss and depth loss. The noise loss is the L2 loss of the noise predicted by the UNet denoiser and the preset noise, the L1 loss is the loss between the generated image and the haze-free original image, and the depth loss is the L2 loss of the depth map corresponding to the generated image and the haze-free original image.

[0085] In this embodiment, image pairing data is constructed and preprocessed. Specifically, images containing scenes such as landscapes and portraits are selected as fog-free original images. Foggy sample images are obtained by fogging the fog-free original images. Paired data of the fog-free original images, foggy sample images, and text descriptions corresponding to the fog-free original images are constructed. The image data is preprocessed with random geometric transformations such as flipping, translation, rotation, and scaling to improve the usability of image data during network training.

[0086] X hazy =Degradation(X hazyfree );

[0087] Where Degradation represents the fog-free image X gazyfree The degradation operation of the foggy image is specifically constructed through the ASM atmospheric scattering model (Atmospheric Scattering Model), which can be expressed as:

[0088] X hazy =X hazyfree t+A(1-t);

[0089] Where A represents the atmospheric scattering coefficient, t represents the transfer map, and the specific formula of t can be expressed as:

[0090] t=e -βd ;

[0091] Among them, β represents the fog density coefficient, d represents X hazyfreeThe corresponding depth map, the depth map d is obtained through the DepthPro model.

[0092] Construct the loss function of the network, which includes noise loss, L1 loss (Mean Absolute Error, MAE) and depth loss. In particular, the noise loss formula can be expressed as:

[0093]

[0094] Where t represents the time step, ∈ represents the preset noise, ∈ θ (·) represents the noise predicted by the UNet denoiser, from which the L2 loss (Mean Squared Error, MSE) between the UNet denoiser output noise and the original preset noise is calculated.

[0095] L1 loss can be expressed as:

[0096]

[0097] Used to calculate the fog-free original image and the generated fog-free image The L1 loss between .

[0098] The depth loss can be expressed as:

[0099]

[0100] Where d represents the depth estimation operation, and the depth map of the corresponding image is obtained, thereby calculating the L2 loss between the depth map of the fog-free original image and the generated fog-free image. Therefore, the overall loss function of the network can be expressed as: L = L noise +L1+L depth .

[0101] The entire network structure is jointly trained. The haze-free image and the original haze-free image are input into the loss function of the entire network structure, the loss is calculated, and the training process is repeated. This joint training step can better train the learning ability of the network model and produce images with better haze removal effects.

[0102] Compared with traditional image processing methods, image dehazing methods based on CNN and GAN can obtain good image effects in light fog scenes, but existing methods still have problems with residual haze on the edges of objects after dehazing and severe color deviation when processing images on foggy days with severe haze. In response to the above shortcomings, this embodiment constructs an image dehazing method based on a diffusion model. The network structure is designed to include five parts: a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser and a VAE image decoder. While based on a powerful diffusion prior, text and depth features are used to interactively guide foggy images, and multimodal features are integrated to obtain a clean, residue-free and color-free dehazing effect. In addition, on the basis of the diffusion model noise loss function, pixel-level L1 and depth loss functions are designed to enhance the texture structure details of the image generation, greatly enhancing the realism and restoration of the dehazed image.

[0103] Reference Figure 3 FIG2 is a schematic structural diagram of an image defogging device based on a diffusion model provided by an embodiment of the present invention.

[0104] In this embodiment, the device is implemented based on an image defogging model constructed based on Stable Diffusion. The network structure of the image defogging model includes a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser and a VAE image decoder. The device 20 includes:

[0105] The encoding unit 21 is used to input the foggy image to be processed into the VAE image encoder to obtain the original image latent code;

[0106] A first extraction unit 22 is configured to input the foggy image into a depth feature extractor to obtain depth features;

[0107] The second extraction unit 23 is used to input the description text into a text encoder to obtain text features;

[0108] A denoising unit 24 is used to input the original image latent code, depth features and text features into a UNet denoiser to obtain a denoised latent code;

[0109] The decoding unit 25 is used to input the denoising latent code and the depth feature into the VAE image decoder and output the defogging fog-free image.

[0110] Each unit module of the device 20 can respectively execute the corresponding steps in the above method embodiment, so each unit module will not be described in detail here. Please refer to the description of the corresponding steps above for details.

[0111] The embodiment of the present invention further provides an image defogging device based on a diffusion model, the device comprising the image defogging device based on a diffusion model as described above, wherein the image defogging device based on a diffusion model can adopt Figure 3 The structure of the embodiment can be executed accordingly. Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects. For details, please refer to the relevant records in the above embodiments and will not be repeated here.

[0112] The device includes: a mobile phone, digital camera, tablet computer, or other device with a camera function, or a device with an image processing function, or a device with an image display function. The device may include components such as a memory, a processor, an input unit, a display unit, and a power supply.

[0113] Among them, the memory can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an image playback function, etc.), etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor and the input unit with access to the memory.

[0114] The input unit can be used to receive input digital, character, or image information, and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, the input unit of this embodiment includes not only a camera, but also a touch-sensitive surface (such as a touch display) and other input devices.

[0115] The display unit can be used to display information input by the user or information provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit may include a display panel. Optionally, the display panel can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Furthermore, the touch-sensitive surface can cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor to determine the type of touch event. The processor then provides a corresponding visual output on the display panel based on the type of touch event.

[0116] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 1 The image defogging method based on the diffusion model is shown. The computer readable storage medium can be a read-only memory, a magnetic disk or an optical disk.

[0117] The embodiment of the present invention further provides a computer program product, including a computer program / instruction, which is loaded and executed by a processor to implement Figure 1 An image dehazing method based on a diffusion model is shown.

[0118] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For similar or identical parts between the various embodiments, reference can be made to each other. For the apparatus embodiments, device embodiments, and storage medium embodiments, since they are generally similar to the method embodiments, their descriptions are relatively simple. For relevant parts, reference can be made to the descriptions of the method embodiments.

[0119] Furthermore, in this document, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0120] While the foregoing description shows and describes preferred embodiments of the present invention, it should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. Rather, the present invention can be used in various other combinations, modifications, and environments, and can be modified within the scope of the present invention by the teachings herein or by techniques or knowledge in the relevant art. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the present invention are intended to be within the scope of the appended claims.

Claims

1. An image defogging method based on a diffusion model, characterized in that: The method is implemented based on an image defogging model built on Stable Diffusion. The network structure of the image defogging model includes a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser and a VAE image decoder. The method includes: Input the foggy image to be processed into the VAE image encoder to obtain the original image latent code; Input the foggy image into the depth feature extractor to obtain the depth feature; Input the description text into the text encoder to obtain text features; Input the original image latent code, deep features, and text features into the UNet denoiser to obtain the denoised latent code; The denoising latent code and deep features are input into the VAE image decoder, and the dehazed image is output.

2. The image defogging method based on the diffusion model according to claim 1, characterized in that: The training process of the image dehazing model includes: Constructing image pairing data, the image pairing data including a non-fog original image, a foggy sample image generated by fogging and degrading the non-fog original image, and text corresponding to the non-fog original image; The image pairing data is input into the network structure of the image dehazing model for joint training, and the model parameters are optimized through a preset loss function to obtain the image dehazing model, wherein the loss function includes noise loss, L1 loss and depth loss. The noise loss is the L2 loss of the noise predicted by the UNet denoiser and the preset noise, the L1 loss is the loss between the generated image and the haze-free original image, and the depth loss is the L2 loss of the depth map corresponding to the generated image and the haze-free original image.

3. The image defogging method based on a diffusion model according to claim 1, characterized in that: The VAE image encoder is built based on the VAE encoder structure of Stable Diffusion and is fine-tuned during the training process.

4. The image defogging method based on a diffusion model according to claim 1, characterized in that: The text encoder is constructed using Alpha-CLIP.

5. The image defogging method based on a diffusion model according to claim 1, characterized in that: The deep feature extractor is constructed using the pre-trained DepthPro model.

6. The image defogging method based on a diffusion model according to claim 1, characterized in that: The UNet denoiser is built based on UNet with pre-trained weights in Stable Diffusion and fine-tuned using LoRA.

7. The image defogging method based on a diffusion model according to claim 1, characterized in that: The VAE image decoder is constructed using the VAE decoder structure with pre-trained weights in Stable Diffusion.

8. An image defogging device based on a diffusion model, characterized in that: The device is implemented based on an image defogging model built on Stable Diffusion. The network structure of the image defogging model includes a VAE image encoder, a text encoder, a deep feature extractor, a UNet denoiser and a VAE image decoder. The device includes: The encoding unit is used to input the foggy image to be processed into the VAE image encoder to obtain the original image latent code; A first extraction unit is used to input the foggy image into a depth feature extractor to obtain depth features; The second extraction unit is used to input the description text into a text encoder to obtain text features; The denoising unit is used to input the original image latent code, deep features and text features into the UNet denoiser to obtain the denoised latent code; The decoding unit is used to input the denoising latent code and deep features into the VAE image decoder and output the defogging fog-free image.

9. An image defogging device based on a diffusion model, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of the image defogging method based on a diffusion model according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises a computer program / instruction, which, when executed by a processor, implements the steps of the image defogging method based on a diffusion model as claimed in any one of claims 1 to 7.

Images