Image defogging method based on diffusion model and fog concentration perception

Abstract

The invention relates to the technical field of image processing, and provides an image defogging method based on a diffusion model and a fog concentration perception network. The image defogging model provided by the invention can process images of high-density haze and complex environments, and the model comprises a forward diffusion process and a backward diffusion process. Specifically, in the back diffusion process, the model introduces a concentration sensing defogging network, and the network gradually recovers a clear fogless image from a haze image. In order to improve the network performance, a cross-feature density extraction module is designed to extract the haze density of the image, and a density-guided feature fusion module is designed to learn effective context features. Besides, in the test sampling process, an indirect sampling strategy is introduced into the provided model, error accumulation is inhibited, and the stability of the result is ensured. A large number of benchmark test experiments verify the superior performance of the method.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image defogging method based on a diffusion model and fog concentration perception. Background Art

[0002] In daily life, fog often affects people's travel and visual perception, not only reducing visibility, but also having a serious impact on the quality of images taken by imaging equipment. Fog can cause image contrast to decrease, details to be lost, and produce a significant light scattering effect, which in turn affects the clarity and recognizability of the image. This situation not only poses a safety hazard in scenarios such as traffic monitoring, autonomous driving, and drone aerial photography, but also affects image analysis and processing tasks in multiple fields including security monitoring and satellite remote sensing. Therefore, image dehazing, as an important image processing task, has become a widely studied topic in the field of computer vision.

[0003] Image dehazing refers to restoring the real information of images taken under foggy conditions through algorithms to improve the clarity and visibility of images. Image dehazing is often an essential preprocessing step for subsequent tasks such as image recognition, target detection, and scene analysis. By dehazing, the target details in the image can be effectively enhanced, making the outlines of scenes and objects clearer, thereby improving the accuracy of recognition and detection. At present, the main technical routes for image dehazing can be divided into traditional image processing methods and methods based on deep learning.

[0004] Traditional image dehazing methods are mainly based on physical models, such as dark channel prior method, Retinex theory, etc. These methods use prior knowledge in natural images to infer the transmittance and atmospheric light of the scene, and then remove the impact of haze on the image. However, traditional methods usually rely on certain assumptions, and the dehazing effect is relatively limited for haze scenes of different concentrations, distributions and types, especially in complex environments where the generalization ability is weak.

[0005] In recent years, with the development of deep learning technology, deep learning-based image defogging methods have begun to attract widespread attention. Through a large number of training samples of foggy and non-fog images, deep learning models can automatically learn and extract the haze features in images, thereby generating defogging images. The advantages of this type of method lie in its adaptability to a variety of complex scenes and high defogging accuracy. However, most existing deep learning defogging methods often cannot accurately perceive the difference in haze concentration when dealing with haze of different concentrations, and are prone to insufficient or excessive defogging, resulting in poor image quality. Summary of the invention

[0006] The present invention is designed to solve the above problems. As a generation model, the diffusion model can simulate the image generation process and restore a clear image by gradually denoising the foggy image. Under the premise of combining fog concentration perception, the method can dynamically adapt to the impact of different concentrations of haze on the image, gradually remove fog interference, and ensure that excellent defogging effects can still be achieved in complex environments, making image defogging technology more reliable in practical applications.

[0007] To achieve the above object, the present invention provides the following technical solution: an image defogging method based on a diffusion model and fog concentration perception, the method comprising the following steps:

[0008] S1: Construct a diffusion model guided by fog concentration to learn the mapping relationship from foggy images to fog-free images;

[0009] S2: Construct a dehazing network guided by fog concentration, which gradually recovers a clear fog-free image from the foggy image in the reverse diffusion process;

[0010] S3: Construct a cross-feature concentration extraction module to extract fog concentration information in the image and provide it as auxiliary information to the defogging network to more accurately estimate the noise in the image;

[0011] S4: Constructing a concentration-guided feature fusion module. The concentration-guided feature fusion module optimizes the feature extraction and image reconstruction process through feature fusion assisted by fog concentration information.

[0012] S5: Construct a loss function and use the corresponding data set to train and optimize the dehazing network guided by fog concentration to obtain an optimized network model;

[0013] S6: Use the trained model and adopt an indirect sampling strategy to sample and defog the foggy image to improve the accuracy and stability of the defogging results.

[0014] Preferably, in step S1, the diffusion model guided by fog concentration includes a denoising process and a denoising process. In the denoising process, the high-quality fog-free image x0 is gradually degraded into a foggy image x by adding noise. T , simulating the fog formation process, where a certain amount of noise is added at each step to form the final fog image with a certain degree of noise. The noise adding process is as follows:

[0015] dx=θ t (μ-x)dt+σ t dw

[0016] where θ t and σ tIt is expressed as a parameter related to the time step t, which represents the degradation rate and the size of the noise, w represents the standard Brownian motion, and μ represents the noisy fog image x. T .

[0017] In the denoising process, the diffusion model learned through training is gradually transformed from the fog image x with noise. T The noise and fog are removed from the image, and a clear fog-free image x0 is restored. This process is the inverse process of the noise addition process. By gradually reducing the noise, a fog-free image is finally obtained. The denoising process is as follows:

[0018]

[0019] in represents the standard Brownian motion in reverse time, is the score tag.

[0020] Preferably, in step S2, the defogging network includes a cross-feature concentration extraction module and multiple concentration-guided feature fusion modules, which are used to estimate noise in the reverse diffusion process and gradually remove noise from the foggy image until a clear fog-free image is restored.

[0021] Preferably, in step S3, the cross-feature concentration extraction module adopts a dual-branch structure to optimize the estimation of fog density, one branch focuses on extracting features from the input image, and the other branch focuses on extracting features from the dark channel map. In the image branch, a dynamic convolution layer is first used to capture the structural features of the image, and then the expression ability of these features is enhanced by a 1×1 convolution layer, and a hardswish activation function is applied to obtain a preliminary structural feature representation. For the dark channel branch, since the dark channel map usually contains rich local details, we use dilated convolution to extract features, and then use a 1×1 convolution layer and a GELU activation function to obtain a preliminary representation of the dark channel features. In order to further optimize the use of features, we cross-fuse the features extracted by the two branches, concatenate these preliminary feature representations, and then use a 3×3 convolution layer and a GELU activation function to obtain the final structural features, and use a 1×1 convolution layer to obtain the final dark channel features. Finally, these features are concatenated again and reduced in dimension by a 1×1 convolution layer to reconstruct the fog density feature D. To ensure the effectiveness of the inverse dehazing process, we perform two downsampling operations on the fog density feature D to obtain fog density information at three different scales (D1, D2, D3), ensuring that the network can fully utilize the fog density features at all scales and provide more accurate and comprehensive guidance for the dehazing process.

[0022] Preferably, in step S4, it is characterized in that two core units are carefully designed: a concentration-aware feature extraction unit and a fog-aware feature optimization unit. The purpose of the concentration-aware feature extraction unit is to reduce the image feature extraction error caused by fog, and to retain key information and filter out noise through a gating mechanism. It first batch-normalizes the input features, and then performs element-wise multiplication with the fog density features to obtain mixed features. These features are further sent to a dual-branch structure including a feature extraction branch and a gating branch for in-depth processing. The feature extraction branch is responsible for extracting useful features, while the gating branch generates control information to adjust the feature output. Finally, a multi-layer perceptron (MLP) module is used to optimize the features and fuse them with the original features to enhance the feature expression capability of the model.

[0023] On the other hand, the fog-aware feature optimization unit focuses on utilizing the output features of the concentration-aware feature extraction unit to extract effective contextual information by learning the fog-aware attention map. First, the output of the concentration-aware feature extraction unit is batch normalized, and then the features are obtained through convolution operations. Then, the channel-level and pixel-level attention maps are obtained through the channel attention unit and the pixel attention unit respectively. These attention maps are globally fused through the softmax function to form the global fog-aware attention, which is used to guide feature reconstruction. Finally, the fog-aware contextual features are obtained by element-wise multiplication of this attention map with the features and fused with the original features to optimize the defogging results.

[0024] Preferably, in step S5, the intermediate state is used and ideal state The L1 loss function between them is used to optimize the network. The loss function is as follows:

[0025]

[0026] The intermediate state The expression is as follows:

[0027]

[0028] in Describes the noise predicted by the network.

[0029] in The expression is as follows:

[0030]

[0031] in

[0032] Preferably, in step S6, an indirect sampling strategy is adopted. This strategy is used to process foggy images during the test process. First, a time point t1 is determined. The sampling before this time point uses a standard diffusion model method to directly calculate the image of the next time step based on the current noise estimate and the inverse diffusion equation.

[0033] After reaching time point t1, the indirect sampling stage begins. At this time, a temporary fog-free image is obtained using the current noise estimation and the inverse diffusion equation. Then, the temporary image and the original fog image are combined to calculate the image of the subsequent time step according to the inverse diffusion equation. The whole process is iteratively executed until all predetermined time steps are completed, and finally a clear fog-free image is obtained.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The diffusion model image defogging method based on fog concentration perception provided by the present invention constructs a new fog concentration guided defogging network model. First, the fog concentration information in the image is extracted by constructing a cross-feature concentration extraction module. This module can effectively capture the changes in fog density in different areas and provide more accurate guidance for subsequent defogging processing. By introducing fog concentration information, the model can better understand the fog effect in the image, thereby improving the defogging effect.

[0036] 2. The concentration-guided defogging network model constructed by the present invention includes two stages. In the first stage, the image features are extracted using a concentration-aware feature extraction unit, and preliminary defogging is performed in combination with fog concentration information, which helps to generate a clearer defogging image. In the second stage, a fog-aware feature optimization unit is used to further refine the defogging results of the first stage through the fusion of attention mechanism and context information. The optimization unit aims to enhance the detailed features in the image to ensure that the final generated fog-free image reaches a high level in clarity and naturalness.

[0037] 3. Through the training and optimization of the loss function, the defogging network model finally obtained by the present invention can effectively remove the fog effect in the image, making the processed image more realistic and natural, and significantly improving the visual quality and readability of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a main flow chart of the image defogging method based on diffusion model and fog concentration perception in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] like Figure 1 As shown, an image defogging method based on a diffusion model and fog concentration perception includes the following steps:

[0041] Step 1: Construct a diffusion model guided by fog concentration to learn the denoising process from foggy images to fog-free images. The diffusion model guided by fog concentration includes a diffusion process and a denoising process.

[0042] Step 2: Construct a fog concentration-guided dehazing network, which is used to estimate the noise in the reverse diffusion process and gradually restore a clear fog-free image from the foggy image. In the process of adding noise, the high-quality fog-free image x0 is gradually degraded into a foggy image x by adding noise. T , simulating the fog formation process, where a certain amount of noise is added in each step to form the final fog image with a certain degree of noise. The noise adding process is as follows:

[0043] dx=θ t (μ-x)dt+σ t dw

[0044] where θ t and σ t It is expressed as a parameter related to the time step t, which represents the degradation rate and the size of the noise respectively. w represents the standard Brownian motion, and η represents the noisy fog image x T .

[0045] In the denoising process, the diffusion model learned through training is gradually transformed from the fog image x with noise. T The noise and fog are removed from the image, and a clear fog-free image x0 is restored. This process is the inverse process of the noise addition process. By gradually reducing the noise, a fog-free image is finally obtained. The denoising process is as follows:

[0046]

[0047] in represents the standard Brownian motion in reverse time, is the score tag.

[0048] Step 2: Construct a fog concentration-guided dehazing network and obtain the input intermediate state image x t The corresponding noise And gradually from the noisy fog image x T A clear haze-free image x0 is restored.

[0049] Step 3: Construct a cross-feature concentration extraction module to extract fog concentration information in the image and provide it as auxiliary information to the defogging network to more accurately estimate the noise in the image. Specifically:

[0050] 3.1) Double branch structure:

[0051] The cross-feature concentration extraction module adopts a dual-branch structure, one branch focuses on extracting features from the input image, and the other focuses on extracting features from the dark channel map. This design enables the network to capture fog density information in the image from different angles.

[0052] 3.2) Dark channel processing: In the dark channel branch, the dynamic convolution layer is first used to extract the structural features of the image, and then the 1×1 convolution layer is used to enhance the expressiveness of the features, and the hardswish activation function is used to obtain the preliminary representation of the structural features. For the dark channel map, the dilated convolution is used to capture more local features, and the preliminary representation of the dark channel features is obtained through the 1×1 convolution layer and the GELU activation function.

[0053] 3.3) Feature Fusion

[0054] Feature fusion: The features extracted by the two branches are cross-fused, and the features from different branches are integrated through splicing and convolution operations to obtain a more comprehensive feature representation of fog density.

[0055] 3.4) Multi-scale features:

[0056] In order to ensure that the network can make full use of fog density information at different scales, the fog density features are downsampled to obtain fog density information at three different scales (D1, D2, D3), which helps the network capture subtle changes in fog at various scales.

[0057] Step 4: Construct a concentration-guided feature fusion module to optimize the feature extraction and image reconstruction process through feature fusion assisted by fog concentration information, including:

[0058] 4.1) Concentration-aware feature extraction unit:

[0059] This unit is designed to enhance the feature extraction process through fog concentration information. It first batch normalizes the input features, and then performs element-wise multiplication with the extracted fog density features to obtain mixed features, which are then fed into a dual-branch structure consisting of a feature extraction branch and a gating branch. The feature extraction branch is responsible for extracting useful features, while the gating branch generates control information to adjust the feature output. Finally, a multi-layer perceptron (MLP) module is used to optimize the features and fuse them with the original features.

[0060] 4.2) Fog perception feature optimization unit:

[0061] This unit focuses on using the features generated by the concentration-guided feature fusion module to extract effective contextual information by learning fog-aware attention maps, which is particularly important for feature reconstruction in areas with high fog density in images. It first batch-normalizes the output of the concentration-guided feature fusion module, then obtains features through convolution operations, and then uses channel attention units and pixel attention units to obtain channel-level and pixel-level attention maps respectively. These attention maps are globally fused through the softmax function to form global fog-aware attention, which is used to guide feature reconstruction.

[0062] Step 5: Construct a loss function to train the dehazing network and obtain the optimized network model:

[0063] Using the intermediate state and ideal state The L1 loss function between them is used to optimize the network. The loss function is as follows:

[0064]

[0065] The intermediate state The expression is as follows:

[0066]

[0067] in Describes the noise predicted by the network.

[0068] in The expression is as follows:

[0069]

[0070] in

[0071] Step 6: Build an indirect sampling strategy and use the trained network model to sample and defog the foggy image to improve the accuracy and stability of the defogging results. Specifically include:

[0072] 6.1) Determine the start time of the sampling strategy:

[0073] By analyzing the peak signal-to-noise ratio (PSNR) distribution between the dehazing results and the real haze-free images obtained at different time steps t1, a time point is determined as the start of the indirect sampling strategy. The selection of this time point is based on the judgment that the PSNR value reaches the optimal value, ensuring that the image quality can be maintained or slightly improved in the subsequent sampling process, rather than over-processing and causing quality degradation.

[0074] 6.2) Two-stage sampling process: When time step t>t1, the standard diffusion model sampling method is used to directly calculate the image of the next time step based on the current noise estimate and the inverse diffusion equation. When time step t≤t1, an indirect sampling strategy is adopted, first using the predicted noise And the inverse diffusion equation calculates the current defogging result Then based on and To calculate the image x at the next time step t-1 The algorithm of the sampling process is as follows:

[0075]

[0076] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0077] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An image defogging method based on diffusion model and fog concentration perception, characterized by: The method comprises the following steps: S1: Construct a diffusion model guided by fog concentration to learn the mapping relationship from foggy images to fog-free images; S2: Construct a dehazing network guided by fog concentration, which gradually recovers a clear fog-free image from the foggy image in the reverse diffusion process; S3: Construct a cross-feature concentration extraction module to extract fog concentration information in the image and provide it as auxiliary information to the defogging network to more accurately estimate the noise in the image; S4: Construct a concentration-guided feature fusion module to optimize the feature extraction and image reconstruction process through feature fusion assisted by fog concentration information; S5: Construct a loss function and use the corresponding data set to train and optimize the diffusion model guided by fog concentration and the defogging network to obtain the optimized network model; S6: Use the trained model and adopt an indirect sampling strategy to sample and defog the foggy image to improve the accuracy and stability of the defogging results.

2. The image defogging method based on diffusion model and fog concentration perception according to claim 1, characterized in that: In step S1, the diffusion model guided by fog concentration includes a denoising process and a denoising process. In the denoising process, the high-quality fog-free image x0 is gradually degraded into a foggy image x by adding noise. T , simulating the fog formation process, where a certain amount of noise is added at each step to form the final fog image with a certain degree of noise. The noise adding process is as follows: dx=θ t (μ-x)dt+σ t dw where θ t and σ t It is expressed as a parameter related to the time step t, which represents the degradation rate and the size of the noise, w represents the standard Brownian motion, and μ represents the noisy fog image x. T . In the denoising process, the diffusion model learned through training is gradually transformed from the fog image x with noise. T The noise and fog are removed from the image, and a clear fog-free image x0 is restored. This process is the inverse process of the noise addition process. By gradually reducing the noise, a fog-free image is finally obtained. The denoising process is as follows: in represents the standard Brownian motion in reverse time, is the score tag.

3. The image defogging method based on diffusion model and fog concentration perception according to claim 1, characterized in that: In step S2, the fog concentration-guided dehazing network includes a cross-feature concentration extraction module and multiple concentration-guided feature fusion modules, which are used to estimate noise in the reverse diffusion process and gradually remove noise from the foggy image until a clear fog-free image is restored.

4. The image defogging method based on diffusion model and fog concentration perception according to claim 1, characterized in that: In step S3, the cross-feature concentration extraction module uses a dual-branch structure to optimize the estimation of fog density. One branch focuses on extracting features from the input image, while the other branch focuses on extracting features from the dark channel map. In the image branch, a dynamic convolution layer is first used to capture the structural features of the image, and then the expression of these features is enhanced by a 1×1 convolution layer, and the hardswish activation function is applied to obtain a preliminary structural feature representation. For the dark channel branch, since the dark channel map usually contains rich local details, we use dilated convolution to extract features, and then use a 1×1 convolution layer and a GELU activation function to obtain a preliminary representation of the dark channel features. In order to further optimize the use of features, we cross-fuse the features extracted by the two branches, concatenate these preliminary feature representations, and then use a 3×3 convolution layer and a GELU activation function to obtain the final structural features, and use a 1×1 convolution layer to obtain the final dark channel features. Finally, these features are concatenated again and reduced in dimension by a 1×1 convolution layer to reconstruct the fog density feature D. To ensure the effectiveness of the inverse dehazing process, we perform two downsampling operations on the fog density feature D to obtain fog density information at three different scales (D1, D2, D3), ensuring that the network can fully utilize the fog density features at all scales and provide more accurate and comprehensive guidance for the dehazing process.

5. The image defogging method based on diffusion model and fog concentration perception according to claim 1, characterized in that: In step S4, it is characterized in that two core units are carefully designed: a concentration-aware feature extraction unit and a fog-aware feature optimization unit. The purpose of the concentration-aware feature extraction unit is to reduce the image feature extraction error caused by fog, and a gating mechanism is used to retain key information and filter out noise. It first batch-normalizes the input features, and then performs element-wise multiplication with the fog density features to obtain mixed features. These features are further sent to a dual-branch structure including a feature extraction branch and a gating branch for in-depth processing. The feature extraction branch is responsible for extracting useful features, while the gating branch generates control information to adjust the feature output. Finally, a multi-layer perceptron (MLP) module is used to optimize the features and fuse them with the original features to enhance the feature expression ability of the model. On the other hand, the fog-aware feature optimization unit focuses on utilizing the output features of the fog-aware feature optimization unit and extracting effective contextual information by learning the fog-aware attention map. First, the output of the fog-aware feature optimization unit is batch normalized, and then the features are obtained through convolution operations. Then, the channel-level and pixel-level attention maps are obtained through the channel attention unit and the pixel attention unit respectively. These attention maps are globally fused through the softmax function to form the global fog-aware attention, which is used to guide feature reconstruction. Finally, the fog-aware contextual features are obtained by element-wise multiplication of this attention map with the features and fused with the original features to optimize the defogging results.

6. The image defogging method based on diffusion model and fog concentration perception according to claim 1, characterized in that: In step S5, the intermediate state is used and ideal state The L1 loss function between them is used to optimize the network. The loss function is as follows: The intermediate state The expression is as follows: in Describes the noise predicted by the network. in The expression is as follows: in 7. The image defogging method based on diffusion model and fog concentration perception according to claim 1, characterized in that: In step S6, an indirect sampling strategy is adopted. This strategy is used to process foggy images during the test process. First, a time point t1 is determined. The sampling before this time point uses the standard diffusion model method to directly calculate the image of the next time step based on the current noise estimate and the inverse diffusion equation. After reaching time point t1, the indirect sampling stage begins. At this time, a temporary fog-free image is obtained using the current noise estimation and the inverse diffusion equation. Then, the temporary image and the original fog image are combined to calculate the image of the subsequent time step according to the inverse diffusion equation. The whole process is iteratively executed until all predetermined time steps are completed, and finally a clear fog-free image is obtained.

Images