Image defogging method based on domain prompt adapter

Abstract

The application discloses an image defogging method based on a domain prompt adapter, first, unpaired training data containing real foggy and fog-free images are constructed, and text prompts capable of distinguishing foggy / fog-free are trained by comparing the text-image matching scores of a CLIP (Contrastive Language-Image Pre-training) model; then, the text prompts are used as discriminators to guide the training of a style domain prompt adapter in a defogging model, that is, the mean and standard deviation of style domain features affecting fog distribution are adaptively modulated, so that the fog distributions of the synthetic domain and the real domain are aligned; finally, after the training converges, the defogging model with the style domain prompt adapter can realize real image defogging.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to an image dehazing method based on a domain cue adapter. Background Technology

[0002] Fog is a common atmospheric phenomenon that significantly degrades the visual quality of images, thus impacting the accuracy of downstream tasks such as object detection and semantic segmentation. Image dehazing, as an image processing technique, aims to recover high-quality, clear image content from foggy conditions, and it is of great significance in practical applications such as traffic monitoring and autonomous driving systems. In recent years, with the rapid development of deep learning, supervised learning-based image dehazing methods have made significant progress. These methods typically rely on a large number of synthetic fog images and their corresponding fog-free images to train the model. While these methods perform well on synthetic fog datasets, they are largely ineffective for real-world foggy images. This is because there is a significant domain difference between synthetic and real foggy images, stemming from the inconsistency in the distribution of synthetic and real fog.

[0003] To address this issue, researchers have proposed various unsupervised and semi-supervised dehazing methods. Some works attempt to introduce unsupervised training strategies based on recurrent adversarial generative networks, aiming to improve the generalization of dehazing models without using synthetic foggy images and their corresponding fog-free images. Other methods incorporate physical priors, such as atmospheric scattering models, during the training process to alleviate overfitting of the model to synthetic fog distributions. However, these methods not only increase training complexity and time costs but also fail to fundamentally address the core issue causing domain distribution differences, thus offering limited improvement in dehazing performance in real-world scenes. Foggy images can be decomposed into a fog-independent content domain and a fog-dependent style domain; the difference in domain distribution between synthetic and real foggy images stems from their different style domains. Therefore, the key to improving the real-world fog removal capability of dehazing models trained on synthetic data lies in designing appropriate domain adapters to align the style domain features of synthetic and real fog.

[0004] Furthermore, since real-world foggy images typically lack corresponding fog-free images as supervised labels, directly training a real-world dehazing model in unsupervised mode is extremely difficult. Currently, the contrastive language-image pre-trained model CLIP possesses powerful image and text matching capabilities, effectively distinguishing between foggy and fog-free images when given appropriate textual cues ("foggy" / "foggy"). However, existing dehazing methods neglect the auxiliary training role of CLIP's textual cues in the domain adapter of the dehazing model, making it difficult to effectively improve real-world dehazing results.

[0005] In summary, existing techniques have the following drawbacks: 1. The training process is complex and time-consuming, requiring significant resources to retrain the model. 2. They do not consider designing domain adapters to modulate style domain features that influence fog, thus failing to leverage the intrinsic knowledge learned by the dehazing model from synthetic data. 3. They neglect the unsupervised auxiliary training effect of the textual prompts prior from the contrastive language-image pre-trained model CLIP on the dehazing model. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image dehazing method based on a domain cue adapter. By using a domain cue adapter, the generalization of the supervised dehazing model in real-world scenes can be improved. Furthermore, by training the model with a text cue-assisted model based on the contrastive language-image pre-training model CLIP, real-world image dehazing can be achieved.

[0007] To achieve the above-mentioned objective, the present invention provides an image dehazing method based on a domain cue adapter, characterized by comprising the following steps:

[0008] (1) Image acquisition and preprocessing;

[0009] Collect several real foggy images and several real fog-free images, and assign a label to each image, with the label of real fog-free images set to 1 and the label of real foggy images set to 0; in addition, download several synthetic foggy images from the database, and combine the three types of images to form the training dataset.

[0010] (2) Build a defogging network based on the domain prompt adapter;

[0011] The dehazing network based on the domain cue adapter consists of two dehazing networks, G1 and G2. Dehazing network G1 is composed of multiple convolutional layers connected in series. Its input is a synthesized hazy image, and its output is the dehazed synthesized image. Dehazing network G2 uses the architecture of dehazing network G1 as a reference, and a domain cue adapter is connected in series at the output of each convolutional layer. Its input is a real hazy image, and its output is the dehazed real image.

[0012] The domain cue adapter includes multiple normalization modules and 1×1 convolutional layers, as well as a single style cue representation update module, fog distribution adjustment module, and Softmax function, and also has a built-in style cue representation matrix;

[0013] Let the convolution result of the i-th convolutional layer in the dehazing network G2 be: Its connected domain cue adapter has a built-in style cue representation matrix of P. i The convolution result of the i-th convolutional layer in the dehazing network G1 is:

[0014] In the style cue representation update module, based on the convolution results... Update the style hint representation matrix to P i Then update the matrix The input is directly fed into a 1×1 convolutional layer, and the convolution result is denoted as W. v At the same time, the updated matrix After passing through a normalization module, the input is fed into a 1×1 convolutional layer, and the convolution result is denoted as W. k ;

[0015] On another branch, the convolution result After passing through a normalization module, the input is fed into a 1×1 convolutional layer and a fog distribution adjustment module, where the convolution result of the 1×1 convolutional layer is denoted as W. q ;

[0016] Next, the convolution result W q and W k Element-wise multiplication is performed, and the result is passed through the Softmax function to obtain the attention map A. i Then A i With the convolution result W v Element-wise multiplication is performed, and then the mean μ and standard deviation δ of the attention-weighted features are calculated based on the multiplication results. Finally, the fog distribution adjustment module adjusts the fog distribution of the real foggy image based on the mean μ and standard deviation δ of the weighted features, resulting in the adjusted real foggy image features.

[0017] (3) Training a defogging network based on a domain cue adapter;

[0018] (3.1) Randomly select a real foggy image I from the training dataset. h and true fog-free images I c Then, for the real foggy image I h Randomly generate foggy text hint label T h For true fog-free images I c Randomly generate fog-free text hint label T c ;

[0019] Will I h I c T h T c The images are input together into the contrastive language-image pre-trained model CLIP, where the image encoder extracts the image features Φ separately. image (I h ), Φ image (I c Text features Φ are extracted using a text encoder. text (T h ), Φ text (Tc ); (3.2)

[0021] Calculate the cross-entropy loss for binary classification

[0022] (3.2.1) Calculate the real foggy image I h Binary cross-entropy loss

[0023]

[0024]

[0025] Wherein, y(I h ) represents a real foggy image I h The actual label value, Represents a real foggy image I h The estimated label value;

[0026] (3.2.2) Calculate the true fog-free image I c Binary cross-entropy loss

[0027]

[0028]

[0029] Wherein, y(I c ) represents a true fog-free image I c The actual label value, Represents a true fog-free image I c The estimated label value;

[0030] (3.2.3) Calculate the cross-entropy loss for binary classification.

[0031] (3.3)

[0033] A synthetic foggy image is randomly selected from the training dataset. s The synthesized foggy image I s And real foggy images I h The input is fed into a domain cueing adapter-based dehazing network, where the synthesized hazy image I... s After passing through the dehazing network G1, the output is the dehazed composite image I. p Realistic foggy image I h After passing through the dehazing network G2, the output is the dehazed, realistic image I. r ;

[0034] (3.4) The real image after the fog I r Input is fed into the contrastive language-image pre-trained model CLIP, which extracts the real image I through an image encoder. r Image features Φ image (I r );

[0035] (3.5) Calculate the comparison similarity loss

[0036]

[0037] (3.6) Calculate the true image I after dehazing. r And real foggy images I h Content domain loss

[0038]

[0039] Norm(x) = (x - μ) / δ

[0040] Where ||·||1 represents the L1 norm, Norm(x) represents the fog-independent content domain obtained by normalizing the input image x along the channel dimension using the weighted feature mean μ and standard deviation δ, x∈[I r ,I h ];

[0041] (3.7) Calculate the total network loss after this round of training;

[0042] L total =L bce +L clip +L c

[0043] (3.8) Based on the total loss, update the parameters of the dehazing network based on the domain cue adapter using gradient descent, and then perform the next round of training until the dehazing network based on the domain cue adapter converges and the training ends.

[0044] (4) Image dehazing test;

[0045] A real foggy image is acquired and then fed into a trained dehazing network based on a domain cue adapter, which outputs a dehazed, fog-free image.

[0046] The objective of this invention is achieved as follows:

[0047] This invention presents an image dehazing method based on a domain cue adapter. First, it constructs unpaired training data containing real foggy and fog-free images. Then, it trains text cue tools capable of distinguishing between foggy and fog-free images by comparing the image-text matching scores of the language-image pre-trained model CLIP. Next, it uses the text cue tools as discriminators to guide the training of the style domain cue adapter in the dehazing model, adaptively modulating the mean and standard deviation of style domain features that influence fog distribution, thereby aligning the fog distribution between the synthetic and real domains. Finally, after training convergence, the dehazing model with the style domain cue adapter can be used to achieve real-world image dehazing.

[0048] Meanwhile, the image dehazing method based on the domain cue adapter of the present invention also has the following beneficial effects:

[0049] (1) This invention is a general real image dehazing framework, applicable to most dehazing models trained under supervision based on synthetic fog data. In order to overcome the problem of domain distribution difference between synthetic fog and real fog, this invention innovatively designs a domain cue adapter. During the training process, the weight parameters of the remaining modules of the original dehazing network are frozen and fixed, and only the weight parameters of the domain cue adapter are needed, which greatly reduces the training cost.

[0050] (2) This invention only requires unpaired real foggy and fog-free images as training data. It uses the image-text matching similarity score of the contrastive language-image pre-training model CLIP to perform cue learning training. Compared with previous unsupervised defogging models, it provides additional cue prior constraints. Attached Figure Description

[0051] Figure 1 This is a flowchart of the image dehazing method based on a domain prompt adapter according to the present invention;

[0052] Figure 2 This is a defogging network structure diagram based on the domain prompt adapter;

[0053] Figure 3 yes Figure 1 The diagram shown indicates the adapter structure.

[0054] Figure 4 This is a flowchart of the cue learning process using the contrastive language-image pre-trained model CLIP;

[0055] Figure 5 These are comparison images showing the dehazing effect on real foggy images. Detailed Implementation

[0056] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0057] Example

[0058] Figure 1 This is a flowchart of the image dehazing method based on the domain prompt adapter of the present invention.

[0059] In this embodiment, as Figure 1 As shown, the present invention provides an image dehazing method based on a domain cue adapter, comprising the following steps:

[0060] S1. Image acquisition and preprocessing;

[0061] Collect several real foggy images and several real fog-free images, and assign a label to each image, with the label of real fog-free images set to 1 and the label of real foggy images set to 0; in addition, download several synthetic foggy images from the database, and combine the three types of images to form the training dataset.

[0062] In this embodiment, the acquired real foggy images and real fog-free images do not need to be paired.

[0063] S2. Build a defogging network based on the domain prompt adapter;

[0064] like Figure 2 As shown, the dehazing network based on the domain cue adapter consists of two dehazing networks G1 and G2. Dehazing network G1 is composed of multiple convolutional layers connected in series. Its input is a synthesized hazy image, and its output is a dehazed synthesized image. Dehazing network G2 uses the architecture of dehazing network G1 as an example. A domain cue adapter is connected in series at the output of each convolutional layer. Its input is a real hazy image, and its output is a dehazed real image.

[0065] Domain cueing adapters can transform input real fog distribution features into synthetic fog distribution features in the feature space, thereby activating their potential defogging capability in the real fog distribution; for example... Figure 3 As shown, the domain cue adapter specifically includes multiple normalization modules and 1×1 convolutional layers, as well as a single style cue representation update module, fog distribution adjustment module, and Softmax function, and also has a built-in style cue representation matrix.

[0066] Let the convolution result of the i-th convolutional layer in the dehazing network G2 be: Its connected domain cue adapter has a built-in style cue representation matrix of P. i The convolution result of the i-th convolutional layer in the dehazing network G1 is:

[0067] In the style cue representation update module, based on the convolution results... Update the style hint representation matrix to P i Style hints representation matrix P i The update method is as follows:

[0068]

[0069] Where λ is a weighting coefficient, which is determined during training based on the content domain loss function. Dynamic adjustments will be made, as follows:

[0070]

[0071] Where, λ s These are the initial coefficients, and η and θ are hyperparameters; when the content domain loss... When θ is greater than θ, the weighting coefficient λ increases with... The value increases as the input decreases. During the testing phase, λ is set to 1, meaning that synthetic foggy features are not required.

[0072] Then the updated matrix The input is directly fed into a 1×1 convolutional layer, and the convolution result is denoted as W. v At the same time, the updated matrix After passing through a normalization module, the input is fed into a 1×1 convolutional layer, and the convolution result is denoted as W. k ;

[0073] On another branch, the convolution result After passing through a normalization module, the input is fed into a 1×1 convolutional layer and a fog distribution adjustment module, where the convolution result of the 1×1 convolutional layer is denoted as W. q ;

[0074] Next, the convolution result W q and W k Element-wise multiplication is performed, and the result is passed through the Softmax function to obtain the attention map A. i Then A i With the convolution result W v Element-wise multiplication is performed, and then the attention-weighted feature mean μ and standard deviation δ are calculated based on the multiplication results. In this embodiment, the weighted feature mean μ and standard deviation δ are calculated as follows:

[0075]

[0076]

[0077] in, This represents matrix multiplication.

[0078] Finally, the fog distribution adjustment module adjusts the fog distribution of the real foggy image based on the weighted feature mean μ and standard deviation δ, obtaining the adjusted real foggy image features. In this embodiment, the specific adjustment method is as follows:

[0079]

[0080] in, This indicates that the input image is processed using the weighted feature mean μ and standard deviation δ in the channel dimension. Normalization yields a fog-independent content domain.

[0081] S3. Train a defogging network based on a domain cue adapter;

[0082] S3.1. Randomly select a real foggy image I from the training dataset. h and true fog-free images I c Then, for the real foggy image I h Randomly generate foggy text hint label T h For true fog-free images I c Randomly generate fog-free text hint label T c ;

[0083] Will I h I c T h T c Together, they are input into the contrastive language-image pre-trained model CLIP, such as... Figure 4 As shown, the image features Φ are extracted using the image encoder in the model. image (I h ), Φ image (I c Text features Φ are extracted using a text encoder. text (T h ), Φ text (T c );

[0084] S3.2 Calculate the cross-entropy loss for binary classification.

[0085] S3.2.1 Calculate the real foggy image I h Binary cross-entropy loss

[0086]

[0087]

[0088] Wherein, y(I h ) represents a real foggy image I h The actual label value, Represents a real foggy image I h The estimated label value;

[0089] S3.2.2 Calculate the true haze-free image I c Binary cross-entropy loss

[0090]

[0091]

[0092] Wherein, y(I c ) represents a true fog-free image I c The actual label value, Represents a true fog-free image I c The estimated label value;

[0093] S3.2.3 Calculate the cross-entropy loss for binary classification.

[0094]

[0095] S3.3, Randomly select a synthetic foggy image I from the training dataset. s The synthesized foggy image I s And real foggy images I h The input is fed into a domain cueing adapter-based dehazing network, where the synthesized hazy image I... s After passing through the dehazing network G1, the output is the dehazed composite image I. p Realistic foggy image I h After passing through the dehazing network G2, the output is the dehazed, realistic image I. r ;

[0096] S3.4, the real image after the fog I r Input is fed into the contrastive language-image pre-trained model CLIP, which extracts the real image I through an image encoder. r Image features Φ image (I r );

[0097] S3.5 Calculate the comparison similarity loss

[0098]

[0099] S3.6 Calculate the true image I after dehazing. r And real foggy images Ih Content domain loss

[0100]

[0101] Norm(x) = (x - μ) / δ

[0102] Where ||·||1 represents the L1 norm, Norm(x) represents the fog-independent content domain obtained by normalizing the input image x along the channel dimension using the weighted feature mean μ and standard deviation δ, x∈[I r ,I h ];

[0103] S3.7 Calculate the total network loss after this round of training;

[0104] L total =L bce +L clip +L c

[0105] S3.8. Based on the total loss, update the parameters of the dehazing network based on the domain cue adapter using gradient descent, and then perform the next round of training until the dehazing network based on the domain cue adapter converges, at which point training ends.

[0106] S4, Image Dehazing Test;

[0107] A real foggy image is acquired and then fed into a trained dehazing network based on a domain cue adapter, which outputs a dehazed, fog-free image.

[0108] Simulation verification

[0109] In this embodiment, we use URHI (unlabeled real foggy images) from the RESIDE dataset as unsupervised training data and validate them on real I-HAZE, O-HAZE, and RTTS datasets. To evaluate the paired I-HAZE and O-HAZE datasets, we use PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Consistency) as full-reference metrics, both indicating that higher scores indicate better image quality. For the RTTS dataset, which lacks fog-free reference images, we apply three no-reference quality metrics for quantitative comparison: BRISQUE (No-Reference Spatial Quality Assessment), NIQE (Naturalness Image Quality Assessment), and FADE (Fog Density Assessment), all indicating that lower scores indicate better image quality.

[0110] In this embodiment, the proposed framework is deployed on supervised dehazing models FFANet, C2PNet, and DehazeFormer. Furthermore, to demonstrate the superiority of this method, it is compared with state-of-the-art semi-supervised and unsupervised dehazing models, including DisentGAN, DAD, PSD, and D4. The comparison results are shown in Table 1.

[0111] Table 1 Evaluation results of I-HAZE, O-HAZE, and RTTS real foggy image datasets

[0112]

[0113] Supervised methods often underperform on real-world datasets due to the discrepancy between synthetic and real fog distributions. However, this invention effectively improves their performance by using a style domain cue adapter. For example, our method achieves average PSNR improvements of 1.65 dB and 0.59 dB on the I-HAZE and O-HAZE datasets, respectively. Furthermore, DehazeFormer, combined with this invention, outperforms other methods on the RTTS dataset, achieving a state-of-the-art BRISQUE score of 25.183 and a second-best NIQE score of 4.355.

[0114] like Figure 5 As shown, supervised methods FFANet and DehazeFormer are almost incapable of removing realistic fog. Although the semi-supervised PSD method achieves the best FADE score, it tends to over-brighten the image, while the DAD method introduces color bias. In contrast, our framework improves the realistic dehazing capability of supervised models while preserving the color fidelity of the input image.

[0115] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. An image dehazing method based on a domain cue adapter, characterized in that, Includes the following steps: (1) Image acquisition and preprocessing; Collect several real foggy images and several real fog-free images, and assign a label to each image, with the label of real fog-free images set to 1 and the label of real foggy images set to 0; in addition, download several synthetic foggy images from the database, and combine the three types of images to form the training dataset. (2) Build a defogging network based on the domain prompt adapter; The dehazing network based on the domain cue adapter consists of two dehazing networks, G1 and G2. Dehazing network G1 is composed of multiple convolutional layers connected in series. Its input is a synthesized hazy image, and its output is the dehazed synthesized image. Dehazing network G2 uses the architecture of dehazing network G1 as a reference, and a domain cue adapter is connected in series at the output of each convolutional layer. Its input is a real hazy image, and its output is the dehazed real image. The domain cue adapter includes multiple normalization modules and 1×1 convolutional layers, as well as a single style cue representation update module, fog distribution adjustment module, and Softmax function, and also has a built-in style cue representation matrix; Let the convolution result of the i-th convolutional layer in the dehazing network G2 be: Its connected domain cue adapter has a built-in style cue representation matrix of P. i The convolution result of the i-th convolutional layer in the dehazing network G1 is: In the style cue representation update module, based on the convolution results... Update the style hint representation matrix to P i Then update the matrix The input is directly fed into a 1×1 convolutional layer, and the convolution result is denoted as W. v At the same time, the updated matrix After passing through a normalization module, the input is fed into a 1×1 convolutional layer, and the convolution result is denoted as W. k ; On another branch, the convolution result After passing through a normalization module, the input is fed into a 1×1 convolutional layer and a fog distribution adjustment module, where the convolution result of the 1×1 convolutional layer is denoted as W. q ; Next, the convolution result W q and W k Element-wise multiplication is performed, and the result is passed through the Softmax function to obtain the attention map A. i Then A i With the convolution result W v Element-wise multiplication is performed, and then the mean μ and standard deviation δ of the attention-weighted features are calculated based on the multiplication results. Finally, the fog distribution adjustment module adjusts the fog distribution of the real foggy image based on the mean μ and standard deviation δ of the weighted features, resulting in the adjusted real foggy image features. (3) Training a defogging network based on a domain cue adapter; (3.1) Randomly select a real foggy image I from the training dataset. h and true fog-free image I c Then, for the real foggy image I h Randomly generate foggy text hint label T h For true fog-free images I c Randomly generate fog-free text hint label T c ; Will I h I c T h T c The images are input together into the contrastive language-image pre-trained model CLIP, where the image encoder extracts the image features Φ separately. image (I h ), Φ image (I c Text features Φ are extracted using a text encoder. text (T h ), Φ text (T c ); (3.2) Calculate the cross-entropy loss for binary classification. (3.2.1) Calculate the real foggy image I h Binary cross-entropy loss Wherein, y(I h ) represents a real foggy image I h The actual label value, Represents a real foggy image I h The estimated label value; (3.2.2) Calculate the true fog-free image I c Binary cross-entropy loss Wherein, y(I c ) represents a true fog-free image I c The actual label value, Represents a true fog-free image I c The estimated label value; (3.2.3) Calculate the cross-entropy loss for binary classification. (3.3) Randomly select a synthetic foggy image I from the training dataset. s The synthesized foggy image I s And real foggy images I h The input is fed into a domain cueing adapter-based dehazing network, where the synthesized hazy image I... s After passing through the dehazing network G1, the output is the dehazed composite image I. p Realistic foggy image I h After passing through the dehazing network G2, the output is the dehazed, realistic image I. r ; (3.4) The real image after dehazing I r Input is fed into the contrastive language-image pre-trained model CLIP, which extracts the real image I through an image encoder. r Image features Φ image (I r ); (3.5) Calculate the comparison similarity loss (3.6) Calculate the true image I after dehazing. r And real foggy images I h Content domain loss Norm(x) = (x - μ) / δ Where ||·||1 represents the L1 norm, Norm(x) represents the fog-independent content domain obtained by normalizing the input image x along the channel dimension using the weighted feature mean μ and standard deviation δ, x∈[I r ,I h ]; (3.7) Calculate the total network loss after this round of training; L total L bce +L clip +L c (3.8) Based on the total loss, update the parameters of the dehazing network based on the domain cue adapter using gradient descent, and then perform the next round of training until the dehazing network based on the domain cue adapter converges and the training ends. (4) Image dehazing test; A real foggy image is acquired and then fed into a trained dehazing network based on a domain cue adapter, which outputs a dehazed, fog-free image.

2. The image dehazing method based on a domain cue adapter according to claim 1, characterized in that, The style cue representation matrix P i The update method is as follows: Where λ is a weighting coefficient, which is determined during training based on the content domain loss function. Dynamic adjustments will be made, as follows: Where, λ s η and θ are the initial coefficients, and η and θ are hyperparameters.

3. The image dehazing method based on a domain cue adapter according to claim 1, characterized in that, The weighted feature mean μ and standard deviation δ are calculated as follows: in, This represents matrix multiplication.

4. The image dehazing method based on a domain cue adapter according to claim 1, characterized in that, The fog distribution adjustment module adjusts the fog distribution of the real foggy image based on the weighted feature mean μ and standard deviation δ in the following way: in, This indicates that the input image is processed using the weighted feature mean μ and standard deviation δ in the channel dimension. Normalization yields a fog-independent content domain.

Images