A visible-infrared joint image defogging method based on pseudo-infrared generation and semi-supervised optimization

Abstract

The application discloses a visible-infrared joint image defogging method based on pseudo-infrared generation and semi-supervised optimization, which comprises preparing a training set and a test set of a MID data set. The method firstly constructs a pseudo-infrared generation network to predict a single-channel pseudo-infrared structural feature map from a foggy visible light image; then a double-branch defogging network comprising a visible light branch and an infrared branch is constructed, and cross-modal feature fusion is realized through an infrared guided feature weighting fusion module, so as to enhance the image structural information expression capability. In the training stage, a teacher-student consistency constraint mechanism is introduced for semi-supervised optimization, and the teacher network parameters are updated through an exponential moving average, and a color calibration module is arranged at the network output end to reduce color deviation. In the inference stage, when there is no real infrared image input, the pseudo-infrared generation network is used to provide structural information and input into the defogging network to complete image restoration, so as to improve the image defogging effect under a complex fog and haze environment.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a semi-supervised visible light-infrared joint image dehazing method based on pseudo-infrared guidance, belonging to the field of cross-modal image enhancement and deep learning application technology. Background Technology

[0002] Single-image dehazing methods have long revolved around the Atmospheric Scattering Model (ASM), which estimates the transmission map and atmospheric light parameters by constructing physical or statistical priors. With the development of deep learning, a large number of CNN structures have been used for end-to-end dehazing. DehazeNet[1] incorporates transmission map estimation into a learnable framework; LKD-Net[2] expands the receptive field through large kernel convolution; Laplace-Mamba[3] introduces frequency decomposition and global modeling strategies to improve restoration quality. Although CNNs perform well in local texture restoration, their fixed receptive field limits their ability to model large-scale fog distributions.

[0003] To enhance global dependency modeling capabilities, the Transformer architecture has been widely introduced into the field of image restoration. Restormer[4] adapts to high-resolution image restoration tasks through an efficient attention mechanism; DehazeFormer[5] optimizes the Transformer design for dehazing tasks; and MB-TaylorFormer[6] reduces computational complexity through linearized self-attention. Although the Transformer can capture global fog distribution information better, its training depends on large-scale pairwise data, and it still faces challenges in generalizing to real-world scenarios.

[0004] It is worth noting that the above methods are all based on single-modal visible light input, and significant recovery bottlenecks still exist in regions with dense fog or blurred structures at a distance. This provides an important motivation for introducing cross-modal structural priors.

[0005] Visible and infrared modes are naturally complementary under adverse weather conditions. Infrared imaging is insensitive to scattering and illumination changes and can provide stable structural and target contour information, while visible light images contain rich texture and color details. Therefore, RGB–Thermal fusion is widely used in segmentation and detection tasks. For example, RTFNet[7] achieves cross-modal feature fusion through a dual encoder structure, which significantly improves semantic segmentation performance under complex illumination conditions.

[0006] In the field of image fusion, DenseFuse[8] uses an encoding-fusion-decoding structure to achieve feature-level integration of infrared and visible light; SwinFuse[9] introduces a Transformer structure for cross-modal global modeling. These methods mainly aim to optimize the visual quality of the fused image.

[0007] For the dehazing task, some studies have begun to explore IR-guided Dehazing. VIFNet

[10] explicitly introduces visible-infrared fusion into the dehazing framework and constructs an AirSim-based VI dehazing dataset to verify the effectiveness of cross-modal structural information. However, existing VI dehazing methods usually rely on two key assumptions:

[0008] (1) The training phase requires fog-free ground truth images to construct supervision signals;

[0009] (2) Real infrared input is still required during the reasoning stage.

[0010] In practical applications, these two conditions are often difficult to meet simultaneously. On the one hand, truly fog-free images are hard to obtain; on the other hand, infrared imaging equipment is expensive, and infrared modal input may not be available in the testing environment. Therefore, how to effectively utilize infrared structural priors without relying on true infrared input and fog-free ground truth (GT) remains an unresolved problem.

[0011] To alleviate the problem of lacking paired labels in realistic foggy images, semi-supervised dehazing has gradually become a research hotspot. Existing methods mainly focus on knowledge transfer, domain adaptation, uncertainty modeling, and generative distribution modeling.

[0012] Uni-Removal

[11] adopts a two-stage semi-supervised framework, and realizes the transfer from the synthetic domain to the real domain through multi-teacher-student distillation and multi-granularity contrastive learning; SFSNiD

[12] combines spatial-frequency modeling and pseudo-label retraining strategy to enhance the brightness consistency in night scenes; Semi-UFormer

[13] introduces an uncertainty estimation module, and strengthens the recovery ability of high uncertainty regions through uncertainty-aware distillation; EM-B3DM

[14] uses diffusion model and EM algorithm to model hazy-clear conditional distribution to improve the generalization ability of real scenes. On the other hand, ART-SS

[15] analyzes the performance degradation problem that unlabeled data may cause from a theoretical perspective, and proposes a sample selection mechanism to improve the stability of semi-supervised learning.

[0013] However, the aforementioned semi-supervised methods mainly focus on single-modal visible light and have not yet explored how to jointly utilize cross-modal structural information for semi-supervised optimization in the absence of a fog-free ground plane. Furthermore, existing VI dehazing methods do not unify the semi-supervised training mechanism with cross-modal structural modeling within the same framework.

[0014] Therefore, under the realistic conditions of "no real infrared input + no fog GT supervision", constructing a unified cross-modal semi-supervised dehazing framework still has important research value.

[0015] References:

[0016] [1]B. Cai, X. Xu, K. Jia, C. Qing and D. Tao, "DehazeNet: An End-to-End System for Single Image Haze Removal," in IEEE Transactions on ImageProcessing, vol. 25, no. 11, pp. 5187-5198, Nov. 2016, doi: 10.1109 / TIP.2016.2598681.

[0017] [2]Luo P, Xiao G, Gao X, et al. LKD-Net: Large kernel convolutionnetwork for single image dehazing[C] / / 2023 IEEE international conference on multimedia and expo (ICME). IEEE, 2023: 1601-1606.

[0018] [3]Wang Y, Chen L, Hu B, et al. Laplace-Mamba: Laplace FrequencyPrior-Guided Mamba-CNN Fusion Network for Image Dehazing[J]. arXiv preprintarXiv:2507.00501, 2025.

[0019] [4]Zamir S W, Arora A, Khan S, et al. Restormer: Efficienttransformer for high-resolution image restoration[C] / / Proceedings of theIEEE / CVF conference on computer vision and pattern recognition. 2022: 5728-5739.

[0020] [5]Song Y, He Z, Qian H, et al. Vision transformers for single imagedehazing[J]. IEEE Transactions on Image Processing, 2023, 32: 1927-1941.

[0021] [6]Qiu Y, Zhang K, Wang C, et al. Mb-taylorformer: Multi-branchefficient transformer expanded by taylor formula for image dehazing[C] / / Proceedings of the IEEE / CVF international conference on computer vision.2023: 12802-12813.

[0022] [7]Sun Y, Zuo W, Liu M. RTFNet: RGB-thermal fusion network forsemantic segmentation of urban scenes[J]. IEEE Robotics and AutomationLetters, 2019, 4(3): 2576-2583.

[0023] [8]Li H, Wu X J. DenseFuse: A fusion approach to infrared and visibleimages[J]. IEEE transactions on image processing, 2018, 28(5): 2614-2623.

[0024] [9]Wang Z, Chen Y, Shao W, et al. SwinFuse: A residual swintransformer fusion network for infrared and visible images[J]. IEEETransactions on Instrumentation and Measurement, 2022, 71: 1-12.

[0025]

[10] Yu M, Cui T, Lu H, et al. VIFNet: an end-to-end visible–infraredfusion network for image dehazing[J]. Neurocomputing, 2024, 599: 128105.

[0026]

[11] Zhang Y, Yan D, Cai Y. Uni-Removal: A Semi-Supervised Frameworkfor Simultaneously Addressing Multiple Degradations in Real-World Images[J].arXiv preprint arXiv:2307.05075, 2023.

[0027]

[12] Cong X, Gui J, Zhang J, et al. A semi-supervised nighttimedehazing baseline with spatial-frequency aware and realistic brightnessconstraint[C] / / Proceedings of the IEEE / CVF conference on computer vision andpattern recognition. 2024: 2631-2640.

[0028]

[13] Tong M, Yan X, Wang Y, et al. Semi-uformer: Semi-superviseduncertainty-aware transformer for image dehazing[C] / / 2024 International JointConference on Neural Networks (IJCNN). IEEE, 2024: 1-8.

[0029]

[14] Liu B, Wang L, Liu M, et al. Semi-supervised Image Dehazing via Expectation-Maximization and Bidirectional Brownian Bridge Diffusion Models[J]. arXiv preprint arXiv:2508.11165, 2025.

[0030]

[15] Yasarla R, Priebe CE, Patel V M. ART-SS: an adaptive rejection technique for semi-supervised restoration for adverse weather-affected images[C] / / European Conference on Computer Vision. Cham: Springer NatureSwitzerland, 2022: 699-718. Summary of the Invention

[0031] To address the aforementioned problems, this invention proposes a joint visible-infrared dehazing method based on pseudo-infrared generation and semi-supervised optimization. The core idea is as follows: a pseudo-infrared generation network predicts structural infrared features from hazy visible light images; an infrared-guided dual-branch dehazing network is constructed; a teacher-student semi-supervised consistency constraint mechanism is employed; a color calibration module is introduced to ensure cross-supervised training stability; and no real infrared input is required during the inference phase. The method includes the following steps:

[0032] Step 1: Prepare two types of data: the first type is supervised data containing foggy images and corresponding fog-free images; the second type is unlabeled visible-infrared data containing only foggy images and infrared images.

[0033] Step 2: Construct a pseudo-infrared generation network to predict single-channel pseudo-infrared structural feature maps from hazy visible light images. The pseudo-infrared generation network includes: a multi-scale encoder-decoder structure based on discrete wavelet transform (DWT); a domain-invariant normalization module; and a structure regularization constraint module.

[0034] Step 3: Construct a dual-branch dehazing network, including: a visible light encoding branch; an infrared encoding branch; an infrared guided feature weighted fusion module; a decoding and reconstruction module; and a color calibration module. The infrared guided feature weighted fusion module is used to generate element-wise weighted coefficients based on the cross-modal structural consistency map to modulate the visible light features.

[0035] Step 4: Phased Semi-Supervised Training. The training process includes three phases: supervised pre-training; transition phase; and semi-supervised joint training phase. In the semi-supervised phase: a teacher network is constructed; the teacher network is updated using exponential moving average; and consistency loss is used to constrain the student network output.

[0036] Step 5: In the inference stage: If there is no real infrared input, the structural features are generated by the pseudo-infrared generation network; input to the dual-modal network to complete dehazing; the color calibration module can be removed.

[0037] Preferably, the method as described in claim 1 includes a pseudo-infrared generation network, an infrared-guided defogging network, and a semi-supervised training module.

[0038] In any of the above schemes, it is preferred that the pseudo-infrared generation network is constructed based on a discrete wavelet transform encoder-decoder structure, extracts structural information by performing multi-scale low-frequency and high-frequency component decomposition on the input foggy image, and outputs a single-channel pseudo-infrared structural feature map.

[0039] In any of the above schemes, it is preferred that the pseudo-infrared generation network introduces a domain-invariant normalization mechanism, performs standardization processing using intra-batch statistical information during the training phase, and updates the running statistics through a moving average method to improve the generalization ability across datasets.

[0040] In any of the above schemes, it is preferable to introduce structural regularization constraints during the pseudo-infrared generation process, apply edge consistency constraints to the generated features through gradient operators, and impose a lower bound limit on the overall contrast.

[0041] In any of the above schemes, the infrared guided feature weighted fusion module includes a cross-modal structure consistency mapping unit and an element-by-element modulation unit, which enhances the visible light features element-by-element by generating infrared guided weights.

[0042] In any of the above schemes, it is preferred that in the teacher-student consistency constraint mechanism, the teacher network parameters are updated by an exponential moving average of the student network parameters, and the consistency loss is calculated based on the output results of both.

[0043] In any of the above schemes, it is preferred that the color calibration module is a lightweight output mapping unit that participates in optimization during the training phase and can be selectively removed during the inference phase.

[0044] In any of the above schemes, the preferred embodiment is that the training process includes a supervised pre-training stage, a transition stage, and a semi-supervised joint optimization stage, wherein supervised data and unsupervised data are mixed and sampled according to a preset ratio for training. Attached Figure Description

[0045] Figure 1 The flowchart illustrates the visible-infrared joint image dehazing method based on pseudo-infrared generation and semi-supervised optimization according to the present invention.

[0046] Figure 2 This diagram illustrates the overall training framework of the visible-infrared joint image dehazing method based on pseudo-infrared generation and semi-supervised optimization proposed in this invention. The diagram shows the joint training process of the pseudo-infrared generation network and the infrared-guided semi-supervised dehazing network. The pseudo-infrared branch learns cross-modal structural representations through infrared reconstruction loss and provides structural priors for the dehazing network in the absence of real infrared input.

[0047] Figure 3 This is a schematic diagram of the pseudo-infrared generation network in this invention. The network is constructed based on the encoder-decoder structure of Discrete Wavelet Transform (DWT), and predicts single-channel pseudo-infrared structural feature maps from hazy visible light images through multi-scale feature decomposition and reconstruction.

[0048] Figure 4 This is a schematic diagram of the infrared-guided semi-supervised dehazing network in this invention. The network adopts a dual-branch structure with visible light and infrared branches, realizes cross-modal feature interaction through an infrared-guided feature weighted fusion module, and outputs the final dehazing result through a decoding module. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0050] Example 1

[0051] like Figure 1 As shown, step 100 is executed to construct a pseudo-infrared generation network. This network is used to extract scene structure information from the input hazy visible light image and generate corresponding pseudo-infrared representations, providing structural guidance information for subsequent dehazing tasks.

[0052] Step 110 is executed to predict a single-channel pseudo-infrared structure feature map from the hazy visible light image. The single-channel pseudo-infrared structure feature map is used to characterize the edge contours, target regions, and structural distribution information in the image, and serves as an auxiliary guiding feature for the subsequent dehazing network.

[0053] Step 120 is executed to construct a dual-branch dehazing network comprising a visible light branch and an infrared branch. The visible light branch is used to extract texture, color, and semantic information from the hazy visible light image, while the infrared branch is used to extract structural, edge, and region distribution information from the single-channel pseudo-infrared structural feature map.

[0054] Step 130 involves cross-modal feature fusion via an infrared-guided feature weighted fusion module. This module receives features from both the visible light and infrared branches and utilizes pseudo-infrared structural features to weighted modulate and enhance the visible light features, thereby improving the clarity and structural consistency of the dehazing result.

[0055] In step 140, during the training phase, a teacher-student consistency constraint mechanism is introduced for semi-supervised optimization. Specifically, a teacher network and a student network are constructed, and the model's ability to utilize unlabeled data is improved by constraining the output of the teacher network to be consistent with the output of the student network.

[0056] In step 150, during the training phase, the teacher network parameters are updated using an exponential moving average. These teacher network parameters are updated using an exponential moving average of the student network parameters to improve training stability and the reliability of the supervision signal.

[0057] In step 160, during the training phase, a color calibration module is set at the network output to mitigate color deviation. This color calibration module is used to adjust the colors and compensate for deviations in the restored image, thereby improving the naturalness and visual quality of the dehazing result.

[0058] During the inference phase, step 170 is executed, providing structural information through the pseudo-infrared generation network. Specifically, the hazy visible light image to be dehazed is input into the trained pseudo-infrared generation network to predict the corresponding single-channel pseudo-infrared structural feature map, which is then used as auxiliary information input into the dual-branch dehazing network.

[0059] During the inference phase, step 180 is executed, where the foggy visible light image and its corresponding pseudo-infrared structural features are input into the dehazing network to complete image restoration and output a clear image after dehazing.

[0060] Example 2

[0061] This invention is a single-image dehazing network based on pseudo-infrared structure guidance, which is a trainable end-to-end image dehazing network.

[0062] The network includes a pseudo-infrared generation network, a dual-branch defogging network, an infrared guided feature weighted fusion module, a teacher-student consistency constraint mechanism, and a color calibration module.

[0063] The pseudo-infrared generation network is used to predict single-channel pseudo-infrared structural feature maps from foggy visible light images.

[0064] The dual-branch defogging network includes a visible light branch and an infrared branch;

[0065] The infrared guiding feature weighted fusion module is used to achieve the fusion of visible light features and pseudo-infrared structural features;

[0066] The teacher-student consistency constraint mechanism is used for semi-supervised training.

[0067] The color calibration module is used to reduce color deviation in the recovery results.

[0068] This invention trains a dataset to obtain a training model, and loads the network parameters from the trained model into the network model to achieve single-image dehazing, thus restoring the image. It effectively improves the model's dehazing performance in complex foggy scenes.

[0069] The network of this invention is implemented using the Python programming language, and its usage steps are as follows:

[0070] 1. Prepare training and testing datasets, where the training dataset includes labeled dehazed data and unlabeled infrared-hazed data;

[0071] 2. Start network training using the training file. Adjust the network parameters such as batch size, lr, and epochs as needed. First train the pseudo-infrared generation network, and then train the dual-branch dehazing network.

[0072] 3. Save the trained model to a local folder, use the validation set to validate the network model. If the validation results do not improve, reduce the learning rate and continue training by resuming training at breakpoints.

[0073] 4. Save the trained model weights to a local folder. During the testing phase, first use the pseudo-infrared generation network to predict the single-channel pseudo-infrared structural feature map, then input the hazy visible light image and the single-channel pseudo-infrared structural feature map into the dehazing network, and output the dehazing result image.

[0074] Example 3

[0075] like Figure 4 As shown in the figure, this embodiment introduces the structure of an infrared-guided semi-supervised defogging network.

[0076] This network utilizes pseudo-infrared images as structural guidance information and improves dehazing performance through a cross-modal feature fusion mechanism.

[0077] The network adopts a dual-branch structure, including: (1) a visible light image feature extraction branch; and (2) an infrared structural feature guidance branch.

[0078] First, the foggy visible light image is input into the visible light branch, and multi-scale visual features are obtained through a multi-layer feature extraction module.

[0079] At the same time, the pseudo-infrared image output by the pseudo-infrared generation network is input into the infrared branch to extract the structural information of the scene.

[0080] Because infrared images are highly robust to haze, this branch can provide more stable edge and structural features.

[0081] In the feature fusion stage, the two features are fused using an infrared-guided fusion module. This module generates guiding weights based on the infrared features and adaptively modulates the visible light features, thereby enhancing structural information and suppressing haze interference.

[0082] Subsequently, the fused features are input into the decoding module, and the final dehazed image is generated by restoring the spatial resolution layer by layer.

[0083] During training, the network is optimized through supervised learning, so that the dehazing result gradually approaches the haze-free image.

[0084] By introducing pseudo-infrared structural information, the method of the present invention can better restore image details in complex haze environments and improve the clarity and structural integrity of defogging results.

[0085] Through the above embodiments, the present invention can achieve infrared-assisted defogging without the need for real infrared equipment and significantly improve the image restoration quality in complex foggy and hazy environments.

[0086] Compared to traditional single-mode dehazing methods, this invention has stronger structural recovery capabilities and better visual effects.

[0087] To better understand this invention, specific embodiments have been described in detail above, but these are not intended to limit the invention. Any simple modifications made to the above embodiments based on the technical essence of this invention still fall within the scope of this invention. Each embodiment in this specification focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to mutually. For system embodiments, since they basically correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

Claims

1. A visible-infrared joint image dehazing method based on pseudo-infrared generation and semi-supervised optimization, characterized in that, Includes the following steps: Step 1: Prepare the training dataset, including supervised data containing foggy visible light images and corresponding fog-free images, and unlabeled visible-infrared paired data containing foggy visible light images and infrared images; Step 2: Construct a pseudo-infrared generation network, using a hazy visible light image as input, to generate a single-channel pseudo-infrared structural feature map; Step 3: Construct a dual-branch visible-infrared dehazing network, which includes a visible light coding branch and an infrared coding branch; Step 4: Input the pseudo-infrared structural feature map or real infrared image generated in Step 2 into the infrared coding branch, and perform cross-modal fusion with the features extracted from the visible light branch through the infrared guided feature weighted fusion module; Step 5: Pre-train the dehazing network on supervised data using reconstruction loss; Step 6: Introduce a teacher-student consistency constraint mechanism on unlabeled visible-infrared data for semi-supervised training, where the teacher network parameters are updated by the exponential moving average of the student network parameters; Step 7: During the inference phase, when there is no real infrared image input, a pseudo-infrared structural feature map is generated through the pseudo-infrared generation network and input into the dehazing network to complete image dehazing.

2. The method as described in claim 1, characterized in that: The pseudo-infrared generation network is constructed based on a discrete wavelet transform encoder-decoder structure. It extracts image structure information by performing multi-scale low-frequency and high-frequency component decomposition on the input foggy image and outputs a single-channel pseudo-infrared structure feature map.

3. The method as described in claim 2, characterized in that: The pseudo-infrared generation network introduces a domain-invariant normalization mechanism, which uses intra-batch statistical information to standardize features during the training phase and updates the running statistics through a moving average method to improve the generalization ability across datasets.

4. The method as described in claim 1, characterized in that: In the process of pseudo-infrared generation, structural regularization constraints are introduced. Gradient operators are used to impose edge consistency constraints on the generated features and lower bound restrictions on the overall contrast, so as to enhance the structural expressiveness of pseudo-infrared images.

5. The semi-supervised visible-infrared joint image dehazing method based on pseudo-infrared guidance as described in claim 1, characterized in that, The infrared guided feature weighted fusion module includes: performing element-wise modeling of visible light structural features and infrared structural features to generate a cross-modal structural consistency mapping; generating infrared guided weights based on the consistency mapping; and using the infrared guided weights to perform element-wise modulation of visible light features to achieve cross-modal feature enhancement.

6. The method as described in claim 1, characterized in that, In the teacher-student consistency constraint mechanism, the teacher network parameters are updated by the exponential moving average of the student network parameters, and the consistency loss is calculated by using the teacher network output and the student network output.

7. The method as described in claim 1, characterized in that, The output of the dehazing network is equipped with a color calibration module, which is used to perform color mapping correction on the dehazing results to reduce the color deviation generated during cross-modal training.

8. The method as described in claim 1, characterized in that, The color calibration module is a lightweight output mapping unit that participates in optimization during the training phase and can be selectively removed during the inference phase.

9. The method as described in claim 9, characterized in that: During training, supervised and unsupervised data are mixed and sampled according to a preset ratio to improve the model's generalization ability.

Images