Infrared-visible image de-fogging method and device based on cross-modal guidance

Abstract

The application relates to an infrared-visible light image defogging method and device based on cross-modal guidance. The method comprises the following steps: acquiring a visible light image blocked by smoke and a registered infrared image; inputting the visible light image into a texture recovery branch to extract multi-scale visible light texture features, and inputting the infrared image into a structure extraction branch to extract multi-scale infrared structure features; performing soft gating fusion on the two features in a spatial self-adaptive manner through a cross-modal guidance module to generate fusion features; using a decoder to reconstruct the visible light image after defogging; and optimizing network parameters by using a structure guidance loss function. Optimal performance is achieved in both synthetic and real smoke scenes, and the accuracy of downstream target detection is significantly improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and apparatus for desmoking infrared-visible images based on cross-modal guidance. Background Technology

[0002] In safety-critical scenarios such as fire rescue and autonomous driving, localized smoke can severely degrade the quality of visible light images. Through scattering and absorption effects, it causes image blurring and reduced contrast, significantly impacting the accuracy of subsequent tasks such as target detection and scene understanding. Infrared imaging utilizes the principle of thermal radiation and has the ability to penetrate smoke, capturing obscured structural contours, but it lacks color information and fine texture. Therefore, how to leverage the structural priors of infrared images to guide the reconstruction of visible light images and achieve high-quality image restoration in localized smoke scenarios has become a key technical challenge in this field.

[0003] In the existing technology, image dehazing methods are mainly divided into three categories: image dehazing methods, single-modal image inpainting methods, and multimodal fusion methods.

[0004] Image dehazing methods are primarily based on atmospheric scattering models, assuming that scene radiation reaches the camera sensor after being scattered by the atmosphere. Classical methods include dark channel prior (DCP), color attenuation prior-based methods, and deep learning-based methods such as DehazeNet, AOD-Net, FFA-Net, and DehazeFormer. However, atmospheric scattering models assume that smoke is uniformly distributed throughout the image, making it difficult to adapt to the strong spatial heterogeneity of local smoke. When smoke exhibits localized aggregation and drastic density changes, the transmittance and atmospheric light estimated by the physical model are often inaccurate, resulting in poor dehazing performance.

[0005] Unimodal image inpainting methods treat localized smoke as occluded areas and utilize deep learning networks to learn semantic information from surrounding unoccluded areas to fill in missing content. Representative methods include LaMa, MAT, and BrushNet. These methods lack external structural guidance and rely solely on prior knowledge learned from training data, easily leading to geometric inconsistencies and semantically unreasonable content, especially when the smoke-occluded area is large.

[0006] Multimodal fusion methods leverage the complementary properties of infrared and visible light images. Infrared images can penetrate smoke, providing structural information about occluded scenes; visible light images contain rich color and texture details. Existing fusion methods include DenseFuse, FusionGAN, RFN-Nest, PIAFusion, YDTR, and TGFuse. However, these methods primarily optimize information preservation rather than smoke removal, resulting in smoke artifacts being preserved as salient structures. Furthermore, they lack explicit generation mechanisms to reconstruct color information completely lost in smoke regions, merely propagating degraded visible light signals and failing to recover true colors.

[0007] Therefore, existing technologies struggle to achieve high-quality visible light image reconstruction in localized smoke scenarios, necessitating a method that can effectively fuse infrared and visible light information, overcome modal interference, and achieve high-quality smoke-free reconstruction. Summary of the Invention

[0008] Therefore, it is necessary to provide a method and apparatus for desmog removal from infrared-visible images based on cross-modal guidance, which can achieve high-quality visible light image reconstruction in localized smoke scenes, in order to address the above-mentioned technical problems.

[0009] A method for desmoking infrared-visible images based on cross-modal guidance, the method comprising: Acquire training data, which includes smoke-obscured visible light images, infrared images registered with the smoke-obscured visible light images, and smoke-free ground truth visible light images; construct an image desmoking model; the image desmoking model includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder; The visible light image obscured by smoke is input into the texture restoration branch, which extracts multi-scale visible light texture features. The infrared image is input into the structure extraction branch, which extracts multi-scale infrared structure features. The multi-scale visible light texture features and multi-scale infrared structure features are input into the cross-modal guidance module, which performs spatially adaptive soft-gated fusion to generate fused features. The fused features are input into the decoder, which then reconstructs the visible light image after removing the smoke. Based on the smoke-free visible light image, the ground truth visible light image, and the infrared image, the value of the loss function is calculated. The value of the loss function is then used to optimize the network parameters of the texture recovery branch, the structure extraction branch, the cross-modal guidance module, and the decoder to obtain the optimized image desmoking model. The optimized image desmoking model is then used to remove smoke from the input smoke-occluded visible light image.

[0010] A smoke removal device for infrared-visible images based on cross-modal guidance, the device comprising: The training data acquisition module is used to acquire training data, which includes smoke-obscured visible light images, infrared images registered with the smoke-obscured visible light images, and smoke-free true visible light images. The model building module is used to build an image desmog model; the image desmog model includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder. The feature extraction module is used to input the smoke-obscured visible light image into the texture restoration branch, and extract multi-scale visible light texture features through the texture restoration branch; and to input the infrared image into the structure extraction branch, and extract multi-scale infrared structural features through the structure extraction branch. The cross-modal fusion module is used to input multi-scale visible light texture features and multi-scale infrared structural features into the cross-modal guidance module, and perform spatially adaptive soft-gated fusion through the cross-modal guidance module to generate fused features; The image reconstruction module is used to input the fused features into the decoder, and the decoder reconstructs the visible light image after removing the smoke; The training optimization module is used to calculate the value of the loss function based on the smoke-free visible light image, the ground truth visible light image, and the infrared image, and to optimize the network parameters of the texture restoration branch, the structure extraction branch, the cross-modal guidance module, and the decoder using the value of the loss function, so as to obtain the optimized image smoke-free model. The smoke removal module is used to remove smoke from input visible light images that are obscured by smoke using an optimized image smoke removal model.

[0011] The aforementioned method and apparatus for desmoking infrared-visible images based on cross-modal guidance first employs an explicitly decoupled texture recovery branch and structure extraction branch to extract multi-scale texture features from smoke-occupied visible light images and multi-scale structural features from registered infrared images, respectively. This dual-branch architecture avoids modal interference caused by early direct stitching or addition in traditional fusion methods, allowing visible light details and infrared structures to be optimized independently, thereby effectively preserving the original color and texture of smoke-free areas and maintaining clear contours in smoke-filled areas with the help of infrared priors. Then, a cross-modal guidance module is introduced, utilizing a spatially adaptive soft gating mechanism to dynamically adjust the fusion of visible light and infrared features pixel by pixel according to local smoke density. By employing a weighted approach, the problem of fixed fusion weights failing to adapt to drastic changes in smoke concentration is addressed. This allows the network to primarily rely on visible light in light fog (to restore true colors) and automatically switch to infrared structures in dense fog (to penetrate smoke and acquire geometry). This enables high-quality reconstruction at any location and smoke density, overcoming the spatial heterogeneity of local smoke. Finally, a bilateral edge-structure-guided loss function is designed, simultaneously constraining the edge consistency between the reconstructed image and the ground truth image, as well as between the reconstructed image and the infrared image. This loss utilizes the penetrability of the infrared image as a geometric anchor, providing effective supervision signals in areas where smoke obstruction causes the complete loss of visible light ground truth edges, thus ensuring the geometric accuracy of the reconstructed image and solving the problem of inconsistent geometric structures. The synergistic effect of these techniques allows this application to effectively preserve visible light details, restore true colors, maintain geometric consistency, and significantly improve the performance of downstream target detection tasks compared to existing technologies. It is suitable for safety-critical scenarios such as fire rescue and autonomous driving. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a cross-modal guided infrared-visible image desmog method in one embodiment. Figure 2 This is a detailed structural diagram of the cross-modal guidance module (CMGM) in one embodiment; Figure 3 This is a qualitative comparison result diagram of a synthetic smoke scene in one embodiment; Figure 3 (a) is a true visible light image (no smoke). Figure 3 (b) is a schematic diagram of a visible light image obscured by synthetic smoke. Figure 3 (c) is a true infrared image. Figure 3 (d) is the visible light image with smoke removed, generated by the DehazeFormer dehazing method. Figure 3 (e) is a visible light image with smoke removed, generated by the image reconstruction method LaMa. Figure 3 (f) is a visible light image with smoke removed, generated by the FusionGAN modality fusion method. Figure 3(g) is the visible light image after removing smoke, generated by the modal fusion method YDTR. Figure 3 (h) is a visible light image generated by the method of this patent after removing the smoke; Figure 4 This is a generalization performance diagram for a real smoke scene in another embodiment; Figure 4 (a) is a visible light scene with real smoke covering the area. Figure 4 (b) is a true infrared image. Figure 4 (c) is a schematic diagram showing that the DehazeFormer dehazing method is ineffective on visible light images with real smoke coverage. Figure 4 (d) The image reconstruction method LaMa cannot generate visible light images behind realistic smoke. Figure 4 (e) The modal fusion method YDTR cannot restore the visible light image in a realistic smoke scene. Figure 4 (f) is a color visible light image generated by (f) after removing smoke using the method of this patent; Figure 5 This is a comparison chart of the effects of object detection application in one embodiment; Figure 5 (a) A schematic diagram showing successful detection of streetlights after the image was generated using the method of this patent. Figure 5 (b) A schematic diagram showing the successful detection of a person in smoke after the image was generated using the method of this patent. Figure 5 (c) A schematic diagram of a vehicle after successfully detecting smoke following the image generated by the method of this patent. Figure 5 (d) is a schematic diagram showing the successful detection of the vehicle and person behind the vehicle after the image is generated by the method of this patent; Figure 6 This is a structural diagram of an infrared-visible image desmog device based on cross-modal guidance in one embodiment. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0014] In one embodiment, such as Figure 1 As shown, a method for desmoking infrared-visible images based on cross-modal guidance is provided, including the following steps: Step 102: Obtain training data, which includes smoke-obscured visible light images, infrared images registered with the smoke-obscured visible light images, and smoke-free ground truth visible light images; construct an image desmoking model; the image desmoking model includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder.

[0015] Smoke-obscured visible light images refer to those that are blurred and have reduced contrast due to local smoke scattering and absorption effects. Registered infrared images are those precisely aligned spatially with the visible light images; they utilize thermal radiation principles to penetrate smoke and provide structural contours of the obscured scene. Smoke-free ground truth visible light images are clean visible light images of the same scene without smoke obscuration, used for supervised training. By constructing a desmoke-free model with two branches (texture recovery and structure extraction), a foundation is provided for the subsequent explicit decoupling of visible light textures and infrared structures, avoiding mutual interference caused by early modal fusion in existing methods.

[0016] Step 104: Input the visible light image obscured by smoke into the texture restoration branch, and extract multi-scale visible light texture features through the texture restoration branch; input the infrared image into the structure extraction branch, and extract multi-scale infrared structure features through the structure extraction branch; input the multi-scale visible light texture features and multi-scale infrared structure features into the cross-modal guidance module, and perform spatially adaptive soft-gated fusion through the cross-modal guidance module to generate fused features.

[0017] The texture restoration branch employs a deep encoder structure (U-Net style) to focus on extracting residual visible light color and detail textures around the smoke-occupied area, capturing color and detail information. The structure extraction branch (SEB) uses a lightweight encoder architecture, comprising four downsampling stages. Unlike the texture restoration branch, the SEB uses standard 3×3 convolutions to extract multi-scale infrared structural features, focusing on geometric contours rather than texture details. The lightweight design reduces computational overhead because infrared images primarily provide structural priors and do not require the same feature resolution as visible light images. The asymmetrical design of the two branches, with different feature dimensions, effectively prevents the degradation of visible light features by smoke from contaminating the infrared structural information.

[0018] like Figure 2 As shown, the Cross-Modal Guidance Module (CMGM) is the core component of this application. Because visible light features are severely degraded in smoke-covered areas but retain rich information in smoke-free areas, while infrared features can provide structural information penetrating smoke, an adaptive mechanism is needed to dynamically adjust the fusion weights of the two features. The CMGM's workflow is as follows: First, infrared features are aligned with visible light features in both channels and space using 1×1 convolution and bilinear interpolation. 1×1 convolution adjusts the number of channels in the infrared features to match the visible light features; bilinear interpolation upsamples or downsamples the infrared features to the same spatial size as the visible light features.

[0019] Next, the aligned features are concatenated and then used to predict a spatial gating map through two consecutive 3×3 convolutional layers, batch normalization, and the ReLU activation function. The gating map is a single-channel feature map whose values ​​are constrained to the range (0,1) by the Sigmoid function.

[0020] CMGM performs feature fusion at four scales (1 / 2, 1 / 4, 1 / 8, and 1 / 16 resolution), ensuring good quality of reconstructed images at different resolutions through multi-scale fusion.

[0021] The cross-modal guidance module predicts the spatial gating map and determines the smoke density of the region pixel by pixel, thereby dynamically deciding whether to rely more on visible light (smokeless area) or infrared light (smoke-rich area) during fusion, solving the problem of fixed fusion weights failing due to the spatial heterogeneity of local smoke.

[0022] Step 106: Input the fused features into the decoder and reconstruct the visible light image after removing the smoke.

[0023] The decoder employs a U-Net-style decoding architecture, progressively restoring image resolution through four upsampling stages. Each upsampling stage first performs upsampling using a 2×2 transposed convolution, then concatenates the features fused with those from the corresponding encoder layer via skip connections, and finally refines the features using a 3×3 convolution. The decoder outputs a reconstructed image with the same resolution as the input image.

[0024] This step utilizes the smoke-adaptive fusion features output by the cross-modal guidance module, which can preserve the original color and texture of the smoke-free area while generating a reasonable geometric contour in the smoke area with the help of infrared structure guidance, thereby achieving high-quality reconstruction.

[0025] Step 108: Calculate the value of the loss function based on the smoke-free visible light image, the ground truth visible light image, and the infrared image. Then, use the value of the loss function to optimize the network parameters of the texture recovery branch, the structure extraction branch, the cross-modal guidance module, and the decoder to obtain the optimized image desmoking model. Use the optimized image desmoking model to remove smoke from the input smoke-occluded visible light image.

[0026] The loss function includes a structure-guided loss, which comprises two terms: the first term constrains the edge consistency between the reconstructed image and the ground truth image, and the second term utilizes infrared edges as a geometric prior (especially when the visible light ground truth edges are completely obscured by smoke) to provide additional supervision. This loss function directly addresses the problem of existing methods failing to recover the correct geometric structure when color is completely lost in smoke regions. Through end-to-end training, the model learns to adaptively utilize infrared priors at arbitrary locations and smoke densities. Upon final deployment, only the smoke-obscured visible light image and its registered infrared image need to be input to output the desmoked result.

[0027] The aforementioned cross-modal guided infrared-visible image desmog method first employs an explicitly decoupled texture recovery branch and structure extraction branch to extract multi-scale texture features from the smoke-occupied visible light image and multi-scale structural features from the registered infrared image, respectively. This dual-branch architecture avoids modal interference caused by early direct stitching or addition in traditional fusion methods, allowing visible light details and infrared structures to be optimized independently, thereby effectively preserving the original color and texture of smoke-free areas and maintaining clear contours in smoke-filled areas with the help of infrared priors. Then, a cross-modal guided module is introduced, utilizing a spatially adaptive soft gating mechanism to dynamically adjust the fusion weights of visible light and infrared features pixel by pixel based on local smoke density. This paper addresses the problem that fixed fusion weights cannot adapt to drastic changes in smoke concentration. The network primarily relies on visible light in light fog (to restore true colors) and automatically switches to infrared structures in dense fog (to penetrate smoke and acquire geometry), thus achieving high-quality reconstruction at any location and smoke density, overcoming the spatial heterogeneity of local smoke. Finally, a bilateral edge-structure-guided loss function is designed to simultaneously constrain the edge consistency between the reconstructed image and the ground truth image, and between the reconstructed image and the infrared image. This loss utilizes the penetrability of the infrared image as a geometric anchor, providing effective supervision signals in areas where smoke obstruction causes the complete loss of visible light ground truth edges, thereby ensuring the geometric accuracy of the reconstructed image and solving the problem of inconsistent geometric structures. The synergistic effect of these techniques enables this application to effectively preserve visible light details, restore true colors, maintain geometric consistency, and significantly improve the performance of downstream target detection tasks compared to existing technologies, making it suitable for safety-critical scenarios such as fire rescue and autonomous driving.

[0028] In one embodiment, the texture recovery branch adopts a U-Net-style encoder structure, which includes eight downsampling layers. Each downsampling layer includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. Each downsampling layer is followed by 2×2 max pooling for downsampling. The structure extraction branch adopts a lightweight encoder architecture, which includes four downsampling stages. Each downsampling stage uses 3×3 convolution to extract multi-scale infrared structural features.

[0029] Specifically, this design enables the texture recovery branch to have stronger feature extraction capabilities to adapt to the complex textures of visible light images, while the structure extraction branch only requires a shallower network to capture the edge contours of infrared images. The depth difference between the two branches (8 layers vs. 4 layers) significantly reduces computational overhead. Experiments show that this asymmetric structure reduces the number of parameters and achieves better performance than the symmetric dual-branch design. While ensuring sufficient extraction of infrared structure, redundant computation is avoided, and the feature dimension mismatch naturally prevents erroneous fusion of low-level features between modalities.

[0030] In one embodiment, multi-scale visible light texture features and multi-scale infrared structural features are jointly input into a cross-modal guidance module. The cross-modal guidance module performs spatially adaptive soft-gated fusion to generate fused features, including: Perform channel alignment and spatial alignment on the infrared structural features at the current scale to obtain aligned infrared features; then stitch the aligned infrared features with visible light texture features of the same scale along the channel dimension. Based on the stitched features, a gating weight between 0 and 1 is predicted pixel by pixel to form a spatial gating map; The visible light texture features and aligned infrared features are soft-gatedly fused using a spatial gating map to obtain preliminary fused features. Convolution and spatial attention operations are performed on the initial fused features. The fused features are refined by using 3×3 convolutional blocks and spatial attention gates to suppress artifacts that may be generated during the fusion process, enhance geometric consistency, and output the final fused features at the current scale.

[0031] Specifically, this process implements a complete cross-modal guidance mechanism from feature alignment and gating prediction to soft fusion and refinement. The spatial gating map is predicted by jointly using visible and infrared information, rather than relying solely on a single modality, thus enabling accurate identification of smoke region boundaries (e.g., ...). Figure 5 (As shown). The cross-modal guidance module enables the network to select a reliable modality pixel by pixel based on the smoke density, like a switch. It preserves visible light details in light fog areas and switches to infrared structure in dense fog areas, thereby reconstructing sharp contours and true colors simultaneously.

[0032] In one embodiment, the alignment process includes: A 1×1 convolution is used to adjust the number of channels of the infrared structural features to match the number of channels of the visible light texture features, and bilinear interpolation is used to adjust the spatial size of the infrared structural features to match the spatial size of the visible light texture features.

[0033] Specifically, 1×1 convolution achieves channel alignment of cross-modal features without disrupting the spatial structure; bilinear interpolation ensures precise correspondence of spatial coordinates, avoiding the checkerboard effect. This alignment method is simple and efficient, with lower computational cost compared to complex methods such as deformable convolution, and is sufficient to meet the registration accuracy requirements in local smoke scenes. It provides a strict pixel-level correspondence for subsequent pixel-by-pixel gating prediction, ensuring the spatial semantic correctness of the gating map.

[0034] In one embodiment, a gating weight between 0 and 1 is predicted pixel by pixel based on the stitched features to form a spatial gating map. This includes: inputting the stitched features sequentially into two consecutive 3×3 convolutional layers, followed by a batch normalization layer and a ReLU activation function after each convolutional layer, and finally restricting the output value to the (0,1) interval through the Sigmoid function to obtain the spatial gating map.

[0035] Specifically, the superposition of the receptive fields of two consecutive 3×3 convolutions is equivalent to a 5×5 convolution, but with fewer parameters and stronger nonlinearity. Batch normalization accelerates training convergence, ReLU introduces sparsity, and the Sigmoid outputs smooth probability values. The gating map visualization shows a smooth transition from 0 to 1 in the gating values ​​at the smoke boundary region, consistent with physical intuition. The gating map possesses clear physical interpretability, facilitates debugging and visualization, and its smoothness avoids boundary artifacts in the fusion result.

[0036] In one embodiment, a spatial gating map is used to perform soft-gated fusion of visible light texture features and aligned infrared features to obtain preliminary fused features, including: Soft-gated fusion of visible light texture features and aligned infrared features is performed using a spatial gating map to obtain preliminary fused features.

[0037] in, This represents the initial fusion features output at the l-th scale. This represents the spatial gating map predicted at the l-th scale. Represents the visible light texture features at the l-th scale. This represents the aligned infrared feature at the l-th scale. This indicates element-wise multiplication.

[0038] In one embodiment, a spatial gating graph ∈(0,1), when →1 indicates that the region depends on visible light characteristics, when →0 indicates reliance on infrared features, enabling adaptive fusion based on smoke density.

[0039] Specifically, when the value at a certain position in the gating graph is close to 1, the fused features are almost entirely taken from visible light; when it is close to 0, they are taken from infrared light. This linear interpolation form ensures the continuity and differentiability of the fusion results, achieves continuous adaptation to smoke density (implied in the gating graph), and avoids abrupt artifacts caused by hard switching.

[0040] In one embodiment, the loss function includes adversarial loss, L1 reconstruction loss, and structure guidance loss; the structure guidance loss is:

[0041] in, This represents the Sobel edge detection operator. For the reconstructed visible light image after desmoking, This is a true visible light image. Infrared image, Infrared guidance weighting coefficient; The L1 reconstruction loss is: .

[0042] Specifically, the first term in the structure-guided loss ensures that the edges of the reconstructed image match the ground truth, while the second term utilizes infrared edges as auxiliary supervision—in smoke-covered areas, the ground truth edges are unknown, but infrared edges remain reliable. L1 loss guarantees pixel-level reconstruction accuracy. Adversarial loss enhances perceptual realism. These loss functions address the problem of missing supervision signals in smoke-covered regions using traditional loss functions. By "anchoring" the geometric structure with infrared edges, the network can correctly reconstruct contours even in areas without visible light ground truth.

[0043] In one embodiment, the method employs a conditional generative adversarial network framework for training. The generator includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder. The discriminator is used to distinguish between the reconstructed image and the ground truth image. The total loss function used during training includes adversarial loss, L1 reconstruction loss, and structure guidance loss.

[0044] Specifically, a Conditional Generative Adversarial Network (cGAN) framework is used for training. The generator is the SADB-Net network, and the discriminator uses a PatchGAN structure to evaluate the difference between the reconstructed image and the real image.

[0045] The total loss function is:

[0046] in, To combat loss, we encourage the generation of images that perceive reality. To compensate for L1 reconstruction loss and ensure pixel-level fidelity; To guide the loss in the structure and ensure geometric consistency, and This indicates different weights.

[0047] Generative adversarial training further improves the clarity and realism of the reconstructed images, making the desmoking results more suitable for downstream tasks.

[0048] In a specific embodiment, the method of this application is experimentally verified.

[0049] First, a training dataset containing 5,188 pairs of registered infrared-visible images was constructed, including: Synthetic smoke subsets (4,409 pairs): Based on an atmospheric scattering model, smoke of different densities is synthesized on clean visible light images to generate smoke-occluded visible light images and corresponding clean ground truth images. Smoke density is uniformly sampled from 0.2 to 0.8 to ensure the model is robust to various smoke densities.

[0050] Realistic Smoke Subset (1,827 pairs): Realistic smoke scenes were collected in a controlled environment, including different scenes (indoor, outdoor, road), different smoke concentrations, and different lighting conditions.

[0051] All images were resized to 768×768 pixels. The Adam optimizer was used for training, with β_1=0.5 and β_2=0.999. An asymmetric learning rate strategy was employed: the generator's learning rate was 2×10^-4, and the discriminator's learning rate was 4×10^-5, with linear decay applied after the first 50 epochs. Training lasted for 100 epochs with a batch size of 2. The loss weights were set as follows: =50, =5, =0.5. The network weights are initialized using a Gaussian distribution with a mean of 0 and a standard deviation of 0.02.

[0052] On a synthetic smoke test set, the method of this application was quantitatively compared with existing methods, and the evaluation metrics included PSNR (peak signal-to-noise ratio), SSIM (structural similarity), and LPIPS (perceptual similarity).

[0053] Dehazing methods: DCP (PSNR 14.31dB), FFA-Net (PSNR 15.51dB), and DehazeFormer (PSNR 29.37dB). DehazeFormer is based on global modeling using Transformer and has better performance, but its atmospheric scattering model assumptions have limitations in localized hazy scenarios.

[0054] Image inpainting methods: LaMa (PSNR 24.23dB), MAT (PSNR 19.71dB). Due to the lack of external structural guidance, these methods produce geometrically inconsistent artifacts.

[0055] Infrared-visible light fusion methods: FusionGAN (PSNR 15.02dB), PIAFusion (PSNR 9.01dB), YDTR (PSNR 10.69dB). These methods preserve smoke artifacts and cannot reconstruct true colors.

[0056] The SADB-Net method in this application achieves PSNR of 31.27dB, SSIM of 0.853, and LPIPS of 0.137, outperforming the comparative methods in all metrics, demonstrating the effectiveness of the explicit decoupling and adaptive fusion mechanism.

[0057] In synthetic smoke scenarios (such as...) Figure 3 As shown, Figure 3 (a) is a true visible light image (no smoke). Figure 3 (b) is a schematic diagram of a visible light image obscured by synthetic smoke. Figure 4 (c) is a true infrared image. Figure 3 (d) is the visible light image with smoke removed, generated by the DehazeFormer dehazing method. Figure 3 (e) is a visible light image with smoke removed, generated by the image reconstruction method LaMa. Figure 3 (f) is a visible light image with smoke removed, generated by the FusionGAN modality fusion method. Figure 3 (g) is the visible light image after removing smoke, generated by the modal fusion method YDTR. Figure 3 (h) is a visible light image with smoke removed generated by the method of this patent), DehazeFormer produces a blurred result in the smoke boundary region; LaMa generates a geometrically inconsistent structure; YDTR preserves smoke artifacts and produces grayscale output; the method of this application restores a clear structural outline and true color information.

[0058] In realistic smoke scenarios (such as...) Figure 4 As shown, Figure 4 (a) is a visible light scene with real smoke covering the area. Figure 4 (b) is a true infrared image. Figure 4 (c) is a schematic diagram showing that the DehazeFormer dehazing method is ineffective on visible light images with real smoke coverage. Figure 4 (d) The image reconstruction method LaMa cannot generate visible light images behind realistic smoke. Figure 4 (e) The modal fusion method YDTR cannot restore the visible light image in a realistic smoke scene. Figure 4 (f) is a color visible light image generated by the method of this patent after removing smoke. The physical model of DehazeFormer fails under non-uniform smoke; LaMa produces unreasonable structures without external guidance; YDTR retains smoke artifacts and desaturates colors; the method of this application uses infrared prior to penetrate smoke of arbitrary density while retaining the original visible light details, restoring sharp contours and true colors, and verifying the effective generalization ability from synthetic training to real deployment.

[0059] Figure 5The visualization results of the spatial gating map are shown: it can be seen that the gating value is close to 1 (red) in the smoke-free area and close to 0 (blue) in the dense smoke area. The smoke boundary area transitions smoothly, which proves that the CMGM can accurately sense the smoke density and adaptively switch modes.

[0060] Training process: Follow steps 102-108. Evaluation on the synthetic smoke test set showed that our method (SADB-Net) achieved a PSNR of 31.27 dB, SSIM of 0.853, and LPIPS of 0.137, significantly outperforming comparable methods such as DehazeFormer (PSNR 29.37 dB), LaMa (24.23 dB), and YDTR (10.69 dB).

[0061] Real-world smoke scenario generalization test: such as Figure 4 As shown, our method recovers clear structural outlines and true colors in real smoke scenes, while the comparison method suffers from blurring, artifacts, or desaturation, proving that the model has the ability to generalize from synthetic training to real deployment.

[0062] Ablation experiments: Removing the dual-branch (single-stream early fusion) reduced PSNR to 26.49dB; removing CMGM (direct splicing) reduced it to 25.59dB; removing the infrared-guided loss term reduced it to 25.74dB; the complete model achieved 31.27dB, verifying the necessity of each module.

[0063] In thin smoke, this application recovers and improves detection confidence, revealing previously obscured targets. In dense smoke, where the original visible light is completely ineffective, this application's recovery is able to accurately detect and locate pedestrians and vehicles.

[0064] Downstream target detection applications: such as Figure 5 As shown, Figure 5 (a) A schematic diagram showing successful detection of streetlights after the image was generated using the method of this patent. Figure 5 (b) A schematic diagram showing the successful detection of a person in smoke after the image was generated using the method of this patent. Figure 5 (c) A schematic diagram of a vehicle after successfully detecting smoke following the image generated by the method of this patent. Figure 5 (d) This diagram illustrates the successful detection of vehicles and pedestrians after the image was generated using the method of this patent. YOLOv11 was used to test the image on both the original smoke image and the image processed by this method. The mAP_50-95 of the original smoke image was 16.1%, which was improved to 45.5% after processing by this method. Specifically, the AP for pedestrian detection increased from 5.0% to 50.1%, and the AP for vehicle detection increased from 10.0% to 55.5%. This demonstrates that the image recovered by this method has good machine-recognizable semantics.

[0065] It is worth noting that while infrared images can penetrate smoke, they lack the color details required for scene understanding; the method in this application preserves these spectral cues while utilizing infrared for structure guidance.

[0066] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0067] In one embodiment, such as Figure 6 As shown, an infrared-visible image desmoking device based on cross-modal guidance is provided, comprising: The training data acquisition module 602 is used to acquire training data, which includes a smoke-obscured visible light image, an infrared image registered with the smoke-obscured visible light image, and a smoke-free true visible light image. Model building module 604 is used to build an image desmog model; the image desmog model includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder. The feature extraction module 606 is used to input the smoke-obscured visible light image into the texture restoration branch and extract multi-scale visible light texture features through the texture restoration branch; and to input the infrared image into the structure extraction branch and extract multi-scale infrared structure features through the structure extraction branch. The cross-modal fusion module 608 is used to input multi-scale visible light texture features and multi-scale infrared structural features into the cross-modal guidance module, and perform spatially adaptive soft-gated fusion through the cross-modal guidance module to generate fused features; Image reconstruction module 610 is used to input fused features into the decoder and reconstruct the visible light image after removing the smoke through the decoder; The training optimization module 612 is used to calculate the value of the loss function based on the smoke-free visible light image, the ground truth visible light image, and the infrared image, and to optimize the network parameters of the texture restoration branch, the structure extraction branch, the cross-modal guidance module, and the decoder using the value of the loss function, so as to obtain the optimized image smoke-free model. The smoke removal processing module 614 is used to remove smoke from the input smoke-occupied visible light image using an optimized image smoke removal model.

[0068] For specific limitations regarding the infrared-visible light image desmoke removal device based on cross-modal guidance, please refer to the limitations of the infrared-visible light image desmoke removal method based on cross-modal guidance mentioned above, which will not be repeated here. Each module in the aforementioned infrared-visible light image desmoke removal device based on cross-modal guidance can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for desmoking infrared-visible images based on cross-modal guidance, characterized in that, The method includes: Acquire training data, which includes a visible light image obscured by smoke, an infrared image registered with the visible light image obscured by smoke, and a ground truth visible light image without smoke; construct an image desmoking model; the image desmoking model includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder; The smoke-obscured visible light image is input into the texture restoration branch, through which multi-scale visible light texture features are extracted; the infrared image is input into the structure extraction branch, through which multi-scale infrared structural features are extracted; the multi-scale visible light texture features and the multi-scale infrared structural features are input into the cross-modal guidance module, through which spatially adaptive soft-gated fusion is performed to generate fused features; The fused features are input into the decoder, and the visible light image after removing the smoke is reconstructed by the decoder. Based on the smoke-free visible light image, the ground truth visible light image, and the infrared image, the value of the loss function is calculated, and the network parameters of the texture recovery branch, the structure extraction branch, the cross-modal guidance module, and the decoder are optimized using the value of the loss function to obtain an optimized image desmoking model; the optimized image desmoking model is then used to desmok the input smoke-occluded visible light image.

2. The method according to claim 1, characterized in that, The texture restoration branch adopts a U-Net-style encoder structure, which includes eight downsampling layers. Each downsampling layer includes a 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function connected in sequence. Each downsampling layer is followed by 2×2 max pooling for downsampling. The structure extraction branch adopts a lightweight encoder architecture, which includes four downsampling stages. Each downsampling stage uses 3×3 convolution to extract multi-scale infrared structural features.

3. The method according to claim 1, characterized in that, The multi-scale visible light texture features and the multi-scale infrared structural features are jointly input into the cross-modal guidance module. The cross-modal guidance module performs spatially adaptive soft-gated fusion to generate fused features, including: Perform channel alignment and spatial alignment on the infrared structural features at the current scale to obtain aligned infrared features; then stitch the aligned infrared features with visible light texture features of the same scale along the channel dimension. Based on the stitched features, a gating weight between 0 and 1 is predicted pixel by pixel to form a spatial gating map; The visible light texture features and the aligned infrared features are soft-gated and fused using the spatial gating map to obtain preliminary fused features. Perform convolution and spatial attention operations on the preliminary fused features to output the final fused features at the current scale.

4. The method according to claim 3, characterized in that, The alignment process includes: The number of channels of the infrared structural feature is adjusted to match the number of channels of the visible light texture feature using 1×1 convolution, and the spatial size of the infrared structural feature is adjusted to match the spatial size of the visible light texture feature using bilinear interpolation.

5. The method according to claim 3, characterized in that, Based on the stitched features, a gating weight between 0 and 1 is predicted pixel by pixel to form a spatial gating map, including: The concatenated features are sequentially input into two consecutive 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function. Finally, the output value is restricted to the (0,1) interval by the Sigmoid function to obtain the spatial gating map.

6. The method according to claim 3, characterized in that, The visible light texture features and the aligned infrared features are soft-gated and fused using the spatial gating map to obtain preliminary fused features, including: The visible light texture features and the aligned infrared features are soft-gated and fused using the spatial gating map to obtain preliminary fused features. in, This represents the initial fusion features output at the l-th scale. This represents the spatial gating map predicted at the l-th scale. Represents the visible light texture features at the l-th scale. This represents the aligned infrared feature at the l-th scale. This indicates element-wise multiplication.

7. The method according to claim 6, characterized in that, The spatial gating diagram ∈(0,1), when →1 indicates that the region depends on visible light characteristics, when →0 indicates reliance on infrared features, enabling adaptive fusion based on smoke density.

8. The method according to claim 1, characterized in that, The loss function includes adversarial loss, L1 reconstruction loss, and structure guidance loss; the structure guidance loss is: in, This represents the Sobel edge detection operator. For the reconstructed visible light image after desmoking, This is a true visible light image. Infrared image, Infrared guidance weighting coefficient; The L1 reconstruction loss is: 。 9. The method according to claim 1, characterized in that, The method is trained using a conditional generative adversarial network framework. The generator includes the texture restoration branch, the structure extraction branch, the cross-modal guidance module, and the decoder. The discriminator is used to distinguish between the authenticity of the reconstructed image and the ground truth image. The total loss function used during training includes adversarial loss, L1 reconstruction loss, and structure-guided loss.

10. A smoke removal device for infrared-visible light images based on cross-modal guidance, characterized in that, The device includes: The training data acquisition module is used to acquire training data, which includes a smoke-obscured visible light image, an infrared image registered with the smoke-obscured visible light image, and a smoke-free true visible light image. The model building module is used to build an image desmog model; the image desmog model includes a texture restoration branch, a structure extraction branch, a cross-modal guidance module, and a decoder. The feature extraction module is used to input the smoke-obscured visible light image into the texture restoration branch, and extract multi-scale visible light texture features through the texture restoration branch; and to input the infrared image into the structure extraction branch, and extract multi-scale infrared structural features through the structure extraction branch. A cross-modal fusion module is used to input the multi-scale visible light texture features and the multi-scale infrared structural features into the cross-modal guidance module, and perform spatially adaptive soft-gated fusion through the cross-modal guidance module to generate fused features; The image reconstruction module is used to input the fused features into the decoder and reconstruct the visible light image after removing the smoke through the decoder; The training optimization module is used to calculate the value of the loss function based on the smoke-free visible light image, the ground truth visible light image, and the infrared image, and to optimize the network parameters of the texture restoration branch, the structure extraction branch, the cross-modal guidance module, and the decoder using the value of the loss function, so as to obtain the optimized image smoke-free model. The smoke removal module is used to remove smoke from the input visible light image that is obscured by smoke using the optimized image smoke removal model.

Images