Infrared and visible light image fusion method suitable for complex meteorological environment

Abstract

The invention provides an infrared and visible light image fusion method suitable for a complex meteorological environment. A multi-modal image fusion system is constructed based on a convolutional neural network and a hybrid expert mechanism MoE. The system comprises a parallel double-branch feature extraction module, a degradation repair SDMR module, a hybrid expert network module, a residual connection module and a three-branch reconstruction module. An infrared image and a visible light image are respectively input into a double-branch module for multi-scale feature extraction, and an SDMR module is used for carrying out defogging, low-illumination enhancement and rain and snow removal processing on the visible light image. The processed features are dynamically fused through expert networks, cross-modal interaction is enhanced through multi-modal residual connection, and finally the three-branch reconstruction module outputs reconstructed infrared and visible light and fused images. According to the method, a gating network and expert network dynamic distribution mechanism is introduced into a hybrid expert mechanism, so that the thermal radiation fidelity and texture detail quality of the fused image are improved, and the problems of low fusion quality, much noise, detail loss and the like under complex weather are effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to a method for fusing infrared and visible light images suitable for complex meteorological environments. Background Art

[0002] Due to their different imaging mechanisms, infrared and visible light images exhibit significant complementarity. Visible light images provide rich texture and color information under well-lit conditions, but their performance is poor in complex environments such as low illumination and rain and fog, and they are easily affected by environmental factors. In contrast, infrared images can clearly depict heat sources at night or in fog and haze, but their resolution is relatively low, resulting in a lack of detailed information. Therefore, fusing these two types of images can significantly enhance visual perception in complex meteorological conditions, offering broad application prospects in security surveillance, autonomous driving, military reconnaissance, and other fields. However, in practical applications, infrared and visible light images often suffer from large registration errors, excessive noise interference, and loss of detailed information due to factors such as inclement weather, dramatic illumination changes, and complex background interference. Traditional image fusion algorithms struggle to simultaneously preserve information and suppress noise, and lack adaptability to multimodal feature differences, which in turn affects the quality of the fused image. Even deep learning fusion methods based on convolutional neural networks (CNNs) or generative adversarial networks (GANs) suffer from insufficient generalization and poor adaptability to complex and degraded scenes.

[0003] To this end, the invention designs a method for fusion of infrared and visible light images suitable for complex meteorological environments, and constructs a new multimodal neural network system. The system extracts feature maps of different channels of infrared and visible light images through a multi-channel feature extraction strategy. In view of the fact that visible light images are easily affected by degradation factors, the invention introduces prior knowledge to perform degradation repair, including preprocessing operations such as defogging, low-light enhancement and rain removal. In addition, the invention designs a mixture of experts (MoE) mechanism based on a gating mechanism to dynamically allocate the participation of different expert sub-networks in the fusion process, so that each expert network can be optimized for specific modal features or degradation types, thereby realizing a fusion strategy with strong adaptability and high robustness. At the same time, the network also introduces residual connections and three-branch reconstruction modules between multiple modalities to synergistically improve the clarity and detail fidelity of the fused image.

[0004] Experimental results demonstrate that this method demonstrates excellent performance across a wide range of degradation scenarios, making it suitable for a variety of practical applications, including all-weather intelligent perception. This method not only achieves high-quality fusion of thermal radiation information from infrared images and texture details from visible light images at both the structural and semantic levels, but also provides a more reliable image input foundation for intelligent perception systems operating under complex meteorological conditions. Therefore, this method is widely applicable to multimodal perception tasks in scenarios such as all-weather surveillance, autonomous driving, and border patrols, demonstrating promising engineering application value and widespread adoption. Summary of the Invention

[0005] The purpose of this invention is to propose a method for fusion of infrared and visible light images suitable for complex meteorological environments. This method aims to address the numerous issues that exist in existing techniques for fusion of infrared and visible light images in complex meteorological environments, including noise sensitivity, unbalanced illumination, the effects of harsh environments such as rain, fog, and snow, and insufficient adaptability to dynamic scenes. The core technical solutions proposed in this invention include:

[0006] Step 1: Construct a parallel two-branch input network, taking infrared images and visible light images as input respectively. Each branch contains a feature extraction module backbone consisting of a convolutional layer and an LReLU activation function, which extracts shallow details and deep semantic features through a hierarchical structure (16 / 32 / 64 / 128 channels).

[0007] Step 2: Design a priori-based degradation restoration network (SDMR) to adaptively enhance and restore shallow features of visible light images. This network achieves integrated image restoration through multi-task joint optimization.

[0008] Step 3: The multi-channel features of the image extracted in step 1 and the shallow features of the visible light image enhanced by the degradation restoration network SDMR in step 2 are used as parallel inputs to the fusion framework of the hybrid expert mechanism MoE.

[0009] Step 4: The MoE hybrid expert mechanism consists of a gated network, GateNet, and a multi-way expert network, ExpertNet. The MoE network's gating mechanism dynamically distributes input features to different expert sub-networks. The infrared feature expert focuses on analyzing infrared modality-specific thermal radiation characteristics and cross-modal alignment, while the visible light expert optimizes and restores visible light details. The expert outputs are then weighted and fused to generate fused features that combine infrared target saliency with the richness of visible light details.

[0010] Step 5: The mixed outputs of MoEs of different channels are residually connected through the residual block to achieve efficient interaction of cross-modal features and stability of the deep network.

[0011] Step 6: Design a multi-task decoder and three-branch output. The fused feature representation is fed simultaneously into three parallel decoding branches: one for reconstructing the infrared image, one for reconstructing the visible light image, and one for outputting the fused image. This three-branch structure allows the network to simultaneously optimize both structural reconstruction and semantic understanding during supervised learning, thereby improving the multidimensional quality of the fused image.

[0012] Step 7: Design an end-to-end training mechanism driven by a joint loss function. This mechanism comprehensively considers the visible light reconstruction loss, infrared image reconstruction loss, and fusion image quality loss, and through joint optimization ensures that the network achieves high-quality fusion output in terms of structural fidelity and modality complementarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the overall network structure of the present invention;

[0014] Figure 2 Schematic diagram of the dehazing network DehazeNet of the present invention;

[0015] Figure 3 Schematic diagram of the lightweight low-light enhancement network zero-DCE of the present invention;

[0016] Figure 4 A schematic diagram of a hybrid expert mechanism MoE network according to the present invention;

[0017] Figure 5 It is the residual block of the present invention; Specific implementation plan

[0018] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings: This algorithm provides a method for fusing infrared and visible light images suitable for complex meteorological environments, such as Figure 1 As shown in the figure, it includes the feature extractor Backbone, the shallow degradation restoration module SDMR for visible light images, the hybrid expert mechanism MoE network, the residual block Residual Block, the three-branch output of the image reconstruction part, and the training mechanism based on the loss function.

[0019] Step 1: This paper proposes a parallel dual-branch multimodal feature extraction network structure. The network includes two independent input branches, one for receiving infrared images and the other for receiving visible light images. Each branch integrates four independent feature extraction modules (backbone modules) to construct and extract multi-channel feature representations for the input images. The backbone network module consists of a convolutional layer and an LReLU activation function.

[0020] The network adopts a hierarchical feature learning mechanism to extract shallow and deep features with 16, 32, 64 and 128 channels from the bottom up. The shallow structure focuses on low-level features such as image edges and textures, while the deep structure extracts high-level features such as semantics and structure, thereby enhancing the complementarity and expression capabilities of different modalities in multi-layer semantics.

[0021] Step 2: Perform degradation restoration on the shallow features of the visible light image in step 1. A priori knowledge-guided image degradation restoration network named SDMR (Semantic Degradation Aware Multi-task Restoration) is used to adaptively enhance shallow features and restore degraded areas in visible light images. The network is based on the optimized low-light enhancement network zero-DCE, the atmospheric scattering model-based DehazeNet, and the single-channel rain and snow removal. The DehazeNet network is as follows: Figure 2 As shown in Figure 3 The network fully integrates the prior knowledge of image content and structural features, and improves the perception of local details and edge textures through a guidance mechanism, thereby achieving high-quality image reconstruction.

[0022] The SDMR network adopts a multi-task joint optimization framework to collaboratively model multiple subtasks of image enhancement and degradation restoration, effectively solving the problem of image quality degradation caused by factors such as noise, blur, and underexposure. The specific implementation is as follows:

[0023] Step 2.1: If Figure 2 As shown in the figure, the shallow features of the visible light image are input into an end-to-end single-image dehazing algorithm DehazeNet network. The network inputs a visible light image with haze, outputs a medium transmission map, and then restores the haze-free image through the atmospheric scattering model. The module adopts a convolutional neural network architecture, and each layer is designed to apply existing prior assumptions.

[0024] The atmospheric scattering model describes the formation process of fog, which can be expressed as follows:

[0025] I(x)=J(x)t(x)+α(1-t(x)) (1)

[0026] Where I(x) represents the original image with fog, J(x) is the restored clear image without fog, t(x) is the medium transmittance of the atmosphere, and α is the global atmospheric light value.

[0027] The network uses a cascaded convolution structure to extract image features and performs multi-scale feature learning through channel transformation. Based on feature extraction, the network is divided into two parallel branches: the transmittance estimation branch compresses the feature map into a single-channel output of atmospheric transmittance t(x) through multiple convolutional layers, and uses the Sigmoid function to ensure the physical rationality of the transmittance estimation; the atmospheric light estimation branch uses adaptive average pooling for global feature aggregation, and then reduces the dimension through a fully connected layer to obtain the scene atmospheric light value α. Finally, the network performs image dehazing and reconstruction based on the estimated transmittance and atmospheric light value, combined with the atmospheric scattering model. The reconstruction image formula obtained by transforming Equation (1) is as follows:

[0028] J(x)=(I(x)-α) / t(x)+α (2)

[0029] Step 2.2: If Figure 3 As shown in the figure, the shallow features of the visible light image are input into the zero-DCE low-light enhancement network. This network consists of three convolutional layers with LReLU activation functions, and performs residual connections to enhance feature extraction. Finally, it outputs the illumination estimate of the image pixel. Based on the illumination characteristic curve trained by this network, the illumination of each pixel in the visible light image is adjusted to achieve low-light degradation restoration and enhancement.

[0030] Step 2.3: To address the image quality degradation caused by rain and snow in visible light images, we propose a shallow feature restoration module based on a combination of convolution and the LReLU activation function. This module uses convolution to model the local structure and complete the contextual information in rain- and snow-interfered areas. The LReLU activation function is then used to enhance the nonlinear feature representation, effectively reconstructing and restoring details in occluded and degraded areas.

[0031] Step 3: The multi-channel image features extracted in step 1 and the shallow features of the visible light image after the degradation restoration network SDMR in step 2 are used as input and introduced into the hybrid expert mechanism fusion framework in parallel.

[0032] Step 4: The MoE fusion module is the core component of multimodal feature fusion. The network model diagram is as follows: Figure 4 As shown in the figure, the hybrid expert mechanism MoE comprises a gate control network GateNet and a multi-branch expert network ExpertNet.

[0033] Step 4.1: The gating network is a learnable module that dynamically determines the participation of each expert in the hybrid expert structure based on the input. It consists of a simple feedforward neural network with raw features as input and a set of weights G(x) as output. These weights represent the activation strength of each expert and are normalized by the Softmax function to form a probability distribution, thereby performing a weighted combination of the outputs of all experts. The gating network output after Softmax normalization is G σ (x) is shown in the following formula:

[0034] G σ (x) = Softmax(x·W g ) (3)

[0035] The design of the gating network not only determines the dynamic behavior of the model, but also has an important impact on the efficiency and expressiveness of the model. Furthermore, mechanisms such as Top-K (K=1) strategy and noise perturbation are introduced to achieve sparse activation and improve generalization ability.

[0036] Step 4.2: The multimodal fusion model realizes dynamic feature integration through the hybrid expert MoE mechanism. Its final output is generated by the weighted sum of the gate network GateNet weights and the expert network ExpertNet output. The specific mathematical expression is:

[0037]

[0038] Where x represents the input data, G(x) i is the weight of the gating network for the i-th expert, a weight value learned according to the input x, which represents the confidence of the current input on the i-th expert; E i (x) represents the output of the i-th expert on x.

[0039] Step 5: Propose a multi-channel MoE feature fusion method based on residual connection, the core architecture of the residual block is as follows Figure 5 As shown in the figure, the network feeds the mixed outputs of multiple experts into a residual block consisting of two layers of 3×3 convolutions and LReLU activation functions, which enhance features. The first convolution layer extracts local cross-modal correlation features, while the second layer refines features. Residual connections ensure that the network maintains training stability even with increasing depth.

[0040] Step 6: Design a three-branch image reconstruction method, such as Figure 1As shown in the right half, the features after residual connection are input into three decoding branches in parallel: the infrared reconstruction branch uses two layers of 3×3 convolution and LReLU activation function, the visible light reconstruction branch shares network parameters but separates modal features through the feature decoupling module; the fused image generation branch uses a deeper three-layer 3×3 convolution structure combined with LReLU nonlinear activation to achieve effective fusion of multimodal features.

[0041] Step 7: Design an end-to-end training mechanism based on multi-task joint optimization to achieve high-quality image fusion by designing a joint loss function. This mechanism contains three key loss terms: visible light reconstruction loss Loss vis , infrared reconstruction loss Loss ir And fusion quality loss Loss fusion , integrating adversarial training and CLIP semantic consistency evaluation, its model loss function is shown as follows:

[0042] Loss total =Loss fusion +Loss vis +Loss ir (5)

[0043] Among them, Loss total Represents the total loss of the model, Loss fusion Represents fusion loss, Loss ir 、Loss vi Represents the loss of reconstructing infrared image and visible light image respectively. The fusion loss includes maximum gradient loss, intensity loss and color consistency loss. fusion Defined as:

[0044] Loss fusion =L int +L grad +L color (6)

[0045] Loss int The intensity loss aims to make the fused image have a better intensity distribution, highlight the significant targets of infrared and visible light, and ensure the saliency of the target by maximizing the intensity value of the result. It is defined as:

[0046]

[0047] Loss grad The maximum gradient loss aims to preserve the maximum edges in the two source images and obtain a clearer texture representation, which can be expressed as:

[0048]

[0049] Loss color Color consistency aims to make the colors of the fused image and the visible light image consistent. The visible light image is transposed to the YCbCr space and constrained using the Euclidean distance of the Cb and Cr channels. The color consistency loss can be expressed as:

[0050]

Claims

1. A method for fusion of infrared and visible light images suitable for complex meteorological environments. Characterized by: The following steps are involved: S1. Construct a multimodal image fusion system based on a convolutional neural network and a hybrid expert mechanism (MoE). The system includes a parallel two-branch feature extraction module, a degradation restoration (SDMR) module, a hybrid expert network (MoE) module, a residual connection module, and a three-branch reconstruction network module. S2. Input the infrared image and the visible light image into the dual-branch feature extraction module to extract shallow texture information and deep semantic information; S3. Degraded features of the shallow layer of the visible light image are repaired by the SDMR module; S4. Input the shallow and deep features of infrared and visible light into the hybrid expert network MoE module, and dynamically assign weights to each expert sub-network through the gating network; S5. The mixed features output by the MoE module are enhanced by the residual connection module; S6. Generate reconstructed visible light images, reconstructed infrared images, and fused images through a three-branch reconstruction network, and optimize network training with a multi-task loss function.

2. The method according to claim 1, characterized in that The dual-branch feature extraction module includes four cascaded Backbone submodules, each of which consists of a convolutional layer and an LReLU activation function; The module adopts a hierarchical feature learning mechanism to output shallow and deep features of 16, 32, 64, and 128 channels in sequence, where shallow features are used to extract image edges and textures, and deep features are used to extract semantic and structural information.

3. The method according to claim 1, characterized in that The degradation repair SDMR module includes the following sub-networks connected in parallel: (1) Low-light enhancement network zero-DCE, which consists of three convolutional layers and LReLU activation function, and outputs the light-adjusted visible light image through residual connection; (2) DehazeNet, which includes a transmittance estimation branch and an atmospheric light estimation branch. The transmittance estimation branch outputs a single-channel transmittance map through a convolutional layer and a sigmoid function, and the atmospheric light estimation branch outputs an atmospheric light value through a global average pooling and a fully connected layer, and reconstructs the dehazed image based on the atmospheric scattering model. (3) The rain and snow removal module consists of a convolutional layer and an LReLU activation function to perform local structural repair on rain and snow degradation areas.

4. The method according to claim 1, wherein The hybrid expert network MoE module includes the following gating network and multi-way expert sub-network: (1) GateNet, a gated network, is used to dynamically generate weight distribution coefficients for each expert network based on input features; (2) Multi-channel expert sub-network ExpertNet, including infrared feature expert sub-network and visible light feature expert sub-network, optimizes infrared thermal radiation features and visible light detail features respectively.

5. The method according to claim 1, wherein The residual connection module consists of a double-layer 3x3 convolutional layer and an LReLU activation function, which is used to nonlinearly enhance the mixed features output by the MoE module.

6. The method according to claim 1, characterized in that The three-branch reconstruction network includes: (1) Infrared reconstruction branch, consisting of two layers of 3×3 convolution and LReLU activation function; (2) The visible light reconstruction branch shares parameters with the infrared reconstruction branch and separates modal features through a feature decoupling module; (3) Fusion image generation branch, consisting of three layers of 3×3 convolution and LReLU activation function; The training loss function of the reconstruction network includes infrared image reconstruction loss, visible light image reconstruction loss and fusion image loss.

Images