Self-adaptive infrared and visible light image fusion method based on environmental perception

Abstract

The invention discloses a self-adaptive infrared and visible light image fusion method based on environmental perception. The method comprises the following steps: step 1, segmenting an infrared characteristic image and a visible light characteristic image into a plurality of block images which are not overlapped with each other along a spatial dimension; step 2, embedding the extracted features in each block image into the corresponding block image by using an encoder to obtain a corresponding final feature image; 3, obtaining an environmental factor vector; 4, adjusting the splicing and fusion of the final feature map of the infrared image and the final feature map of the visible light image by using the environmental factor vector to obtain a fused image; according to the method, the environment information contained in the visible light image is used as priori knowledge to guide the infrared and visible light fusion process, so that the fusion process can be adaptively adjusted according to the weather condition and the illumination level in the environment, and a more reliable fusion result is ensured to be obtained in the diversified and dynamic environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared and visible light image fusion, and in particular relates to an adaptive infrared and visible light image fusion method based on environment perception. Background Art

[0002] With the rapid development of sensing hardware, multimodal imaging has attracted great attention in a wide range of applications. Environmental perception capability is crucial for modern sensing systems, enabling them to perceive their surroundings in an adaptive and efficient manner, thereby improving performance in all-weather environmental conditions. Therefore, the integration of infrared and visible light images has become a powerful and widely used technique in visual sensing systems, leveraging the inherent complementarity of the two modalities to achieve environmental perception. Specifically, the fused image generated by this technique combines the advantages of both modalities, which can not only highlight the thermal characteristics of objects, but also represent complex and detailed visual information. Therefore, the infrared-visible light fusion technology is of great significance to different application fields, including security monitoring, drone navigation, and environmental monitoring, etc.

[0003] Fundamentally, image fusion can be viewed as a process of recombining received visual information of interest (e.g., shape, texture, and edge of the foreground) by decomposition. Therefore, the key to image fusion lies in the design and implementation of decomposition and recombinant strategies, which is the difference between image fusion methods. Traditional image fusion methods are analytical synthesis models built using expert knowledge in the field of signal processing. These models cover a range of techniques, including multi-scale transform-based methods, subspace-based methods, sparse representation-based methods, and saliency representation-based methods. By leveraging these signal models, researchers are able to obtain effective fusion results with interpretability and a solid theoretical foundation, enabling them to better understand and control the fusion process. However, these methods may have limitations in adaptively extracting complex hierarchical features of different images and are computationally inefficient when processing large-scale data. The emergence of deep learning has profoundly reshaped the field of infrared and visible light image fusion. These methods use a series of hierarchical nonlinear transformations to decompose and generate visible and infrared images. They mainly include methods based on convolutional neural networks (CNNs), methods based on autoencoders, methods based on generative adversarial networks, and methods based on transformers. Deep learning methods are good at automatically and adaptively learning complex features and patterns, and can effectively process large-scale data.

[0004] Traditional image fusion methods usually use the same transformation to extract features from images from different sources, without taking into account the feature differences between visible light images and infrared images, resulting in poor expressiveness of the extracted features. In addition, the fusion rules and decomposition methods need to be set manually, and are not robust enough to noise and artifacts, and have limited ability to handle complex scenes or environments.

[0005] Existing deep learning-based fusion algorithms have achieved advanced performance, but have failed to incorporate environmental prior knowledge in the modeling process. Specifically, visible light images are prone to overexposure under strong direct sunlight, resulting in degradation of image quality. In severe weather conditions such as rain, fog and haze, image clarity and contrast are also greatly reduced. For infrared images, strong light can cause unclear edges of objects and loss of image details. Weather conditions such as rain, snow and haze can scatter or absorb infrared radiation, resulting in reduced resolution and contrast of infrared images.

[0006] Although existing fusion methods have demonstrated excellent performance, their practical deployment in different environments still faces significant challenges. A key reason is that environmental factors are not fully considered in current fusion models, resulting in a fixed fusion process that cannot be dynamically adjusted according to environmental changes. As a result, these methods have difficulty responding effectively to changing environmental conditions, which hinders their performance in real-world scenarios. Specifically, environmental factors such as weather conditions and changing lighting levels significantly affect the imaging capabilities of visual sensors, and this environmental information is encoded in the acquired images during the imaging process. Current fusion models often ignore these environmental factors hidden in the input images, and the developed decomposition and recombination strategies cannot explicitly perceive the environmental information, making it impossible for the fusion process to achieve adaptive control, resulting in a further deterioration in the quality of the fusion output and leading to suboptimal, unstable or even failed fusion results in several key scenarios. Summary of the invention

[0007] The purpose of the present invention is to provide an adaptive infrared and visible light image fusion method based on environmental perception to solve the problem that the fusion process cannot achieve adaptive control, resulting in further degradation of the quality of the fusion output and leading to suboptimal, unstable or even failed fusion results in several key scenarios.

[0008] The present invention adopts the following technical solution: an adaptive infrared and visible light image fusion method based on environmental perception, comprising:

[0009] Step 1: Use a convolutional neural network to extract features of the infrared image and the visible light image to obtain an infrared feature map and a visible light feature map respectively; and divide the infrared feature map and the visible light feature map into multiple non-overlapping block maps along the spatial dimension;

[0010] Step 2: Use the encoder to embed the extracted features in each block map into the corresponding block map to obtain the corresponding final feature map;

[0011] Step 3: Extract weather information features and lighting information features from the visible light image, concatenate the weather information features and lighting information features, reduce their dimensions, and normalize them to obtain an environmental factor vector;

[0012] Step 4: Use the environmental factor vector to adjust the final feature map of the infrared image and the final feature map of the visible light image to obtain a fused image.

[0013] Furthermore, in step 4, the fused image is obtained by a CNN-based decoder combined with a loss function, and the loss function is:

[0014]

[0015] In the formula, α, β and γ are all hyperparameters. is the fidelity loss function, is the strength loss function,

[0016] is the texture loss function.

[0017] Further,

[0018]

[0019] In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I vi is the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image.

[0020] Further,

[0021]

[0022] In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I vi is the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image.

[0023] Further,

[0024]

[0025] In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I viis the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image. The beneficial effects of the present invention are:

[0026] The present invention uses the environmental information contained in the visible light image as a priori knowledge to guide the fusion process of infrared and visible light, so that the fusion process can be adaptively adjusted according to the weather conditions and light levels in the environment, thereby ensuring more reliable fusion results in a diverse and dynamic environment;

[0027] The present invention constructs a network architecture of encoder-fusion module-decoder based on the concept of environmental perception, uses the CNN-Transformer model to extract and fuse features, and realizes adaptive image fusion for current weather and lighting conditions;

[0028] The present invention integrates complementary environmental information from source images by adaptively considering weather and lighting conditions. This environmental perception capability enables the model to dynamically adapt to changing environmental factors in the real world, thereby addressing the limitations of existing methods in dealing with these fluctuations and improving image fusion performance in complex real-world scenes.

[0029] The present invention can adaptively adjust the fusion process according to the dynamic weather conditions and light levels in the surrounding environment to generate high-quality fused images. This adaptive fusion capability optimizes the fusion results according to the dynamic real-world context, rather than relying on the traditional unified fusion strategy. While improving the adaptability and robustness of the model, the present invention demonstrates superior performance in terms of fused image information content, visual quality, correspondence from source to fused image, and retention of salient target information and detail texture.

[0030] The present invention can dynamically adapt to real-time environmental factors, enhance the adaptability and robustness of the algorithm, and thus ensure more reliable fusion results in a diverse and dynamic environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 The overall network architecture diagram of the present invention;

[0032] Figure 2 This is the qualitative comparison result on the MSRS and M3FD datasets of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] The present invention discloses an adaptive infrared and visible light image fusion method based on environmental perception. The overall network architecture is shown in the figure below: Figure 1 As shown, including:

[0035] Step 1: Use a convolutional neural network to extract the features of the infrared image and the visible light image to obtain an infrared feature map and a visible light feature map respectively; and divide the infrared feature map and the visible light feature map into multiple non-overlapping block maps along the spatial dimension.

[0036] The present invention belongs to the field of deep learning and infrared-visible light image fusion. Compared with traditional image fusion technology and image fusion methods based on deep learning, the present invention uses the environmental information contained in the visible light image to provide prior knowledge for the image fusion process, guides the fusion process, and realizes adaptive image fusion according to environmental conditions; uses a multi-layer perceptron and a vision transformer (VIT) to mine the weather and lighting information contained in the visible light image, and realizes the all-day adaptive fusion of infrared and visible light according to the image fusion paradigm of encoder-fusion module-decoder.

[0037] The present invention sends the infrared image and the visible light image to the visual factor perception encoder to extract preliminary visual features, and sends the visible light image to the environment perception encoder to extract environmental features including weather and lighting information. Then, the extracted features are sent to the environment perception fusion module, and under the guidance of environmental prior knowledge, the infrared and visible light adaptive features are fused, and finally the fused features are sent to the decoder for image generation.

[0038] For infrared and visible light images, the network structure of the visual factor-aware encoder is the same, but with different parameters. Therefore, the CNN-based visual factor-aware encoder extracts preliminary visual features from infrared and visible light images. However, the local receptive field of each neuron in the convolutional layer limits their ability to capture global information, which may cause the fusion results to be limited to local features. The convolution operation lacks sufficient complex dependencies and spatial context awareness in local / global regions, and only provides simple local translation invariance properties. To alleviate these limitations, the encoder adopts a transformer-based local region information aggregation module to take advantage of the multi-head self-attention mechanism.

[0039] Step 2: Use the encoder to embed the extracted features in each block map into the corresponding block map to obtain the corresponding final feature map.

[0040] Taking visible light images as an example, the feature map generated by the initial CNN layer Split into several non-overlapping M×M blocks along the spatial dimension, and then use the local Transformer encoder to embed all feature vectors in the block To model the pairwise correlations within the corresponding local regions, thus producing local fusion and transformation features By collecting feature vectors from all blocks To construct the final feature map that encodes the visual factors of the visible light image This CNN-local Transformer structural design will highlight the richer local visual information in visible light and infrared images, which is conducive to improving the fusion performance. Among them, M = 28.

[0041] Step 3: Extract weather information features and lighting information features of the visible light image, concatenate the weather information features and lighting information features, reduce their dimensions, and normalize them to obtain an environmental factor vector.

[0042] In order to give full play to the advantages of each sensor in different working environments and achieve dynamic adaptation to actual conditions, relevant environmental factors, including weather and lighting, must be taken into account. An intuitive solution is to use weather and lighting data as additional monitoring signals to guide the fusion process. However, this supervision approach may also face significant challenges. In addition to the additional burden or increased hardware cost for collecting and annotating all-weather datasets, how to transform weather and lighting conditions into latent feature subspaces that retain their respective semantic relationships is also a difficult problem. Typical one-hot discrete encoding cannot effectively capture the semantic correlation between different environmental variables. At the same time, the continuous nature of lighting levels further complicates this task. To address this problem, this paper proposes a new simulated dataset containing images with different weather conditions and lighting levels. Using these datasets, an environment-aware encoder can be effectively pre-trained, which can reveal the latent factors affecting weather and lighting from the input source images without relying on explicit monitoring or specialized acquisition hardware.

[0043] A dataset containing various weather conditions and lighting levels was built based on the Carla autonomous driving simulation platform. Carla is an open source platform based on Python and Unreal Engine that is designed to simulate the behavior of autonomous vehicles in different scenarios, allowing developers to customize simulation parameters such as traffic flow, pedestrian behavior, and weather conditions. The weather dataset includes nine weather conditions: clear, cloudy, humid, humid and cloudy, light rain, moderate rain, moderate fog, heavy rain, and heavy fog, where each image has a unique weather discrete label.

[0044] Similarly, the present invention uses the sun altitude angle as an approximation to construct the illumination level dataset, normalizing the (-90°, 90°) range to (0, 1) as a continuous illumination level. To further enhance the diversity of the data, all-day images under three different scenarios are simulated. These two simulation datasets enable effective pre-training and evaluation of the environment-aware encoder.

[0045] In order to effectively extract factors representing discrete weather types, this task is reformulated as a classification problem solved by a Weather Awareness Module (WAM), which is based on a visual Transformer network. vi Representing a visible light image, the image is first divided into non-overlapping blocks, and then these blocks are embedded using a linear transformation. The resulting embedding vector is then fed into a local Transformer encoder to output a feature vector f for weather classification. wea This process can be expressed as f wea =WAM(I vi ). After pre-training on a simulated dataset, the module is able to produce an embedded feature vector that effectively encodes information about discrete weather categories, which allows downstream fusion tasks to leverage the learned weather-aware features.

[0046] Image histogram is a fundamental representation for understanding pixel intensity distribution, providing valuable insights into the illumination characteristics of an image. vi and design an illumination awareness module (IAM) based on a multi-layer perceptron to map the histogram to a feature embedding vector for the lighting level prediction task. This process can be expressed as f illu =IAM(hist(I vi )), where hist(·) represents the image histogram extraction operator. Similar to WAM, IAM will also be pre-trained on the constructed illumination simulation dataset to generate a feature vector f that captures the illumination information from the visible light image. illu .

[0047] Step 4: Use the environmental factor vector to adjust the final feature map of the infrared image and the final feature map of the visible light image to obtain a fused image.

[0048] According to the encoder designed above, infrared and visible light images can be decomposed into several groups of information factors that capture visual texture, infrared information, weather type, and lighting level. For image fusion tasks, these factors will be recombined or enhanced to highlight the features of interest. wea and f illu To adjust F vi and F ir The fusion process first uses the following operation to fuse f wea and f illu Environmental factors:

[0049]

[0050] in represents the cascade operation along the channel dimension, MLP env represents a multilayer perceptron network, σ is the Each value in is normalized to a Sigmoid function between (0,1). The resulting vector q env Encode comprehensive environmental information extracted from visible light images.

[0051] Then, for F vi and F ir The visual factors in are linearly transformed into query matrix, key matrix and value matrix, namely Q vi ,K vi ,V vi and Q ir ,K ir ,V ir . Using q env Q vi and Q ir The weighting is as follows:

[0052]

[0053] Where ⊙ represents the element-by-element broadcast product, and then, the query matrix modulated based on the environment information and are exchanged and utilized in the subsequent local cross-attention mechanism. Specifically, Inject infrared visual information including environmental modulation into K vi and V vi This operation enables the model to selectively focus on the informative infrared features and fuse them with the corresponding visible features, thereby enhancing the cross-modal representation. Feed into the local Transformer encoder. The local constrained cross-attention operation produces the final feature map F ir→vi , which captures the refined cross-modal representation.

[0054] Will K ir and V ir Feed into another local Transformer encoder to produce the final feature map F vi→ir This bidirectional cross-attention mechanism ensures that the model can effectively utilize the complementary information from visible light and infrared sources, thereby improving the fusion performance. Finally, these feature maps are concatenated to obtain the environmental modulation fusion feature, as shown below:

[0055]

[0056] The local nature of the criss-cross attention mechanism is particularly advantageous for visible and infrared image fusion, as it enables the model to focus on the most relevant and complementary local regions between the two modalities without the computationally expensive global attention. By restricting attention to a local neighborhood, the model is able to better capture the spatial correspondence between visible and infrared features, allowing for more effective feature fusion while maintaining computational efficiency. Using a local Transformer encoder to process criss-cross attention triplets helps reduce computational complexity, making this approach more practical and applicable in resource-constrained real-world applications.

[0057] Given the fusion feature map F fus , the final fused image I fus It will be obtained by designing a universal decoder based on CNN and combining it with a loss function. In order to ensure the quality of the final fused image, the following loss functions are considered from three different perspectives.

[0058] First, the basic generation fidelity loss function measures I fus and I ir Between, and I fus and I vi The l1 distance between them is used to preserve the pixel intensity information:

[0059]

[0060] In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I vi is the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image. Next, the goal is to minimize I fus with I ir and I vi The l1 distance between the maximum values ​​of the elements of fus Ideally contains the most salient pixels in both infrared and visible light images.

[0061]

[0062] Finally, in order to make the fused image contain more texture detail information in the source image, a texture loss is designed to guide the fused image to retain important information of the object surface or scene:

[0063]

[0064] The final objective function is a weighted combination of the above three loss functions, expressed as:

[0065]

[0066] Among them, α, β and γ are three hyperparameters.

[0067] Example 1

[0068] To pre-train the environment-aware encoder, two simulated image datasets were built using the Carla platform. These datasets are used for weather classification and lighting level prediction, respectively. The weather dataset contains nine weather categories: clear, cloudy, humid, humid and cloudy, light rain, moderate rain, moderate fog, heavy rain, and heavy fog. Each category has 1007 images with a spatial resolution of 1280×640, capturing various scenes from morning to night throughout the day. The lighting level dataset contains a total of 1083 images, covering three scenes, and the spatial resolution is also 1280×640.

[0069] For the image fusion task, two mainstream publicly available datasets are used to train the fusion network: the Multi-Spectral Road Scene (MSRS) dataset and the Multi-Scene Multi-Modal (M3FD) dataset. The MSRS dataset includes 1444 pairs of high-quality, aligned infrared and visible light images. Among them, 1083 pairs are used for training and 361 pairs are used for testing. The images in this dataset have a spatial resolution of 640×480 pixels and can capture road scenes during the day and at night. The M3FD dataset contains 4200 pairs of high-quality, aligned infrared and visible light images. 3780 pairs of this dataset are used for training and 420 pairs are used for testing. It covers a wide variety of scenes, including various environments, lighting conditions, seasons, and weather, showing diverse pixel changes.

[0070] The models will be trained in two stages, first training the weather classification model and the light intensity prediction model. For the weather classification model training, 100 epochs, a batch size of 8, a learning rate of 0.001, and a stochastic gradient descent optimizer are used. For the light intensity prediction model training, 100 epochs, a batch size of 32, a learning rate of 0.001, and an Adam optimizer are used. When training the fusion network, 100 epochs, a batch size of 4, a learning rate of 0.0001, and an Adam optimizer are used. The loss function contains three weight factors: α=5, β=9, and γ=50. All experiments were conducted on a computer equipped with an NVIDIA GeForceRTX 3060 GPU, and the proposed deep model was implemented using Pytorch.

[0071] To verify the performance of this embodiment for infrared and visible light image fusion, it is compared with seven state-of-the-art methods. In order to subjectively evaluate the all-day fusion capability, four representative image pairs are selected, namely daytime, nighttime, rainy, and foggy images. This diverse range of environmental conditions allows for a comprehensive visual demonstration of the model's fusion performance. In each image, two local areas are enlarged as close-ups for better comparison. The comparative experimental results are shown in Figure 2. Figure 2 As shown, it can be seen that the method proposed in this embodiment can better preserve texture and detail information while appropriately highlighting related target structures.

[0072] In order to quantitatively evaluate the method proposed in this embodiment, six indicators were selected, and the comparative experimental results with other seven state-of-the-art methods are shown in Table 1. The experimental results show that the present invention has significant superiority in all evaluation indicators. The highest EN indicates that the fused image generated by the proposed method contains the largest amount of information. The highest MI and Qabf indicate that the method can most effectively transfer information from the source image to the fused image, adapting to different weather and lighting conditions, whether it is day, night, sunny or rainy. In addition, the highest VIF shows that the fused image produced by the method presents high contrast and satisfactory visual quality. The best SF and AG also indicate that the fused image maximizes the preservation of detailed texture information. This is thanks to the intra-source and inter-source fusion modules, which promote the effective integration of global background information. Overall, the quantitative results on the MSRS and M3FD test sets emphasize the superior performance of the method proposed in this embodiment in terms of information content, correspondence from source to fused image, visual quality, and preservation of significant details and textures.

[0073] Table 1 Quantitative comparison results on MSRS and M3FD datasets

[0074]

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An adaptive infrared and visible light image fusion method based on environmental perception, characterized in that: include: Step 1: Use convolutional neural network to extract the features of infrared image and visible light image to obtain infrared feature map and visible light feature map respectively; and dividing the infrared feature map and the visible light feature map into a plurality of non-overlapping block maps along the spatial dimension; Step 2: Use the encoder to embed the extracted features in each block map into the corresponding block map to obtain the corresponding final feature map; Step 3: Extract weather information features and lighting information features from the visible light image, concatenate the weather information features and lighting information features, reduce their dimensions, and normalize them to obtain an environmental factor vector; Step 4: Use the environmental factor vector to adjust the final feature map of the infrared image and the final feature map of the visible light image to obtain a fused image.

2. The method for adaptive infrared and visible light image fusion based on environment perception according to claim 1, characterized in that: In step 4, the fused image is obtained by a CNN-based decoder combined with a loss function, and the loss function is: In the formula, α, β and γ are all hyperparameters. is the fidelity loss function, is the strength loss function, is the texture loss function.

3. The method for adaptive infrared and visible light image fusion based on environment perception according to claim 2, characterized in that: In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I vi is the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image.

4. The method for adaptive infrared and visible light image fusion based on environment perception according to claim 2, characterized in that: In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I vi is the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image.

5. The method for adaptive infrared and visible light image fusion based on environment perception according to claim 2, characterized in that: In the formula, I fus is the fused image; I ir is the final feature map of the infrared image; I vi is the final feature map of the visible light image; H is the height of the infrared image or the visible light image; W is the width of the infrared image or the visible light image.

Images