Infrared and visible light fusion method based on multi-task learning
By employing a multi-task learning approach, shared feature extraction, and adaptive feature enhancement, the problems of information inconsistency and feature degradation in low-light scenes during infrared and visible light image fusion are solved. This approach achieves high-quality fused image generation and semantic segmentation improvements, demonstrating significant application value.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2026-02-02
- Publication Date
- 2026-05-01
AI Technical Summary
Existing infrared and visible light image fusion methods suffer from inconsistencies in information distribution among different modal features, feature degradation in low-light scenes, and a lack of optimization for downstream tasks, resulting in poor performance of the fusion results under complex lighting and low-light conditions.
Employing a multi-task learning approach, this method utilizes shared feature extraction, adaptive feature enhancement, and task-driven feature allocation to generate high-quality fused images using the Restormer encoder and Clip encoder. Furthermore, by combining a semantic image interaction module and a decoding head, it achieves efficient fusion and semantic segmentation of infrared and visible light images.
Generate robust, high-quality fused images, significantly improving the performance of fusion results in downstream tasks such as semantic segmentation, and enhancing fusion effects under complex lighting and low-light conditions.
Smart Images
Figure CN121961871A_ABST
Abstract
Description
An Infrared and Visible Light Fusion Method Based on Multi-Task Learning Technical Field
[0001] This invention relates to the field of image fusion technology, and specifically to an infrared and visible light image fusion method based on multi-task learning. Background Technology
[0002] Infrared and visible light image fusion is a key technology that effectively integrates complementary information from infrared and visible light images to generate a fused image with more comprehensive visual information. This technology can simultaneously utilize the fact that infrared images are insensitive to illumination and can stably present target outlines and significant structural characteristics even at night, in smoke, and in low-light environments, as well as the rich texture details and color information possessed by visible light images under normal lighting conditions. Therefore, it has significant application value in fields such as autonomous driving and intelligent security.
[0003] Infrared images can effectively reflect the structural information and thermal radiation characteristics of targets, but due to limitations in their imaging mechanism, their ability to express texture details is relatively weak. Visible light images, on the other hand, can provide fine textures and background details, but in low-light, backlight, and complex weather conditions, the features of the target area are prone to blurring and underexposure. Therefore, obtaining fused images that combine structural clarity and rich texture through the rational design of fusion strategies has become a long-standing research focus in the field of image fusion.
[0004] Traditional image fusion methods primarily rely on transform domain modeling techniques such as wavelet decomposition and multi-scale pyramid representation. While these methods achieved some success in early applications, they struggle to simultaneously extract high-resolution details and model cross-regional information, particularly performing poorly in complex lighting conditions, target occlusion, or low-light scenarios. With the development of deep learning, fusion methods based on convolutional neural networks (CNNs) and Transformer architectures have gradually become mainstream. CNN models possess excellent local texture modeling capabilities, but their limited receptive field makes it difficult to effectively model long-range dependencies. Transformer models, through their self-attention mechanism, can capture globally consistent features, but they suffer from a large number of network parameters and high training and inference costs. Against this backdrop, existing research has attempted to combine the advantages of CNNs and Transformers, balancing feature representation capabilities with computational efficiency.
[0005] Despite existing methods improving the quality of fused images, the following key issues remain to be addressed: 1. Inconsistent information distribution among different modal features leads to bias in the model's extraction of structural and detail information, making it difficult to balance low-frequency global structure and high-frequency local texture; 2. Visible light image features degrade significantly in low-light scenes, making it difficult for fusion models to generate high-quality fusion results; 3. Most studies only improve the visual perception quality of fused images, lacking collaborative optimization for downstream tasks such as object detection and semantic segmentation, resulting in limited performance of fusion results in practical tasks. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides an infrared and visible light fusion method based on multi-task learning for the fusion and segmentation of infrared and visible light images. By using shared feature extraction, adaptive feature enhancement, and task-driven feature allocation, it achieves high-quality fused image generation and effectively improves the performance of the fusion results in downstream tasks such as semantic segmentation, thereby obtaining infrared and visible light fusion results that are both robust and have practical application value.
[0007] An infrared and visible light image fusion method based on multi-task learning includes the following steps: Step 1, input a visible light image and a single-channel infrared image. The visible light image is converted to a single-channel visible light image by performing color space conversion. Step 2: Select a deep learning framework and establish a multi-task fusion network structure for infrared and visible light images; Step 3: Select an optimization function and set hyperparameters including the number of iterations; , Decline text prompt and task text prompts Input the data into the network for training, and save the model for testing.
[0008] As a further improvement of the present invention, in step 1, the image color conversion process includes: converting the visible light image from RGB space to YCbCr space; wherein, the conversion formula for the visible light image is: After the conversion, the Y channel in the YCbCr space is separated to obtain a single-channel visible light image. .
[0009] As a further improvement of the present invention, step 2 specifically includes: step 2.1, placing... and The features are input in parallel to the Restormer encoder module, and the features encoded by Restormer are used as task-shared features. The Restormer encoder module consists of several levels of Transformer blocks and a self-enhancing feedforward network based on channel normalization connected sequentially. The Transformer blocks provide multi-head attention interaction across spatial regions, enabling the two input modalities to form infrared and visible light encoded features of the same scale and multiple levels during encoding output. Step 2.2: The infrared and visible light encoded features obtained in step 2.1 are input into the cross-modal fusion module to obtain unified fused features. The cross-modal fusion module includes two attention branches with the same structure. The mutual diffusion operation of the two attention matrices realizes the migration of salient target features to texture detail regions, and the features after diffusion are concatenated and output as unified fused features through the fusion projection layer. Step 2.3: The degenerate text prompts are processed... Task text prompts The input is fed into the Clip encoder to obtain the encoded text vector; the encoded text vector and the unified fusion feature are then input into the semantic image interaction module; the semantic image interaction module consists of a degradation modulation unit and a task attention interaction unit. The degradation modulation unit uses a feature channel affine mapping to generate modulation parameters for enhancing brightness and edge details. The task attention interaction unit uses image features as query Q and task text as key-value pairs KV to generate dual-path task-specific feature outputs, one for fused image reconstruction and the other for semantic segmentation prediction; step 2.4: the fused task-specific features obtained in step 2.3 are input into the fusion decoding head to generate a high-quality fused image; the segmentation task-specific features are input into the semantic segmentation decoding head to obtain pixel-level classification results; the fusion decoding head consists of a Transformer reconstruction module, and the semantic segmentation decoding head includes a three-level structure of two-level convolutional extraction layers and a classification mapping layer.
[0010] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention achieves high-quality fused image generation through shared feature extraction, adaptive feature enhancement, and task-driven feature allocation, and effectively improves the performance of the fusion results in downstream tasks such as semantic segmentation, thereby obtaining infrared and visible light fusion results that are both robust and have practical application value; it has important significance and application value in the field of infrared and visible light fusion, and proposes new ideas and solutions for infrared and visible light fusion. Attached Figure Description
[0011] Figure 1 is a flowchart of the multi-task infrared and visible light fusion method of the present invention; Figure 2 is an infrared image of the publicly available training set used in the present invention; Figure 3 is a visible light image of the publicly available training set used in the present invention; Figure 4 is a semantic segmentation label of the publicly available training set used in the present invention; Figure 5 is an infrared image of the publicly available test set used in the present invention; Figure 6 is a visible light image of the publicly available test set used in the present invention; Figure 7 is a semantic segmentation label of the publicly available test set used in the present invention; Figure 8 is a fusion test result image of the present invention; Figure 9 is a segmentation test result image of the present invention. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0013] The invention will now be described in further detail with reference to the accompanying drawings: As shown in Figure 1, the invention provides a method for infrared and visible light image fusion based on multi-task learning. It uses the publicly available infrared and visible light fusion datasets EMS and MSRS, selects PyTorch as the deep learning framework, and constructs a multi-task learning network for image fusion and semantic segmentation. Specifically, it includes: Step 1, reading the single-channel infrared image to be trained, denoted as... As shown in Figure 2; read the RGB visible light image to be tested, as shown in Figure 3. Then convert the RGB visible light image to the YCbCr color space, and extract its Y channel, denoted as... The conversion formula is as follows: Step 2: Select PyTorch as the deep learning framework and build a multi-task learning network; specifically, this includes: Step 2.1, ... and The features are input in parallel to the Restormer encoder module, and the features encoded by Restormer are used as task-shared features. The Restormer encoder module consists of several levels of Transformer blocks and a self-enhancing feedforward network based on channel normalization connected sequentially. The Transformer blocks provide multi-head attention interaction across spatial regions, enabling the two input modalities to form infrared and visible light encoded features of the same scale and multiple levels during encoding output. Step 2.2: The infrared and visible light encoded features obtained in step 2.1 are input into the cross-modal fusion module to obtain unified fused features. The cross-modal fusion module includes two attention branches with the same structure. The mutual diffusion operation of the two attention matrices realizes the migration of salient target features to texture detail regions, and the features after diffusion are concatenated and output as unified fused features through the fusion projection layer. Step 2.3: The degenerate text prompts are processed... Task text prompts The input is fed into the Clip encoder to obtain the encoded text vector. The encoded text vector and the unified fusion feature are then fed into the semantic image interaction module. The semantic image interaction module consists of a degradation modulation unit and a task attention interaction unit. The degradation modulation unit generates modulation parameters for enhancing brightness and edge details using feature channel affine mapping. The task attention interaction unit generates dual-path task-specific feature outputs using image features as query Q and task text as key-value pairs KV. One path is used for fused image reconstruction, and the other path is used for semantic segmentation prediction. Step 2.4: The fused task-specific features obtained in step 2.3 are input into the fusion decoding head to generate a high-quality fused image. The segmentation task-specific features are input into the semantic segmentation decoding head to obtain pixel-level classification results. The fusion decoding head consists of a Transformer reconstruction module to generate a high-quality fused image. The semantic segmentation decoding head includes a three-level structure with two convolutional extraction layers and a classification mapping layer to obtain pixel-level classification results.
[0014] Step 3: Select the Adam optimization function and set hyperparameters such as the number of iterations. , , , The input network is used for training, where the output image is fused with the original image. , A fusion loss function is constructed, and the cross-entropy loss is built by combining the segmented output image and the semantic segmentation labels, as shown in Figure 4. The trained network model is obtained and saved for testing; Step 4: Read the single-channel infrared image to be tested, denoted as... As shown in Figure 5. Read the RGB visible light image to be tested, as shown in Figure 6, and convert it to the YCbCr color space. Take its Y channel, denoted as... Step 5: , , , Inputting the trained network model yields fusion test result images and segmentation test result images, as shown in Figures 8 and 9.
[0015] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for fusing infrared and visible light images based on multi-task learning, characterized in that, Includes the following steps: Step 1: Input a visible light image and a single-channel infrared image. The visible light image is converted to a single-channel visible light image by performing color space conversion. Step 2: Select a deep learning framework and establish a multi-task fusion network structure for infrared and visible light images; Step 3: Select an optimization function and set hyperparameters including the number of iterations; 、 Decline text prompt and task text prompts Input the data into the network for training, and save the model for testing.
2. The image fusion method as described in claim 1, characterized in that, In step 1, the image color conversion process includes: converting the visible light image from RGB space to YCbCr space; wherein, the conversion formula for the visible light image is: After the conversion, the Y channel in the YCbCr space is separated to obtain a single-channel visible light image. 。 3. The image fusion method as described in claim 1, characterized in that, Step 2 specifically includes: Step 2.1, placing and The features are input in parallel to the Restormer encoder module, and the features encoded by Restormer are used as task-shared features. The Restormer encoder module consists of several levels of Transformer blocks and a self-enhancing feedforward network based on channel normalization connected sequentially. The Transformer blocks provide multi-head attention interaction across spatial regions, enabling the two input modalities to form infrared and visible light encoded features of the same scale and multiple levels during encoding output. Step 2.2: The infrared and visible light encoded features obtained in step 2.1 are input into the cross-modal fusion module to obtain unified fused features. The cross-modal fusion module includes two attention branches with the same structure. The mutual diffusion operation of the two attention matrices realizes the migration of salient target features to texture detail regions, and the features after diffusion are concatenated and output as unified fused features through the fusion projection layer. Step 2.3: The degenerate text prompts are processed... Task text prompts The input is fed into the Clip encoder to obtain the encoded text vector; the encoded text vector and the unified fusion feature are then input into the semantic image interaction module; the semantic image interaction module consists of a degradation modulation unit and a task attention interaction unit. The degradation modulation unit uses a feature channel affine mapping to generate modulation parameters for enhancing brightness and edge details. The task attention interaction unit uses image features as query Q and task text as key-value pairs KV to generate dual-path task-specific feature outputs, one for fused image reconstruction and the other for semantic segmentation prediction; step 2.4: the fused task-specific features obtained in step 2.3 are input into the fusion decoding head to generate a high-quality fused image; the segmentation task-specific features are input into the semantic segmentation decoding head to obtain pixel-level classification results; the fusion decoding head consists of a Transformer reconstruction module, and the semantic segmentation decoding head includes a three-level structure of two-level convolutional extraction layers and a classification mapping layer.