An infrared and visible light image fusion system based on distillation-fusion-semantic joint driving

Abstract

The application discloses an infrared and visible light image fusion system based on distillation-fusion-semantic joint driving, comprising a teacher network, a student network and a semantic segmentation network; the teacher network is used for supervising student network training and feature transmission, realizing strong feature extraction and image reconstruction capability, and guaranteeing that features can be better continued; the student network utilizes the features transmitted by the teacher network through distillation operation, and further trains to extract target and texture features of a source image and generate a fusion image; the semantic segmentation network performs semantic constraint on the generated fusion image, so as to guarantee that the student network obtains good fusion performance and semantic characteristics. The application proposes a distillation-fusion-semantic joint driving network architecture, which is used for realizing an infrared and visible light image fusion task. Meanwhile, the requirements of feature transmission, feature fusion and high-level semantics are considered.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, specifically to a network architecture based on distillation-fusion-semantic joint driving for infrared and visible light image fusion. Background Technology

[0002] Infrared and visible light image fusion, as an image enhancement technique, aims to merge two different modalities of images into a single, high-quality fused image containing rich visual information. Infrared and visible light images, as important source images, differ in contrast, texture, and brightness. Infrared images, acquired by infrared sensors, can detect the thermal radiation of targets. However, infrared images have low resolution and weak scene representation capabilities. In contrast, visible light images provide more visual detail but are more sensitive to environmental interference and easily affected by adverse factors. To compensate for these shortcomings, image fusion technology can effectively combine the two types of images, providing a more comprehensive understanding of complex environments and scenes. Therefore, it has wide practical applications in fields such as medical diagnosis, target detection and tracking, and search and rescue.

[0003] The key to infrared and visible light image fusion lies in effectively combining the two images to provide a more comprehensive and clearer picture. Over the past few decades, researchers have proposed many methods to solve the task of infrared and visible light image fusion, which can be divided into three categories: traditional image fusion systems, deep learning-based fusion systems, and fusion systems jointly trained with high-level vision tasks.

[0004] Traditional image fusion systems use image processing algorithms to calculate and fuse image information to obtain a fusion result. These methods can perform pixel-weighted fusion, fuse images in the transform domain after transformation, or fuse according to certain rules. Because these methods are not limited by resources, they meet real-time requirements. However, these methods usually require manual design and formulation of fusion rules, resulting in low flexibility and adaptability. Furthermore, the application of traditional fusion systems is often limited because they do not fully consider the feature differences between source images and cannot adapt to complex scene requirements.

[0005] To address the shortcomings of traditional fusion systems, the development of deep learning technology in recent years has offered possibilities for improving fusion performance. Deep learning-based fusion systems leverage the powerful feature extraction and reconstruction capabilities of deep neural networks to significantly alter image quality, enhance scene understanding, and solve the problem of fusion processes requiring no manual intervention. However, these methods fail to meet the demands of advanced vision tasks when implementing image fusion, neglecting the information required for these tasks.

[0006] Recently, researchers have proposed numerous fusion systems jointly trained with advanced vision tasks to preserve semantic and other information as much as possible. These methods simultaneously meet the needs of image fusion and semantic segmentation, or image fusion and object detection, further improving the visual quality of fused images through joint training. While these methods have demonstrated excellent fusion performance, several pressing challenges remain. First, convolutional neural networks extract image features from local areas, while Transformer fusion networks, leveraging self-attention, capture global dependencies but offer less insight into features from a frequency component perspective. Second, the low utilization of features and semantic information passed by the teacher and segmentation networks hinders the student network from achieving superior fusion performance and higher operational efficiency. Furthermore, feature distillation designed for infrared and visible light image fusion suffers from insufficient generalization ability, failing to adequately achieve medical image fusion. Summary of the Invention

[0007] The purpose of this invention is to propose a fusion framework that combines knowledge distillation and semantic segmentation. While ensuring feature transfer, it focuses on regions with specific semantics, prioritizes capturing salient targets and preserving texture details, promotes the comprehensive utilization of feature information and semantic information, and improves fusion performance.

[0008] To achieve the above objectives, this application proposes an infrared and visible light image fusion system based on distillation-fusion-semantic joint driving, comprising a teacher network, a student network, and a semantic segmentation network. The teacher network supervises the training and feature transmission of the student network, achieving strong feature extraction and image reconstruction capabilities, and ensuring better feature continuation. The student network utilizes the features transmitted by the teacher network through distillation to further train and extract target and texture features from the source image to generate a fused image. The semantic segmentation network imposes semantic constraints on the generated fused image to ensure that the student network achieves good fusion performance and semantic characteristics.

[0009] Furthermore, the encoder portion of the teacher network includes a modal difference block, a spectral transformation block, and an efficiency attention block;

[0010] The modal difference block is used to obtain the differences between images of different modalities, extract common features and complementary features, and integrate complementary features for images of different modalities;

[0011] The spectral transform block is used to capture texture and edge information from the spectral space. First, the feature map is mapped from the physical space to the spectral space using the fast Fourier transform. Then, the components of each frequency are determined. Finally, the inverse transform is used to convert it back to the physical space, thereby realizing the function of capturing different frequency components in the image.

[0012] The efficiency attention block is used to capture texture and edge information from physical space; this is achieved by increasing the number of heads and changing the order of operations for Q, K, and V values.

[0013] Furthermore, the decoder portion of the teacher network comprises convolutional layers and two efficient attention blocks for integrating and reconstructing features to generate a fused image.

[0014] Furthermore, the encoder portion of the student network includes an infrared target block and a visible detail block;

[0015] The infrared target block includes a Scharr filter, multiple convolutional layers, and skip connection layers; the Scharr filter is used to extract information about structure, contour, and salient regions.

[0016] The visible detail block contains a Sobel filter and a dense connection layer, which ensures the extraction of detail information while enhancing the transmission of features.

[0017] Furthermore, the skip connection layer and the dense connection layer focus on the gradient and intensity information of different modal images through filters.

[0018] Furthermore, the decoder portion of the student network consists of a skip connection layer and four convolutional layers, each with a kernel size of 3×3; except for the last convolutional layer, all other convolutional layers are activated by ReLU.

[0019] Furthermore, the semantic segmentation network obtains a semantic segmentation map through a pre-trained semantic segmentation extended residual network to determine the semantic content of the fused image.

[0020] Furthermore, the process of refining the teacher network is quantified as follows:

[0021]

[0022]

[0023]

[0024]

[0025] Among them, I r and I v These represent the input infrared and visible light images, respectively; M(·) represents the modal difference block, S(·) represents the spectral transform block, and E(·) represents the efficiency attention block; and Feature maps obtained from modal difference; and The feature map is obtained after passing through the spectral transform block and the efficiency attention block; cat(·) represents the cascade in the channel dimension, Fft This represents the result after cascading; DT(·) represents the teacher network decoder, which is used to generate the teacher network fused image. intermediate feature map and

[0026] Furthermore, the student network refinement process is quantified as follows:

[0027] F rs =conv(I r ),F vs =conv(I v )

[0028]

[0029]

[0030]

[0031]

[0032] Where conv(·) represents the convolution operation, and the feature F is obtained after convolution. rs and F vs I(·) and V(•) represent the infrared target block and the visible detail block, respectively; scharr(•) and sobel(·) are filters; and The modal characteristics are obtained after passing through two modules and a filter; and It is a feature used to skip connections; F fs The features of the student network after fusion; DS(•) represents the decoder of the student network. A fused image generated after the reconstruction of the student network; and This represents the intermediate features of the student network, used to calculate the loss with the intermediate features of the teacher network.

[0033] Furthermore, utilizing the fusion loss L fus and distillation semantic loss L dise Constraints are imposed on the distillation and fusion processes; the distillation semantic loss, used to guide distillation and semantic segmentation, is expressed as:

[0034] L dise =η(L de +L du )+μL seg

[0035] Among them, L de L du and Lxeg η and μ represent the distillation feature loss, distillation fusion loss, and feature semantic loss, respectively; η and μ are hyperparameters that balance each component.

[0036] Distillation characteristic loss L de Defined as:

[0037]

[0038] Where ||·||1 represents L1 normal form; This indicates the measurement of the feature gradient, where C refers to the number of channels in the feature map; and These represent the intermediate feature graphs of the teacher network and the student network, respectively.

[0039] Distillation fusion loss L du The definition is as follows:

[0040]

[0041] Where SSIM(·) represents the structural similarity function, which is defined as:

[0042]

[0043] Among them, I * Representative source image I r I v or fused images of teacher networks The fused image represents the student network; μ and σ represent the mean and standard deviation, respectively; C1, C2, and C3 are constants that maintain stability.

[0044] Feature semantic loss L seg The definition is as follows:

[0045]

[0046] Where n represents the number of batches; the parameter α is used to add weights to the loss of samples of different categories, while γ focuses on optimizing the loss of hard-to-distinguish samples for the sample prediction value.

[0047] Fusion loss L fus The definition is as follows:

[0048] L fus =t1L pix +t2L int +t3L grad +t4L ssim

[0049] Where t1, t2, t3, and t4 are hyperparameters that balance each loss; L pix Lint L grad and L ssim They are defined as follows:

[0050]

[0051]

[0052]

[0053]

[0054] Where ||·||1 represents L1 normal form, and max{·} represents selection by maximum element; This represents the Sobel gradient operator, used to measure the texture details of an image.

[0055] Compared with existing technologies, the technical solution adopted in this invention has the following advantages: This invention proposes a distillation-fusion-semantic joint-driven network architecture for achieving infrared and visible light image fusion tasks. It simultaneously considers the requirements of feature transfer, feature fusion, and high-level semantics. A teacher network is designed, which, by combining spectral transformation and efficient attention, captures global and local features in both the spatial and spectral domains. To improve feature utilization, intermediate features from the teacher network and the fused image are passed to the student network during distillation training, effectively supervising the student network's training. The student network employs a dual-branch mechanism, focusing simultaneously on capturing salient targets and texture details. To meet the demands of high-level visual tasks while achieving fusion, a segmentation network is used to pass semantic information to the student network, improving the fusion effect. Furthermore, to promote the comprehensive utilization of feature and semantic information, a novel distillation semantic loss function is used. This function effectively constrains the distillation and segmentation processes, thereby improving the overall fusion performance. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 This is a schematic diagram of an infrared and visible light image fusion system driven by a distillation-fusion-semantic joint approach.

[0058] Figure 2 A diagram of the teacher network structure;

[0059] Figure 3 This is a structural diagram of the encoder portion of the teacher network;

[0060] Figure 4 A diagram of the student network structure;

[0061] Figure 5 This is a structural diagram of the encoder part of the student network;

[0062] Figure 6 This is a qualitative comparison graph between our system and other advanced fusion systems on the MSRS image dataset;

[0063] Figure 7 This is a qualitative comparison graph between our system and other state-of-the-art fusion systems on the M3FD and TNO image datasets;

[0064] Figure 8 A quantitative comparison plot of our system with other state-of-the-art fusion systems on the M3FD and TNO datasets;

[0065] Figure 9 This is a qualitative comparison graph of semantic segmentation between our system and other state-of-the-art fusion systems on the MSRS dataset. Specific implementation methods

[0066] The principles of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. While preferred embodiments of this disclosure are shown in the drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0067] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment".

[0068] This embodiment proposes a fusion framework combining knowledge distillation and semantic segmentation, driven by a distillation-fusion-semantic joint approach. While ensuring feature transfer, it focuses on semantically specific regions to train a lightweight student network. During extraction, a semantic segmentation model is used for supervision, effectively understanding regions of interest and significantly improving the fusion performance of the student network. The designed teacher network can capture global and local features in both physical and spectral dimensions. The designed student network focuses on capturing salient targets and preserving texture details. A distillation-semantic loss function, composed of distilled features, distillation fusion, and semantic features, constrains the distillation training process, promoting the comprehensive utilization of feature and semantic information and improving fusion performance.

[0069] like Figure 1 As shown, an infrared and visible light image fusion system based on distillation-fusion-semantic joint driving is characterized by comprising a teacher network, a student network, and a semantic segmentation network;

[0070] The teacher network acts as a supervisor, undertaking the tasks of supervising training and feature propagation. By adaptively integrating intermediate features from both physical and spectral dimensions, the teacher network understands features from both global and frequency component perspectives, achieving excellent feature extraction capabilities. The overall structure of the teacher network is as follows: Figure 2 As shown in the figure, the encoder section includes a modal difference block, a spectral transformation block, and an efficiency attention block to capture local frequency information and long-range feature dependencies.

[0071] like Figure 3 As shown in an example embodiment, the modality difference block is used to calculate the differences between images of different modalities, extracting common and complementary features. By integrating complementary features from different modalities, feature utilization is ensured. Furthermore, the spectral transform block and the efficient attention block are responsible for fully capturing texture and edge information from the spectral and physical spaces, respectively, and addressing the problem of not being able to extract features from both global and local sources simultaneously. The spectral transform block first uses a Fast Fourier Transform to map the feature map from the physical space to the spectral space, then determines the components at each frequency, and finally uses an inverse transform to convert it back to the physical space, thus capturing different frequency components in the image. The efficient attention block significantly reduces memory and computational costs by increasing the number of heads and changing the order of operations for Q, K, and V values. The decoder part of the teacher network includes convolutional layers and two efficient attention blocks, aiming to integrate and reconstruct features to generate a fused image.

[0072] Teacher networks can produce fused images with rich texture information and excellent quality. In contrast, student networks suffer from loss of detail and low sharpness. Therefore, distillation is used to improve the fusion performance of student networks. The structure of student networks is simpler than that of teacher networks, such as... Figure 4 As shown. In the student network, the encoder section includes infrared target blocks and visible detail blocks, such as... Figure 5 The two modules shown are designed to fully extract texture and target features, achieving good feature extraction capabilities. The infrared target block includes a Scharr filter, multiple convolutional layers, and skip connection layers; the Scharr filter is used to extract information about structure, contours, and salient regions. The visible detail block contains a Sobel filter and dense connection layers, ensuring detail extraction while enhancing feature transfer. Furthermore, the skip connection layers and dense connection layers also use filters to focus on the gradient and intensity information of different modalities. The decoder part of the student network consists of skip connections and four convolutional layers, each with a kernel size of 3×3. Except for the last convolutional layer, all other convolutional layers are activated by ReLU.

[0073] After generating the fused image, a pre-trained semantic segmentation diffusing residual network is used to obtain a semantic segmentation map to determine the semantic content of the fused image. Furthermore, semantic constraints are utilized to fully capture semantic features, further enhancing the fusion effect.

[0074] In this system, the joint distillation-fusion-semantic driving requires the utilization of fusion loss L. fus and distillation semantic loss L dise Constraints are imposed on the distillation and fusion processes. Distillation characteristic loss L de This is used to calculate the difference between intermediate feature maps generated by the teacher and student networks to determine the effective transfer rate of features. The number of channels is corresponding for both intermediate feature maps. The distillation-fusion loss L... du The fusion error is determined by measuring the fused image of the teacher-student network. Semantic loss L seg Used to calculate the difference between the predicted values ​​and the ground truth of the fused image. Fusion loss L fus The differences between the fused image and the source image in terms of pixel-level, texture, salient regions, and structure are calculated based on pixel, gradient, intensity, and structural similarity, respectively.

[0075] Results Explanation

[0076] This system is first compared with ten state-of-the-art infrared and visible light image fusion systems on the MSRS dataset. Furthermore, to further demonstrate the superiority of this system, it is compared with M... 3 Image pairs were selected from the FD and TNO datasets for further validation. In the qualitative evaluation of this system, the fused images were evaluated by the human visual system, considering factors such as the objects, brightness, and scene integrity. Qualitative analysis results on the MSRS dataset are as follows: Figure 6 As shown. Figure 7 Displayed in M 3 Qualitative analysis results on the FD and TNO datasets.

[0077] Compared to other systems, the proposed distillation-fusion-semantic joint driving network effectively preserves salient object and texture information, demonstrating better scene representation capabilities. Furthermore, the generated fused image exhibits clear texture features and edge content, rich features, and better conforms to human visual perception.

[0078] To comprehensively evaluate the fusion performance of the proposed system and avoid the interference of subjective factors, this invention also utilizes quantitative indicators EN, MI, SCD, and Q. abf VIF and SSIM are used to evaluate fused images from the perspectives of information content, human visual perception, and structural similarity. M... 3Test image pairs from the FD and TNO datasets were used as test sets to complete different infrared and visible light image fusion tasks. Quantitative evaluation results are as follows: Figure 8 As shown. By Figure 8 It can be seen that the quantitative results obtained by this system are in MI and Q. abf The fusion results are optimal and second-best in terms of visual fidelity, VIF, and SSIM, indicating that the fusion result is closest to the source image in terms of visual fidelity and structural similarity, making full use of the information in the source image. The EN and SCD indices follow only slightly, still able to integrate sufficient features into the fused image.

[0079] To further explore the impact of the distillation-fusion-semantic joint driving architecture on semantic segmentation performance, this system was trained and tested on the MSRS dataset. DeeplabV3+ from MMSegmentation was used as the backbone network, and training and testing were performed according to the MSRS dataset specifications. The semantic segmentation results are as follows: Figure 9 As shown in the figure, the results demonstrate that the addition of semantic-driven processing during the distillation process allows for the fusion of images to fully capture salient features and texture information, while also effectively integrating semantic features. These acquired features, after thorough reconstruction and utilization, improve segmentation accuracy and enhance the model's ability to perceive target boundaries. Furthermore, this system uses YOLOv7 as a benchmark model, comparing the effectiveness of the proposed system using mAP@0.5. The detection results exhibit high accuracy and precision, demonstrating excellent detection performance. This system also utilizes medical images for fusion, proving that the proposed system possesses good fusion performance and strong generalization ability.

[0080] Therefore, extensive experiments on multiple datasets demonstrate that the proposed architecture possesses superior features in feature utilization, fusion performance, operational efficiency, and semantic compliance. The generated fused images effectively preserve scene details and salient objects, ensuring clear detail features. In terms of computational efficiency, the proposed system exhibits excellent operational efficiency and can be deployed to real-time fusion tasks to meet application needs. Its strong generalization ability and potential in advanced vision tasks are also evident in applications such as medical image fusion, object detection, and semantic segmentation.

[0081] The above description is merely an optional embodiment of this disclosure and is not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0082] Although the claims in this application have been formulated for specific combinations of features, it should be understood that the scope of this disclosure also includes any novel feature or any novel combination of features, whether express or implied or generalized herein, whether or not it relates to the same scheme in any of the claims currently claimed.

Claims

1. An infrared and visible light image fusion system based on distillation-fusion-semantic joint driving, characterized in that, It includes a teacher network, a student network, and a semantic segmentation network; the teacher network is used to supervise the training and feature transmission of the student network to achieve feature extraction and image reconstruction; the student network utilizes the features transmitted by the teacher network through a distillation operation to further train and extract target and texture features of the source image and generate a fused image; the semantic segmentation network applies semantic constraints to the generated fused image to ensure that the student network obtains fusion performance and semantic characteristics. The process of refining the teacher network is quantified as follows: in, and These represent the input infrared and visible light images, respectively; Represents modal difference blocks, For spectral transformation blocks, Represents efficiency attention blocks; and Feature maps obtained from modal difference; and The feature map is obtained after passing through the spectral transformation block and the efficiency attention block; This indicates cascading within the channel dimension. This represents the result of cascading. This represents the teacher network decoder, which is used to generate a teacher network fusion image. intermediate feature map and .

2. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, The encoder portion of the teacher network includes a modal difference block, a spectral transformation block, and an efficiency attention block; The modal difference block is used to obtain the differences between images of different modalities, extract common features and complementary features, and integrate complementary features for images of different modalities; The spectral transform block is used to capture texture and edge information from the spectral space. First, the feature map is mapped from the physical space to the spectral space using the fast Fourier transform. Then, the components of each frequency are determined. Finally, the inverse transform is used to convert it back to the physical space, thereby realizing the function of capturing different frequency components in the image. The efficiency attention block is used to capture texture and edge information from physical space; this is achieved by increasing the number of heads and changing the order of operations for Q, K, and V values.

3. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, The decoder portion of the teacher network consists of convolutional layers and two efficient attention blocks for integrating and reconstructing features to generate a fused image.

4. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, The encoder portion of the student network includes an infrared target block and a visible detail block; The infrared target block includes a Scharr filter, multiple convolutional layers, and skip connection layers; the Scharr filter is used to extract information about structure, contour, and salient regions. The visible detail block contains a Sobel filter and a dense connection layer, which ensures the extraction of detail information while enhancing the transmission of features.

5. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 4, characterized in that, The skip connection layer and dense connection layer focus on the gradient and intensity information of different modal images through filters.

6. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, The decoder portion of the student network consists of a skip connection layer and four convolutional layers, each with a kernel size of [size missing]. Except for the last convolutional layer, all other convolutional layers are activated by ReLU.

7. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, The semantic segmentation network obtains a semantic segmentation map through a pre-trained semantic segmentation extended residual network to determine the semantic content of the fused image.

8. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, The process of refining the student network is quantified as follows: in, This represents a convolution operation, after which features are obtained. and ; and These are respectively the infrared target block and the visible detail block; and For filters; and The modal characteristics are obtained after passing through two modules and a filter; and It is a feature used to skip connections; Characteristics of student network integration; Decoder representing the student network, A fused image generated after the reconstruction of the student network; and This represents the intermediate features of the student network, used to calculate the loss with the intermediate features of the teacher network.

9. The infrared and visible light image fusion system based on distillation-fusion-semantic joint driving as described in claim 1, characterized in that, Utilizing fusion loss and distillation semantic loss Constraints are imposed on the distillation and fusion processes; the distillation semantic loss, used to guide distillation and semantic segmentation, is expressed as: in, , and These represent the distillation feature loss, distillation fusion loss, and feature semantic loss, respectively. and It's a hyperparameter; Distillation characteristic loss Defined as: in, express Paradigm; Indicates characteristic gradient measurement, This refers to the number of channels in the feature map; , , and These represent the intermediate feature graphs of the teacher network and the student network, respectively. Distillation fusion loss The definition is as follows: in, The structural similarity function is defined as follows: in, Representative source image , or fused images of teacher networks ; A fused image representing the student network; and These represent the mean and standard deviation, respectively. , and It is a constant that maintains stability; Feature semantic loss The definition is as follows: Where n represents the batch size; parameter Its function is to add weights to the loss of samples from different categories. For the sample prediction values, the focus is on optimizing the loss of hard-to-distinguish samples; Fusion loss The definition is as follows: in, , , and To balance the hyperparameters of each loss; , , and They are defined as follows: in, express Paradigm This indicates selection based on the largest element. This represents the Sobel gradient operator, used to measure the texture details of an image.

Images