Infrared and visible light image fusion network

Abstract

The invention discloses an infrared and visible light image fusion network, and relates to the field of image fusion. The system comprises a double-branch feature coding unit, a fusion unit and an image reconstruction unit, the double-branch feature coding unit comprises a multi-scale edge gradient module, a significant dense residual module and a spatial bias module, and infrared and visible light images pass through the multi-scale edge gradient module to obtain enhanced texture features. The infrared image is processed by a significant dense residual module to obtain enhanced significant features, global information and local information of the enhanced texture features and the strong significant features are fused by a spatial bias module, the features processed by a fusion unit are input into an image reconstruction unit, the fused feature information is reconstructed, and the reconstructed feature information is output to the image reconstruction unit. And outputting a final fused image. According to the method, the clarity of image edges and target details can be better maintained, and higher contrast and visual clarity are provided.

Description

Technical Field

[0001] The present invention relates to the field of image fusion, and in particular to an infrared and visible light image fusion network. Background Art

[0002] Images and visual information play a vital role in many fields, but images in a single modality often cannot provide comprehensive scene information. For example, visible light images contain rich details and conform to human visual habits, which are easy to understand and analyze intuitively, but the image quality is easily affected by lighting conditions, weather conditions and background complexity. In low-light environments or severe weather conditions, the image quality will significantly decrease, affecting the effective acquisition of information. In contrast, infrared images have the advantages of being unaffected by light and weather changes, and can highlight targets based on temperature differences. However, infrared images usually lack color information, and the background texture features are not obvious, which is not suitable for specific scenes. The fusion of infrared images and visible light images aims to integrate the information of these two different modal images to make up for the limitations of insufficient representation of single modal information. Through fusion, not only can the contrast advantage of infrared images be retained to enhance the prominence of targets, but also the high-resolution characteristics of visible light images can be combined to ensure the clarity and detail presentation of images. This fusion technology has been widely used in many fields such as intelligent traffic monitoring, drone disaster detection, and security monitoring, providing strong support for improving the performance and reliability of the system.

[0003] Traditional image fusion methods generally use traditional algorithms based on sparse representation (SR), low rank representation (LRR), and multi-scale transformation to achieve the fusion of infrared and visible light images. With the deepening of research on visual saliency, more and more image fusion methods based on visual saliency have emerged. Tang et al. combined the needs of high-level visual tasks (object detection, semantic segmentation, etc.) and proposed a high-level visual task-driven image fusion framework (A semantic-aware real-time infrared and visible image fusion network, SeAFusion) to integrate as much semantic information as possible to improve the performance of high-level visual tasks on fused images. Although SeAFusion takes into account the needs of high-level visual tasks in the process of modeling fusion problems, as a preliminary attempt, it only considers improvements in the loss function.

[0004] Most of the existing image fusion models do not consider the impact of specific category semantic information on the fusion results. They use inappropriate fusion rules when fusing deep features and ignore the complex and diverse feature information of infrared and visible light images, resulting in the inability to achieve high-quality image fusion in all weather conditions, insufficient processing of significant information in the fusion results, and lack of reconstruction of semantic information. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the present invention provides an infrared and visible light image fusion system based on dual-branch feature encoding and category weight guidance, which can better maintain the clarity of image edges and target details and provide higher contrast and visual clarity.

[0006] The invention provides an infrared and visible light image fusion network, comprising a dual-branch feature encoding unit, a fusion unit and an image reconstruction unit, wherein the dual-branch feature encoding unit comprises a multiscale edge gradient block (MEGB), a significantly dense residuals module (SDRB) and a spatial bias block (SBM), the infrared and visible light images obtain enhanced texture features through the multiscale edge gradient module, the infrared images obtain enhanced significant features through the significantly dense residual module, the enhanced texture features and the strong significant features are fused with global information and local information through the spatial bias module, the features processed by the fusion unit are input into the image reconstruction unit, the fused feature information is reconstructed, and a final fused image is output.

[0007] Furthermore, the significant dense residual module includes a dense network module, a residual module and a channel attention module, and the infrared image is convolved to obtain features ,feature Input into the dense network module, the features are reused to obtain the features , the characteristics Get features through channel attention module .

[0008] Furthermore, the feature Get features through channel attention module The methods include:

[0009] feature After the convolution kernel, global average pooling is performed to obtain the global feature vector; the importance of each channel is learned through the fully connected layer; use The activation function obtains the weights, assigns higher weights to features with higher contrast, and multiplies the weights with the features obtained by the jump connection of the original input to obtain the infrared feature V containing significant information. C ,for:

[0010] ;

[0011] Add the salient features to the residual branch features to obtain the salient features of the final output ,for: ;

[0012] Among them, Conv() represents the convolution operation, represents global average pooling, represents the fully connected layer, represents the activation function, is the salient feature output by the salient dense residual module.

[0013] Furthermore, the working process of the spatial offset module includes:

[0014] Salient features at the output of a significant dense residual module Adding the spatial offset to , we get:

[0015] ;

[0016] in, represents the spatial offset, and and Represent batch normalization and nonlinear activation layers respectively;

[0017] The texture features output by the multi-scale edge gradient module and the salient features After progressive fusion, the fusion features are obtained ,for:

[0018] ;

[0019] Among them, Conv() represents the convolution operation, C() represents the cascade in the channel dimension, Represents the output of salient features after adding the spatial bias term.

[0020] Furthermore, the infrared and visible light image fusion network includes a category weight allocation unit, which includes a SegFormer module, a pooling module and a linear module. The infrared and visible light images obtain category weights of the two images through the SegFormer module, the pooling module and the linear module. The category weights are input into the fusion unit to guide image fusion.

[0021] Furthermore, the class activation weight obtained by the class weight allocation unit is for:

[0022] ;

[0023] in, is the classification function, is the input image of class c. c∈{ir, vi}, respectively represents infrared image and visible light image, express The probability of belonging to class c is:

[0024] ;

[0025] Among them, GAP(·) represents global average pooling, Represents the softmax function.

[0026] Furthermore, the fusion rule of the fusion unit is:

[0027] ;

[0028] Among them, ⊗ represents the channel weighted operation; and is the feature map and class activation weight obtained by the class weight allocation unit; is the normalization function.

[0029] Compared with the prior art, the present invention has the following technical effects:

[0030] The MEGB module of the present invention helps to extract and retain the texture gradient of the entire network, and enhances the ability of the fused image to describe the global scene information; on the other hand, the SDRB module extracts the salient targets of the infrared image and generates a salient map under the guidance of the salient loss. Finally, the SBM module is used to compensate the salient features of the tributaries to the mainstream to complete the information fusion. The qualitative and quantitative aspects of the present invention were compared with the existing methods to prove the effectiveness of the present invention. Fusion experiments in different scenarios also show that the full fusion of the semantic information of infrared and visible light images in different scenarios can be effectively achieved. The fusion result of the proposed method retains clearer edge and target information, and has better contrast and clarity

[0031] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a structural principle diagram of an infrared and visible light image fusion network according to a specific embodiment of the present invention;

[0033] Figure 2 is a structural principle diagram of a multi-scale edge gradient module according to a specific embodiment of the present invention;

[0034] Figure 3 is a structural principle diagram of a significant dense residual module according to a specific embodiment of the present invention;

[0035] Figure 4 is a structural principle diagram of a space offset module according to a specific embodiment of the present invention;

[0036] Figure 5 is a visual quality comparison result based on the MSRS data set according to a specific embodiment of the present invention, wherein: Figure 5 a is a visible light image; Figure 5 b is the infrared image; Figure 5 c is the image obtained using the DenseFuse model; Figure 5 d is the image obtained using the RFN-Nest model; Figure 5 e is the image obtained using the FusionGAN model; Figure 5 f is the image obtained using the SDNet model; Figure 5 g is the image obtained using the U2Fusion model; Figure 5 h is the image obtained using the SeAFusion model; Figure 5 i is the image obtained using the PIAFusion model; Figure 5 j is an image obtained by using the image fusion network of the present invention;

[0037] Figure 6 is a visual quality comparison result based on the TNO data set according to a specific embodiment of the present invention, wherein: Figure 6 a is a visible light image; Figure 6 b is the infrared image; Figure 6 c is the image obtained using the DenseFuse model; Figure 6 d is the image obtained using the RFN-Nest model; Figure 6 e is the image obtained using the FusionGAN model; Figure 6 f is the image obtained using the SDNet model; Figure 6 g is the image obtained using the U2Fusion model; Figure 6 h is the image obtained using the SeAFusion model; Figure 6 i is the image obtained using the PIAFusion model; Figure 6 j is the image obtained by using the image fusion network of the present invention

[0038] Figure 7 is a visual quality comparison result of the segmentation result of a specific embodiment of the present invention;

[0039] Figure 8 is a visual quality comparison result of an important loss function and module ablation study of a specific embodiment of the present invention, wherein, Figure 8 a represents visible light image; Figure 8 b represents infrared image, Figure 8 c represents the result of removing the CWA-Net loss, Figure 8 d represents the result of removing the significant loss, Figure 8 e represents the result of removing content loss, Figure 8 f represents the result of removing the MEGB module, Figure 8 g represents the result of removing the SDRB module; Figure 8 h represents the result of removing the SBM module; Figure 8 i is the result of the present invention. DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] like Figure 1 As shown, in a specific embodiment, an infrared and visible light image fusion network is provided, including a dual-branch feature encoding unit, a category weight allocation unit, a fusion unit and an image reconstruction unit. The dual-branch feature encoding unit can fully integrate the complementary features and consistent features of different modalities; the class activation weights are obtained through the category weight allocation network to guide the fusion; finally, the fused feature information is reconstructed through the image reconstruction network to output the final fused image.

[0042] The dual-branch feature encoding unit includes a multiscale edge gradient block (MEGB), a significantly dense residuals module (SDRB) and a spatial bias block (SBM). The infrared and visible light images are subjected to the multiscale edge gradient module to obtain enhanced texture features. The infrared image is subjected to the significantly dense residual module to obtain enhanced significant features. The enhanced texture features and the strong significant features are subjected to the spatial bias module to fuse global information and local information.

[0043] like Figure 2 As shown in the figure, the multi-scale edge gradient module consists of a multi-scale branch and a residual gradient branch. Convolution blocks with different sizes of convolution kernels are used to extract spatial features of different scales, enhance the receptive field, and better extract the texture detail information of infrared and visible light. At the same time, the residual gradient is combined with the Sobel operator to further retain the edge information of the feature, which can retain more edge and texture information. Finally, the multi-scale branch output is combined with the residual gradient branch output to complete the texture detail enhancement.

[0044] In the multi-scale branch, infrared images and visible light images Add in the channel dimension to get the fusion feature , fusion features Input to the multi-scale enhancement module (MEGB), the output features of MEGB for:

[0045] (1)

[0046] in Represents the fusion feature obtained by adding the infrared and visible light images in the channel dimension, Represents features extracted at different scales, Conv() represents convolution operation, and C() represents cascade on channel dimension. represents the Sobel operator, Represents the sum of elements.

[0047] like Figure 3 As shown, the significant dense residual module includes a dense network module, a residual module and a channel attention module. The infrared image is convolved to obtain features ,feature Input into the dense network module, the features are reused to obtain the features , the characteristics After convolution with a 3×3 convolution kernel, global average pooling is performed to obtain the global feature vector; the importance of each channel is learned through a fully connected layer, and the global feature vector is obtained by using The activation function obtains the weights and assigns higher weights to features with higher contrast. The weights are multiplied by the features obtained by the jump connection of the original input to obtain the infrared feature V containing significant information. C , the above process is specifically as follows:

[0048] (2)

[0049] (3)

[0050] By adding the salient features to the residual branch features, contrast enhancement is achieved, thereby highlighting the salient targets and obtaining the salient features of the final output. ,for:

[0051] (4)

[0052] Among them, Conv() represents the convolution operation, represents global average pooling, represents the fully connected layer, represents the activation function, is the salient feature output by the salient dense residual module.

[0053] like Figure 4 As shown, the working process of the spatial offset module includes:

[0054] They are the semantic features of MEGB and the salient features of SDRB, which are integrated in a progressive manner. Since the simple convolutional layer cannot learn long-range dependencies due to its limited receptive field, a spatial bias module is added to TBFEN to learn the relationship between different distant objects to enhance the semantic information of the image. This module is lightweight and faster to calculate than the self-attention mechanism. For the salient features of SDRB, the spatial bias term B can be expressed as:

[0055] (5)

[0056] in, represents the spatial offset, and and Represent batch normalization and nonlinear activation layers respectively;

[0057] The semantic features of MEGB are represented by the grayscale distribution of pixels and their surrounding spatial domains, reflecting local information. In order to capture both local and global information, the spatial bias features are concatenated with the semantic features in the channel dimension; in order to aggregate global knowledge, a 1×1 convolution is applied in the channel dimension to effectively compress and fuse the information in the feature map. Finally, the semantic features With notable features Splicing is performed to complete the process of progressive fusion, which can be expressed as:

[0058] (6)

[0059] in, represents the fusion features after progressive fusion, Conv() represents the convolution operation, and C() represents the cascade in the channel dimension. represents the output of the salient features after adding the spatial bias term, is the characteristic output of the MEGB module, It is the feature output of the SDRB module.

[0060] like Figure 5 As shown in the figure, the visual quality comparison results based on the MSRS dataset are selected, among which 7 typical fusion models such as DenseFuse, RFN-Nest, FusionGAN, SDNet, U2Fusion, SeAFusion, and PIAfusion are compared with the fusion model in this paper. In the figure, a represents a visible light image; b represents an infrared image; c represents an image obtained by a DenseFuse (A FusionApproach to Infrared and Visible Images) model; d represents an image obtained by a RFN-Nest (An end-to-end residual fusion network for infrared and visible images) model; e represents an image obtained by a FusionGAN (A generative adversarial network for infrared and visible image fusion) model; f represents an image obtained by a SDNet: (A versatile squeeze-and-decomposition network for real-time image fusion) model; g represents an image obtained by a U2Fusion (A unified unsupervised image fusion network) model; h represents an image obtained by a SeAFusion (Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network) model; i represents an image obtained by a PIAFusion (A progressive infrared and visible image fusion network based on illumination aware) model; and j represents an image obtained by the method used in the present invention.

[0061] like Figure 6As shown in the figure, the visual quality comparison results based on the TNO dataset are selected, among which 7 typical fusion models such as DenseFuse, RFN-Nest, FusionGAN, SDNet, U2Fusion, SeAFusion, and PIAfusion are compared with the fusion model of this paper. In the figure, a represents a visible light image; b represents an infrared image; c represents an image obtained by the DenseFuse model; d represents an image obtained by the RFN-Nest model; e represents an image obtained by the FusionGAN model; f represents an image obtained by the SDNet model; g represents an image obtained by the U2Fusion model; h represents an image obtained by the SeAFusion model; i represents an image obtained by the PIAFusion model; and j represents the method used in the present invention.

[0062] like Figure 7 As shown, the visual quality comparison results of the segmentation results of the semantic segmentation experiment, wherein the upper part of the figure represents VIS representing visible light image; IR represents infrared image; Ours represents the segmentation result image, and Ground Truth represents the true value; for the image in the first row and fourth column, the present invention uses 5 categories for comparative analysis, Background represents background, Car represents car, Person represents person, Bike represents bicycle, and Curve represents curve.

[0063] like Figure 8 As shown, the visual quality comparison results of the important loss function and module ablation study, in the figure, a represents a visible light image; b represents an infrared image, c represents the result of removing the CWA-Net loss, d represents the result of removing the saliency loss, e represents the result of removing the content loss, f represents the result of removing the MEGB module, g represents the result of removing the SDRB module; h represents the result of removing the SBM module; i is the result of the present invention.

[0064] like Figure 1 As shown, the category weight allocation unit includes a SegFormer module, a pooling module and a linear module. The infrared and visible light images obtain category weights of the two images through the SegFormer module, the pooling module and the linear module. The category weights are input into the fusion unit to guide image fusion.

[0065] Each feature map is reduced to a scalar through global average pooling, and then the linear layer learns to map these scalars to each category. In order to eliminate human intervention in the fusion process, the feature fusion strategy guided by the category-specific weights does not need to be manually designed, and the fusion weights are obtained by deep learning. The goal of the category weight allocation network is to distinguish between infrared images and visible light images, and the class activation weights can be expressed as:

[0066] (7)

[0067] in is the classification function, is the input image of class c. c∈{ir, vi} represents infrared image and visible light image respectively. express The probability of belonging to class c. Represents the weight of the last linear layer in the distribution network. It is a 1×1×M vector generated during the training phase and is also called class activation weight according to CAM theory.

[18] The CAM algorithm is designed to explore the interpretability in machine learning and aims to help humans understand the internal working mechanism of the black box model. The proposed fusion rule Γ(·) is shown in (8):

[0068] Feature map obtained by the semantic segmentation module and class activation weights ,

[0069] (8)

[0070] Where ⊗ represents the channel weighted operation, and Feature maps and class activation weights obtained from CWA-Net. is the normalization function, which is defined as shown in formula (10):

[0071] (9)

[0072] Normalization by nonlinear function By multiplying the adjustment factor / , the mean of the original weights is corrected to δ, which reduces the brightness reduction of the fusion result. The channel weighting operation in equation (8) requires the feature map and class activation weights Strict correspondence between them is established. represents the output features of the last MLP, and Secondly, in the output stage, a CAM mechanism is established to ensure and And A global average pooling operation (GAP) is performed and reduced to a 1 × 1 ×M vector. Then, a linear layer with a softmax activation function is used at the end of the network. The operation of the GAP and linear layers is shown in Equation (11):

[0073] (10)

[0074] Where GAP(·) represents global average pooling, represents the softmax function. With the above settings, the linear weights can directly adjust the contribution of channel m to the classification result. Therefore, after the training phase, a larger This shows that channel m plays an important role in the distribution network. In other words, It can reflect the classification basis of the distribution network and reveal the richness of the information contained in each channel. For example, if the features of channel m carry important features of the visible image (such as texture details), then the corresponding weight It will be larger. and In channels with common characteristics, due to and With similar values, the fusion rule can avoid information loss on either side. The class activation weights are used in the fusion rule to achieve flexible information selection and enhancement at the deep feature level.

[0075] In order to ensure the accuracy of category weight assignment, the cross entropy loss function is used to guide and optimize the learning process of the network, as shown in the following formula:

[0076] (11)

[0077] in, Represents the label of the input image. represents the output probability of the class assignment network. Represents a softmax function that normalizes the output probability to [0,1].

[0078] The MEGB branch hopes that the fused image will retain rich texture details while maintaining the optimal intensity distribution, so content loss is introduced. The content loss includes two parts: intensity loss and texture loss. The intensity loss is used to measure the intensity distribution of the fused image and the source image at the pixel level, and the texture loss is used to make the texture details clearer. The gradient of the fused image is close to the maximum gradient of the visible light and infrared images. Specifically, as shown in formula (12):

[0079] (12)

[0080] Where H and W are the height and width of the input image, respectively. Used to achieve a balance between intensity loss and texture loss. is the Sobel gradient operator, which is used to measure the gradient texture of the image;.

[0081] The purpose of the CWA-Net network in this embodiment is to guide fusion, optimize the model's ability to identify salient areas in the image, and adapt to the problem of class imbalance. Therefore, in order to retain the salient objects in the fused image, the target mask is used to construct the intermediate salient loss, as shown in formula (13):

[0082] (13)

[0083] in is the target mask, is the channel average .

[0084] Finally, the loss function of the deep learning model combines the category weight loss, content loss and saliency loss, and optimizes the fusion effect of infrared and visible light images through weighted λ combination. The total loss is shown in formula (14):

[0085] (14)

[0086] The ideal texture details with salient objects are obtained under the guidance of cross entropy loss, content loss and saliency loss, and the meaningful information in the source image is integrated.

[0087] The reconstruction unit consists of four convolutional layers, such as Figure 1 As shown in the figure. The task of these convolutional layers is to fully integrate common and complementary information to generate the final fused image. Except for the last layer which uses a kernel size of 1x1, all other layers use a 3x3 convolution kernel. During the image reconstruction process, the number of channels of the feature map gradually decreases, and the information can be gradually refined. The last convolutional layer uses the Tanh function, and the first three convolutional layers use Leaky ReLU. In order to ensure that information is effectively transmitted throughout the network and reduce information loss, the padding of all layers except the first and last layers is set to 1, avoiding the use of downsampling, thereby ensuring that the output fused image has the same size as the source image.

[0088] In order to further illustrate the effect of the present invention, the effect of the present invention is verified by comparative experiments below.

[0089] In this embodiment, TNO and MSRS datasets are used for training. The MSRS dataset contains 1569 pairs of infrared and visible light images after high-quality registration. The TNO multi-band image set provides night images of different military and surveillance scenes, showing different objects and targets in different backgrounds. The training set selects 45 pairs of images of different scenes from TNO, and 416 daytime scene images and 354 nighttime scene images from the MSRS dataset, all of which are 480×640 in size. The test set is selected from the public datasets TNO and MSRS for infrared and visible light image fusion, and 42 pairs, 20 pairs, 361 pairs and 300 pairs of images are selected for algorithm comparison experiments to verify the effectiveness of the method proposed in this embodiment. Seven typical fusion models, including DenseFuse, RFN-Nest, FusionGAN, SDNet, U2Fusion, SeAFusion, and PIAfusion, are selected for comparison with the fusion model of this embodiment, and comparative experimental analysis is performed using 6 subjective and objective evaluation indicators.

[0090] All parameters in the training are set as follows: The training cycle is set to: epoch = 10. Hyperparameter settings: 𝜆1 = 3, 𝜆2 = 7, 𝜆3 = 50. In addition, the algorithm uses the Adam optimizer with a bath size of 8. The initial learning rate is 10 -4 The experiment of this embodiment was conducted on a GeForce RTX 2080Ti 16GB using PyTorch as the deep learning framework. The network model was trained under the Pytorch framework.

[0091] In order to verify the effectiveness of the method in this embodiment, tests were conducted on the public datasets TNO and MARS. Mutual Information (MI), visual information fidelity (VIF), average gradient (AG), sum of correlation differences (SCD), average gradient (AG) and edge information-based indicators (Q AB / F ) six commonly used objective evaluation indicators to compare and analyze the quality of fused images. Among them, EN, MI, SCD, SF and Q abf It is an indicator for evaluating spatial information, VIF is an indicator for visual perception, and all six indicators are positive indicators, that is, the higher the indicator, the better the fusion performance.

[0092] The comparison of indicators of different methods is shown in Table 1 and Table 2. The best value of AG shows that the fusion method of this embodiment has richer contrast information and richer texture details; the best value of SF shows that the gradient distribution of the fused image reveals the details and texture information of the fused image; the best value of EN shows that the method of this embodiment retains sufficient edge information; the best value of SCD shows that the fusion result contains more real information; VIF is optimal or suboptimal on the two data sets, indicating that the method of this embodiment has richer contrast information and generates a fused image that is more in line with the human visual system; Q abf The best value of indicates that the source image transfers more edge information to the fused image. In addition, the six indicators are optimal or suboptimal on the two datasets, which shows that the method of this embodiment has superior generalization performance and can be applied to different types of datasets. In summary, the method of this embodiment is able to mine effective information in class imbalance and occlusion scenes, and integrate the information into the fused image with the help of the channel space attention module. Therefore, the method of this embodiment has greater advantages than other methods in obtaining high-quality fused images.

[0093] Table 1 Quantitative evaluation results of different methods on the TNO dataset (Bold: best)

[0094]

[0095] Table 2 Quantitative evaluation results of different methods on the MSRS dataset (Bold: best)

[0096]

[0097] The visualization results of 8 pairs of images in the 2 datasets are as follows: Figure 5-6 The regions with larger differences are highlighted with red and green boxes, and the enlarged images of the red boxes are located in the lower right or lower left corners.

[0098] like Figure 5 In the daytime scene shown, DenseFuse and RFN-Nest weakened the infrared target, FusonGAN caused the problem of blurred edge texture, while SDNet and SeAFusion weakened the background texture details of the image, as shown in the green box; only PIAFusion and the method of this embodiment can integrate effective information.

[0099] In such Figure 6In the night scene shown, the visible light image contains only a small amount of texture information, while the infrared image contains background texture detail information in addition to the prominent target. Many methods pay too much attention to the information of a certain modality image, making it difficult to obtain good results in different scenes. Among them, the infrared targets in DenseFuse, RFN-Nest, U2Fusion, and SeAFusion are weakened, and the fused images of FusonGAN and SDNet are more inclined to infrared images, resulting in blurred background information. By fully extracting the details contained in the texture mainstream image and using the salient tributary to supplement the weakened salient targets, semantic information in complex scenes can be effectively extracted.

[0100] It can be seen that this embodiment not only has comprehensive scene information, but also retains rich contrast information and texture details of the target area.

[0101] The semantic segmentation algorithm is trained on the source image and the fused image respectively. 1000 images are selected as the training set, and the segmentation performance of different models on 360 images is tested. The qualitative and quantitative results are shown in Figure 7 And as shown in Table 3.

[0102] Table 3 Segmentation performance (mIoU) of visible light, infrared and fusion images at different times in the same scene

[0103]

[0104] exist Figure 7 In the daytime scenes shown in the first and second rows, the visible light images contain a lot of information, so the segmentation accuracy of the visible light images is high, as shown in the second row of Table 3. However, due to the lack of guidance of infrared targets in visible light images, some detections of people are lost. In addition, the infrared images lack the complement of the visible light image background, and the segmentation accuracy of the bicycle is low, as shown in the third row and sixth column of Table 2.

[0105] In the night scene, Figure 7 As shown in the third and fourth rows, due to insufficient light, visible light images cannot capture enough information, so the segmentation network has a low segmentation accuracy for people in the scene, as shown in the fifth row and fifth column of Table 3. Infrared images capture thermal targets, so the segmentation accuracy for people is higher, as shown in the sixth row and fifth column of Table 3; however, infrared images reduce the segmentation accuracy of bicycles.

[0106] The method of this embodiment is as follows Figure 7 As shown in the third column of , since the addition of the channel space attention module enables the network to perceive long-distance information and enhances the semantic information of the image, the method of this embodiment fully integrates the useful information of the source image. Therefore, regardless of daytime or nighttime scenes, the segmentation accuracy of pedestrians and bicycles by the method of this embodiment is better than that of a single image.

[0107] In order to study the influence of different modules in the model on the final reconstruction results, the guiding role of multi-scale edge gradient block, spatial bias removal block, saliency dense residual block and CWA-Net on fusion is studied; the optimization of the fusion network is analyzed by the loss function. The above six indicators are used as objective evaluation indicators of the ablation experiment. The experimental results show that the model achieves the best performance when using various modules at the same time. The results are shown in Table 4 and Figure 8 As shown,

[0108] Table 4 Quantitative evaluation results of ablation studies. (Bold: best)

[0109]

[0110] (1). CWA-Net loss

[0111] The purpose of using cross entropy loss in CWA-Net is to measure the difference between the prediction result and the true label produced by the fusion process guided by the category weight. Figure 8 As shown in Figure c, when the cross entropy loss is removed, the model's ability to handle class imbalance problems will be weakened, reducing its focus on minority classes, which will affect the overall performance and generalization ability of the model. On the other hand, the values ​​of MI and EN vary greatly, indicating that the fused image contains less information and has a lower image contrast.

[0112] (2) Loss of significance

[0113] The saliency loss guides the branch network to preserve the high contrast of infrared targets, aiming to compensate for the salient target features to the mainstream features. Figure 8 As shown in Figure d, after removing the significant loss, the contrast of the target marked in the red box is significantly reduced, and the SD value (evaluation contrast) in Table 4 is reduced, indicating that there is no significant loss and the infrared target of the network is weakened.

[0114] (3) Content loss

[0115] Content loss uses intensity loss and gradient loss to jointly constrain the network, maintaining the optimal intensity distribution while retaining rich texture details. Figure 8 As shown in Figure e, after removing the content loss, it can be clearly seen that the background texture details of the fused image are significantly reduced, and various indicators are significantly reduced, tending to infrared images. This shows that content loss plays an important role in the feature synthesis of infrared and visible light images in the entire network.

[0116] (4). Multi-scale Edge Gradient Block (MEGB)

[0117] The multi-scale edge gradient block MEGB can fully extract the texture information of the image by embedding the multi-scale feature extraction of the gradient operator. Figure 8 f, when excluding the multi-scale edge gradient blocks, the entire scene is relatively smooth and the gradient changes are small. In addition, AG and Q in Table 4 AB / F The value drops significantly, indicating that this module indeed enhances the representation of network texture details.

[0118] (5). Significant Dense Residual Block (SDRB)

[0119] The significant dense residual block SDRB uses attention to extract pixel distribution in the attention channel. Figure 8 As shown in Fig. 5, after removing the saliency-intensive residual blocks, we can see that the saliency of the fused image object is reduced. The MI values ​​in Table 4 are significantly reduced, indicating that the attention block is crucial to the pixel distribution.

[0120] (6). Spatial Bias Module (SBM)

[0121] SBM effectively completes the progressive fusion process by adding the information of the prominent tributaries to the mainstream. Figure 8 h and Table 4 show that the overall brightness of the fused image becomes darker and the target contrast decreases after removing the spatial bias block (SBM). On the other hand, the VIF, SCD and MI values ​​are significantly reduced, indicating that adding the spatial bias module in the attached figure can not only effectively enhance the infrared target, but also make the fused image more consistent with the human visual system.

[0122] In summary, the module designed in this embodiment not only facilitates the fusion of images visually, but also has obvious improvements in indicators. Therefore, the designed module facilitates the maintenance of textures and the maintenance of salient targets.

[0123] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. An infrared and visible light image fusion network, characterized in that: The invention comprises a dual-branch feature encoding unit, a fusion unit and an image reconstruction unit. The dual-branch feature encoding unit comprises a multi-scale edge gradient module, a significant dense residual module and a spatial bias module. The infrared and visible light images obtain enhanced texture features through the multi-scale edge gradient module. The infrared image obtains enhanced significant features through the significant dense residual module. The enhanced texture features and the strong significant features are fused with global information and local information through the spatial bias module. The features processed by the fusion unit are input into the image reconstruction unit. The fused feature information is reconstructed and the final fused image is output.

2. The infrared and visible light image fusion network according to claim 1, characterized in that: The significant dense residual module includes a dense network module, a residual module and a channel attention module. The infrared image is convolved to obtain features. ,feature Input into the dense network module, the features are reused to obtain the features , the characteristics Get features through channel attention module .

3. The infrared and visible light image fusion network according to claim 2, characterized in that: The features Get features through channel attention module The methods include: feature After the convolution kernel, global average pooling is performed to obtain the global feature vector; the importance of each channel is learned through the fully connected layer; use The activation function obtains weights, assigns higher weights to features with higher contrast, and multiplies the weights with the features obtained by the jump connection of the original input to obtain infrared features containing significant information. ,for: ； Add the salient features to the residual branch features to obtain the salient features of the final output ,for: ; Among them, Conv() represents the convolution operation, represents global average pooling, represents the fully connected layer, represents the activation function, is the salient feature output by the salient dense residual module.

4. The infrared and visible light image fusion network according to claim 3, characterized in that: The working process of the spatial offset module includes: Salient features at the output of the significant dense residual module Adding the spatial offset to , we get: ； in, represents the spatial offset, and and Represent batch normalization and nonlinear activation layers respectively; The texture features output by the multi-scale edge gradient module and the salient features After progressive fusion, the fusion features are obtained ,for: ； Among them, Conv() represents the convolution operation, C() represents the cascade in the channel dimension, Represents the output of salient features after adding the spatial bias term.

5. The infrared and visible light image fusion network according to claim 1, characterized in that: The infrared and visible light image fusion network includes a category weight allocation unit, which includes a SegFormer module, a pooling module and a linear module. The infrared and visible light images obtain category weights of the two images through the SegFormer module, the pooling module and the linear module. The category weights are input into the fusion unit to guide image fusion.

6. The infrared and visible light image fusion network according to claim 5, characterized in that: The class activation weight obtained by the class weight allocation unit for: ； in, is the classification function, is the input image of class c, c∈{ir, vi}, representing infrared image and visible light image respectively, express The probability of belonging to class c is: ； Among them, GAP(·) represents global average pooling, Represents the softmax function.

7. The infrared and visible light image fusion network according to claim 6, characterized in that: The fusion rule of the fusion unit is: ； Among them, ⊗ represents the channel weighted operation; and is the feature map and class activation weight obtained by the class weight allocation unit; is the normalization function.

Images