An infrared and visible light image fusion network

Abstract

This invention discloses an infrared and visible light image fusion network, relating to the field of image fusion. The invention includes a dual-branch feature encoding unit, a fusion unit, and an image reconstruction unit. The dual-branch feature encoding unit comprises a multi-scale edge gradient module, a salient dense residual module, and a spatial bias module. The infrared and visible light images are processed by the multi-scale edge gradient module to obtain enhanced texture features, and the infrared image is processed by the salient dense residual module to obtain enhanced salient features. The enhanced texture features and the enhanced salient features are fused using the spatial bias module to obtain global and local information. The features processed by the fusion unit are input into the image reconstruction unit, where the fused feature information is reconstructed, and the final fused image is output. This invention can better maintain the clarity of image edges and target details, providing higher contrast and visual clarity.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and more particularly to an infrared and visible light image fusion network. Background Technology

[0002] Image and visual information play a crucial role in many fields, but images in a single modality often fail to provide comprehensive scene information. For example, visible light images, rich in detail and conforming to human visual habits, are easy to understand and analyze intuitively. However, their image quality is easily affected by lighting conditions, weather conditions, and background complexity. In low-light environments or harsh weather conditions, image quality deteriorates significantly, affecting the effective acquisition of information. In contrast, infrared images have the advantage of being unaffected by changes in light and weather, and can highlight targets based on temperature differences. However, infrared images typically lack color information and have indistinct background texture features, making them unsuitable for specific scenarios. The fusion of infrared and visible light images aims to integrate information from these two different modalities, thereby overcoming the limitations of single-modality information representation. Through fusion, not only can the contrast advantage of infrared images be preserved to enhance target prominence, but the high-resolution characteristics of visible light images can also be combined to ensure image clarity and detail. This fusion technology has wide applications in intelligent traffic monitoring, drone-based disaster detection, security monitoring, and many other fields, providing strong support for improving system performance and reliability.

[0003] Traditional image fusion methods typically employ algorithms based on sparse representation (SR), low-rank representation (LRR), and multi-scale transformations to fuse infrared and visible light images. With the deepening research on visual saliency, more and more image fusion methods based on visual saliency have emerged. Tang et al., considering the needs of high-level visual tasks (object detection, semantic segmentation, etc.), proposed a high-level visual task-driven image fusion framework (A semantic-aware real-time infrared and visible image fusion network, SeAFusion) to integrate as much semantic information as possible to improve the performance of high-level visual tasks on fused images. Although SeAFusion considers the needs of high-level visual tasks in modeling the fusion problem, as an initial attempt, it only considers improvements to the loss function.

[0004] Most existing image fusion models do not consider the impact of specific category semantic information on the fusion results. They use inappropriate fusion rules when fusing deep features, ignoring complex and diverse infrared and visible light image feature information, which makes it impossible to achieve high-quality image fusion in all weather conditions. This results in insufficient processing of significant information in the fusion results and a lack of semantic information reconstruction. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides an infrared and visible light image fusion system based on dual-branch feature coding and class weight guidance, which can better maintain the clarity of image edges and target details, and provide higher contrast and visual clarity.

[0006] This invention provides an infrared and visible light image fusion network, comprising a dual-branch feature encoding unit, a fusion unit, and an image reconstruction unit. The dual-branch feature encoding unit includes a multiscale edge gradient block (MEGB), a significantly dense residuals module (SDRB), and a spatial bias block (SBM). The infrared and visible light images are enhanced with texture features by the multiscale edge gradient block, and enhanced with salient features by the significantly dense residuals module. The enhanced texture features and the enhanced salient features are fused with global and local information by the spatial bias block. The features processed by the fusion unit are input into the image reconstruction unit, where the fused feature information is reconstructed, and the final fused image is output.

[0007] Furthermore, the significantly dense residual module includes a dense network module, a residual module, and a channel attention module; features are obtained from the infrared image through convolution. ,feature In the dense network module, features are obtained after feature reuse. The features Features are obtained through the channel attention module. .

[0008] Furthermore, the features Features are obtained through the channel attention module. The methods include:

[0009] feature After passing through a convolutional kernel, global average pooling is performed to obtain a global feature vector; the importance of each channel is learned through a fully connected layer; and then... The activation function assigns weights, with higher weights given to features with higher contrast. These weights are then multiplied by the features obtained from the original input via skip connections to yield the infrared feature V containing significant information. C ,for:

[0010] ;

[0011] The salient features are added to the residual branch features to obtain the final output salient features. ,for: ;

[0012] Where Conv() represents the convolution operation, Indicates global average pooling. Indicates a fully connected layer. This represents the activation function. This is a significant feature of the output of the significantly dense residual module.

[0013] Furthermore, the operation of the spatial bias module includes:

[0014] Significant features of the output of the significantly dense residual module Adding a spatial offset, we get:

[0015] ;

[0016] in, This indicates spatial offset, while and These represent batch normalization and non-linear activation layers, respectively.

[0017] The texture features output by the multi-scale edge gradient module and the salient features Fusion characteristics are obtained after gradual fusion. ,for:

[0018] ;

[0019] Where Conv() represents convolution operation, and C() represents concatenation along the channel dimension. This represents the output of significant features after adding a spatial bias term.

[0020] Furthermore, the infrared and visible light image fusion network includes a category weight allocation unit, which includes a SegFormer module, a pooling module, and a linear module. The infrared and visible light images are processed by the SegFormer module, the pooling module, and the linear module to obtain the category weights of the two images. The category weights are input into the fusion unit to guide image fusion.

[0021] Furthermore, the class activation weights obtained by the category weight allocation unit for:

[0022] ;

[0023] in, For classification functions, Let c be the input image of class c. c∈{ir, vi} represents the infrared image and the visible light image, respectively. express The probability of belonging to class c is:

[0024] ;

[0025] Where GAP(·) represents global average pooling, This represents the softmax function.

[0026] Furthermore, the fusion rule of the fusion unit is as follows:

[0027] ;

[0028] Where ⊗ represents channel weighting operation; and These are the feature maps and class activation weights obtained from the category weight allocation unit; This is the normalization function.

[0029] Compared with the prior art, the present invention has the following technical effects:

[0030] The MEGB module of this invention helps extract and preserve the texture gradient of the entire network, enhancing the ability of the fused image to describe global scene information. On the other hand, the SDRB module extracts salient targets from infrared images and generates a saliency map guided by saliency loss. Finally, the SBM module is used to compensate the saliency features of the branches into the mainstream, completing information fusion. Comparative experiments of the qualitative and quantitative aspects of this invention with existing methods demonstrate its effectiveness. Fusion experiments in different scenarios also show that it can effectively achieve full fusion of semantic information from infrared and visible light images in different scenarios. The fusion result of the proposed method retains clearer edge and target information, exhibiting better contrast and clarity.

[0031] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the structure of an infrared and visible light image fusion network according to a specific embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram of the structure of a multi-scale edge gradient module according to a specific embodiment of the present invention;

[0034] Figure 3 This is a schematic diagram of the structural principle of a significantly dense residual module according to a specific embodiment of the present invention;

[0035] Figure 4 This is a schematic diagram of the spatial biasing module according to a specific embodiment of the present invention;

[0036] Figure 5 This is a visual quality comparison result based on the MSRS dataset according to a specific embodiment of the present invention, wherein, Figure 5 a is a visible light image; Figure 5 b represents the infrared image; Figure 5 c is the image obtained using the DenseFuse model; Figure 5 d represents the image obtained using the RFN-Nest model; Figure 5 e is the image obtained using the FusionGAN model; Figure 5 f is the image obtained using the SDNet model; Figure 5 g is the image obtained using the U2Fusion model; Figure 5 h is the image obtained using the SeAFusion model; Figure 5 i represents the image obtained using the PIAFusion model; Figure 5 j is the image obtained using the image fusion network of the present invention;

[0037] Figure 6 This is a visual quality comparison result based on the TNO dataset according to a specific embodiment of the present invention, wherein, Figure 6 a is a visible light image; Figure 6 b represents the infrared image; Figure 6 c is the image obtained using the DenseFuse model; Figure 6 d represents the image obtained using the RFN-Nest model; Figure 6 e is the image obtained using the FusionGAN model; Figure 6 f is the image obtained using the SDNet model; Figure 6 g is the image obtained using the U2Fusion model; Figure 6 h is the image obtained using the SeAFusion model; Figure 6 i represents the image obtained using the PIAFusion model; Figure 6 j is the image obtained using the image fusion network of the present invention.

[0038] Figure 7 This is a visual quality comparison result of the segmentation results according to a specific embodiment of the present invention;

[0039] Figure 8 This is a visual quality comparison result of an important loss function and module ablation study in a specific embodiment of the present invention, wherein, Figure 8 'a' represents a visible light image; Figure 8 b represents infrared image. Figure 8 c represents the result after removing the CWA-Net loss. Figure 8 d represents the result after removing the significant loss. Figure 8 e represents the result of removing content loss. Figure 8 f represents the result of removing the MEGB module. Figure 8 g represents the result of removing the SDRB module; Figure 8 h represents the result of removing the SBM module; Figure 8 i represents the result of this invention. Detailed Implementation

[0040] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0041] like Figure 1 As shown, in a specific embodiment, an infrared and visible light image fusion network is provided, including a dual-branch feature encoding unit, a class weight allocation unit, a fusion unit, and an image reconstruction unit. The dual-branch feature encoding unit can fully integrate complementary and consistent features of different modalities; class activation weights are obtained through the class weight allocation network to guide the fusion; finally, the fused feature information is reconstructed through the image reconstruction network to output the final fused image.

[0042] The dual-branch feature encoding unit includes a multiscale edge gradient block (MEGB), a significantly dense residuals module (SDRB), and a spatial bias block (SBM). Infrared and visible light images are enhanced with texture features by the multiscale edge gradient block, and the infrared image is enhanced with salient features by the significantly dense residuals module. The enhanced texture features and the strong salient features are then fused using the spatial bias block to integrate global and local information.

[0043] like Figure 2 As shown, the multi-scale edge gradient module consists of a multi-scale branch and a residual gradient branch. It uses convolutional blocks with kernels of different sizes to extract spatial features at different scales, enhancing the receptive field and better extracting texture details from infrared and visible light. Simultaneously, it combines the residual gradient with the Sobel operator to further preserve edge information, retaining more edge and texture details. Finally, the outputs of the multi-scale branch and the residual gradient branch are combined to complete the texture detail enhancement.

[0044] In multi-scale branches, infrared images and visible light images Adding them along the channel dimension yields the fused features. fusion features The input is fed into the Multi-Scale Augmentation Module (MEGB), and the output features of the MEGB are... for:

[0045] (1)

[0046] in This represents the fused features obtained by adding infrared and visible light images along the channel dimension. This represents features extracted at different scales. Conv() represents convolution operation, and C() represents concatenation along the channel dimension. Represents the Sobel operator. This indicates summation of elements.

[0047] like Figure 3 As shown, the significant dense residual module includes a dense network module, a residual module, and a channel attention module. Features are obtained from the infrared image through convolution. ,feature In the dense network module, features are obtained after feature reuse. The features After convolution with a 3×3 kernel, global average pooling is performed to obtain the global feature vector; the importance of each channel is learned through a fully connected layer, and then... The activation function assigns weights, with higher weights given to features with higher contrast. These weights are then multiplied by the features obtained from the original input via skip connections to yield the infrared feature V containing significant information. C The above process is as follows:

[0048] (2)

[0049] (3)

[0050] Contrast enhancement is achieved by adding salient features to residual branch features, thereby highlighting salient targets and obtaining the salient features of the final output. ,for:

[0051] (4)

[0052] Where Conv() represents the convolution operation, Indicates global average pooling. Indicates a fully connected layer. This represents the activation function. This is a significant feature of the output of the significantly dense residual module.

[0053] like Figure 4 As shown, the working process of the spatial bias module includes:

[0054] These are the semantic features of MEGB and the salient features of SDRB, respectively, which are progressively integrated. Since simple convolutional layers, due to their limited receptive field, cannot learn long-range dependencies, a spatial bias module is added to TBFEN to learn the relationships between different distant objects, thereby enhancing the semantic information of the image. This module is lightweight and computationally faster than self-attention mechanisms. For the salient features of SDRB, the spatial bias term B can be expressed as:

[0055] (5)

[0056] in, This indicates spatial offset, while and These represent batch normalization and non-linear activation layers, respectively.

[0057] MEGB's semantic features are represented by the grayscale distribution of pixels and their surrounding spatial domain, reflecting local information. To capture both local and global information simultaneously, spatial bias features and semantic features are concatenated along the channel dimension; to aggregate global knowledge, a 1×1 convolution is applied along the channel dimension to effectively compress and fuse information from the feature maps. Ultimately, the semantic features... Significant features The process of splicing and gradually merging can be represented as follows:

[0058] (6)

[0059] in, This represents the fused features after progressive fusion. Conv() represents the convolution operation, and C() represents the concatenation along the channel dimension. This represents the output of salient features after adding a spatial bias term. For the characteristic output of the MEGB module, This is the characteristic output of the SDRB module.

[0060] like Figure 5 As shown, the visual quality comparison results based on the MSRS dataset were obtained, in which seven typical fusion models, namely DenseFuse, RFN-Nest, FusionGAN, SDNet, U2Fusion, SeAFusion, and PIAfusion, were selected for comparison with the fusion model in this paper. In the figure, a represents a visible light image; b represents an infrared image; c represents an image obtained by the DenseFuse (A Fusion Approach to Infrared and Visible Images) model; d represents an image obtained by the RFN-Nest (An end-to-end residual fusion network for infrared and visible images) model; e represents an image obtained by the FusionGAN (A generative adversarial network for infrared and visible image fusion) model; f represents an image obtained by the SDNet (A versatile squeeze-and-decomposition network for real-time image fusion) model; g represents an image obtained by the U2Fusion (A unified unsupervised image fusion network) model; h represents an image obtained by the SeAFusion (Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network) model; i represents an image obtained by the PIAFusion (A progressive infrared and visible image fusion network based on illumination aware) model; and j represents an image obtained by the method used in this invention.

[0061] like Figure 6As shown, the visual quality comparison results based on the TNO dataset were obtained by comparing seven typical fusion models—DenseFuse, RFN-Nest, FusionGAN, SDNet, U2Fusion, SeAFusion, and PIAfusion—with the fusion model presented in this paper. In the figure, a represents a visible light image; b represents an infrared image; c represents an image obtained by the DenseFuse model; d represents an image obtained by the RFN-Nest model; e represents an image obtained by the FusionGAN model; f represents an image obtained by the SDNet model; g represents an image obtained by the U2Fusion model; h represents an image obtained by the SeAFusion model; i represents an image obtained by the PIAfusion model; and j represents the method used in this invention.

[0062] like Figure 7 As shown, the visual quality comparison results of the semantic segmentation experiment are presented. In the figure, the top part represents the visible light image (VIS), the infrared image (IR), the segmentation result image (Ours), and the ground truth image (Ground Truth). In the first row and fourth column, the image uses five categories for comparative analysis: Background, Car, Person, Bike, and Curve.

[0063] like Figure 8 As shown, the visual quality comparison results of the important loss function and module ablation study are presented. In the figure, a represents the visible light image; b represents the infrared image; c represents the result of removing CWA-Net loss; d represents the result of removing saliency loss; e represents the result of removing content loss; f represents the result of removing MEGB module; g represents the result of removing SDRB module; h represents the result of removing SBM module; and i represents the result of this invention.

[0064] like Figure 1 As shown, the category weight allocation unit includes a SegFormer module, a pooling module, and a linear module. Infrared and visible light images are processed by the SegFormer module, the pooling module, and the linear module to obtain the category weights of the two images. The category weights are input into the fusion unit to guide image fusion.

[0065] Each feature map is reduced to a scalar using global average pooling, and then linear layers learn to map these scalars to the respective classes. To eliminate human intervention in the fusion process, the class-specific weight-guided feature fusion strategy does not require manual design; the fusion weights are obtained by deep learning. The class weight assignment network aims to distinguish between infrared and visible light images, and the class activation weights can be represented as:

[0066] (7)

[0067] in For classification functions, Let c be the input image of class c. c∈{ir, vi} represents the infrared image and the visible light image, respectively. express The probability of belonging to class c. This represents the weights assigned to the last linear layer in the network. It is a 1×1×M vector generated during the training phase, also known as class activation weights according to CAM theory.

[18] The CAM algorithm is designed to explore interpretability in machine learning, aiming to help humans understand the internal workings of black-box models. The proposed fusion rule Γ(·) is shown in (8):

[0068] Feature mapping obtained from semantic segmentation module and class activation weights ,

[0069] (8)

[0070] Where ⊗ represents channel weighting operation. and Feature maps and class activation weights obtained from CWA-Net. The normalization function is defined as shown in equation (10):

[0071] (9)

[0072] Normalization by nonlinear functions Achieve this by multiplying by an adjustment factor. / The mean of the original weights is corrected to δ, which mitigates the decrease in brightness of the fusion result. The channel weighting operation in equation (8) requires feature mapping. With class activation weights Establish a strict correspondence between them, and establish a correspondence relationship. Let... This represents the output feature of the last MLP. and They correspond to each other in channel order. Secondly, in the output stage, a CAM mechanism is established to ensure... and The correspondence. And for Perform a global average pooling (GAP) operation and reduce it to a 1 × 1 × M vector. Then, use a linear layer with a softmax activation function at the end of the network. The operations of GAP and the linear layer are shown in Equation (11):

[0073] (10)

[0074] Where GAP(·) represents global average pooling. This represents the softmax function. With the above settings, the linear weights... The contribution of channel m to the classification result can be directly adjusted. Therefore, after the training phase, a larger... This indicates that channel m plays an important role in the distribution network. In other words, It can reflect the classification basis of the assignment network and reveal the richness of information contained in each channel. For example, if the features of channel m carry important features of the visible image (such as texture details), then the corresponding weights... It will be relatively large. In some areas containing and In channels with common characteristics, due to and With similar values, the fusion rule can avoid the loss of information from either side. The fusion rule utilizes class activation weights to achieve flexible information selection and enhancement at the deep feature level.

[0075] To ensure the accuracy of class weight allocation, a cross-entropy loss function is used to guide and optimize the network's learning process, as shown in the following equation:

[0076] (11)

[0077] in, The label represents the input image. This represents the output probability of the category assignment network. This represents the softmax function that normalizes the output probability to [0,1].

[0078] The MEGB branch aims to preserve rich texture details while maintaining the optimal intensity distribution in the fused image. Therefore, it introduces content loss, which includes intensity loss and texture loss. The intensity loss measures the pixel-level intensity distribution between the fused image and the source image, while the texture loss aims to make texture details clearer. The gradient of the fused image is close to the maximum gradient values ​​of visible light and infrared images. Specifically, it is shown in equation (12):

[0079] (12)

[0080] Where H and W are the height and width of the input image, respectively. Used to achieve a balance between intensity loss and texture loss. This is the Sobel gradient operator, used to measure the gradient texture of an image.

[0081] The purpose of the CWA-Net network in this embodiment is to guide fusion, optimize the model's ability to identify salient regions in the image, and adapt to the class imbalance problem. Therefore, in order to preserve the salient targets in the fused image, an intermediate salient loss is constructed using a target mask, as shown in Equation (13):

[0082] (13)

[0083] in For the target mask, Channel average .

[0084] Finally, the loss function of this deep learning model combines class weight loss, content loss, and saliency loss, and optimizes the fusion effect of infrared and visible light images through a weighted λ combination. The total loss is shown in Equation (14):

[0085] (14)

[0086] By obtaining ideal texture details with salient targets under the guidance of cross-entropy loss, content loss, and saliency loss, it is able to fuse meaningful information from the source image.

[0087] The reconstruction unit consists of four convolutional layers, such as Figure 1 As shown. The task of these convolutional layers is to fully integrate common and complementary information to generate the final fused image. Except for the last layer which uses a 1x1 kernel size, all other layers use 3x3 convolutional kernels. During image reconstruction, the number of channels in the feature map gradually decreases, allowing for progressive refinement of information. The last convolutional layer uses the Tanh function, while the first three convolutional layers use Leaky ReLU. To ensure that information is effectively transmitted throughout the network and to reduce information loss, the padding of all layers except the first and last layers is set to 1, avoiding the use of downsampling and thus ensuring that the output fused image has the same size as the source image.

[0088] To further illustrate the effects of the present invention, comparative experiments are conducted below to verify its effectiveness.

[0089] In this embodiment, the TNO and MSRS datasets are used for training. The MSRS dataset contains 1569 pairs of high-quality registered infrared and visible light images. The TNO multi-band image set provides nighttime images of different military and surveillance scenarios, showing different objects and targets in different backgrounds. The training set selects 45 pairs of images from different scenarios from TNO, and 416 daytime scene images and 354 nighttime scene images from the MSRS dataset, all with a size of 480×640. The test set selects infrared and visible light image fusion from the public datasets TNO and MSRS, respectively, and selects 42, 20, 361, and 300 pairs of images for algorithm comparison experiments. To verify the effectiveness of the method proposed in this embodiment, seven typical fusion models, including DenseFuse, RFN-Nest, FusionGAN, SDNet, U2Fusion, SeAFusion, and PIAfusion, are selected for comparison with the fusion model of this embodiment, and the comparative experiments are analyzed using six subjective and objective evaluation indicators.

[0090] All parameters were set as follows during training: The training epoch was set to 10. Hyperparameters were set to: γ1 = 3, γ2 = 7, γ3 = 50. Furthermore, the algorithm used the Adam optimizer with a bath size of 8. The initial learning rate was 10. -4 The experiment in this embodiment was conducted on a GeForce RTX 2080Ti 16GB using PyTorch as the deep learning framework. The network model was trained using the PyTorch framework.

[0091] To verify the effectiveness of the method in this embodiment, tests were conducted on the publicly available datasets TNO and MARS. Mutual Information (MI), visual information fidelity (VIF), average gradient (AG), sum of correlation differences (SCD), and an edge-information-based index (Q) were selected. AB / F Six commonly used objective evaluation indicators were used to compare and analyze the quality of fused images. These included EN, MI, SCD, SF, and Q. abf VIF is an indicator for evaluating spatial information. It is a visual perception indicator, and all six indicators are positive indicators, meaning that the higher the indicator, the better the fusion performance.

[0092] The metrics comparisons of different methods are shown in Tables 1 and 2. The optimal value of AG indicates that the fusion method of this embodiment has richer contrast information and richer texture details; the optimal value of SF indicates that the gradient distribution of the fused image reveals the details and texture information of the fused image; the optimal value of EN indicates that the method of this embodiment retains sufficient edge information; the optimal value of SCD indicates that the fusion result contains more realistic information; VIF performs best or second best on both datasets, indicating that the method of this embodiment has richer contrast information and generates a fused image that is more consistent with the human visual system; Q abf The optimal value indicates that more edge information is transferred from the source image to the fused image. Furthermore, the six metrics are optimal or near-optimal on both datasets, demonstrating the superior generalization performance of the method in this embodiment, making it applicable to different types of datasets. In summary, the method in this embodiment can extract effective information in class imbalance and occlusion scenes and integrate this information into the fused image using a channel spatial attention module. Therefore, the method in this embodiment has a greater advantage than other methods in obtaining high-quality fused images.

[0093] Table 1. Quantitative evaluation results of different methods on the TNO dataset (Bold: best)

[0094]

[0095] Table 2. Quantitative evaluation results of different methods on the MSRS dataset (Bold: best)

[0096]

[0097] Visualization results of 8 pairs of images from 2 datasets are as follows Figure 5-6 As shown. Areas with significant differences are highlighted with red and green boxes, with magnified images of the red boxes located in the lower right or lower left corners.

[0098] like Figure 5 In the daytime scene shown, DenseFuse and RFN-Nest weaken infrared targets, FusonGAN causes edge texture blurring, and SDNet and SeAFusion weaken the background texture details of the image, as shown in the green box in the figure; only PIAFusion and the method of this embodiment can integrate effective information.

[0099] In such Figure 6In the night scene shown, the visible light image contains only a small amount of texture information, while the infrared image, in addition to the prominent target, also contains background texture details. Many methods overemphasize the information of a single modality of image, making it difficult to achieve good results in different scenes. Among them, infrared targets are weakened in DenseFuse, RFN-Nest, U2Fusion, and SeAFusion, and the fused images of FusonGAN and SDNet are more biased towards infrared images, resulting in blurred background information. By fully extracting the details contained in the mainstream texture image and using salient branches to supplement the weakened salient targets, semantic information in complex scenes can be effectively extracted.

[0100] As can be seen, this embodiment not only has comprehensive scene information, but also retains rich contrast information and texture details of the target area.

[0101] The semantic segmentation algorithm was trained on both the source and fused images. A training set of 1000 images was selected, and the segmentation performance of different models was tested on 360 images. Qualitative and quantitative results are as follows: Figure 7 As shown in Table 3.

[0102] Table 3. Segmentation performance (mIoU) of visible light, infrared, and fused images at different times in the same scene.

[0103]

[0104] exist Figure 7 In the daytime scenes shown in the first and second rows, the visible light images contain a wealth of information, resulting in high segmentation accuracy, as shown in the second row of Table 3. However, due to the lack of guidance from infrared targets in the visible light images, some human detections are missed. Furthermore, the infrared images lack background compensation from the visible light images, leading to lower segmentation accuracy for bicycles, as shown in the sixth column of the third row of Table 2.

[0105] In the night view, such as Figure 7 As shown in the third and fourth rows, due to insufficient light, visible light images cannot capture enough information, resulting in lower segmentation accuracy for people in the scene, as shown in the fifth column of the fifth row of Table 3. Infrared images, on the other hand, capture thermal targets, thus achieving higher segmentation accuracy for people, as shown in the fifth column of the sixth row of Table 3; however, infrared images reduce the segmentation accuracy for bicycles.

[0106] The method in this embodiment is as follows: Figure 7 As shown in the third column, the addition of the channel spatial attention module enables the network to perceive long-distance information, enhancing the semantic information of the image. Therefore, the method in this embodiment fully integrates useful information from the source image. Consequently, regardless of whether it is a daytime or nighttime scene, the method in this embodiment achieves better segmentation accuracy for pedestrians and bicycles than for a single image.

[0107] To investigate the impact of different modules in the model on the final reconstruction result, the guiding role of multi-scale edge gradient blocks, spatial bias removal blocks, saliency-dense residual blocks, and CWA-Net in the fusion process was analyzed. The optimization of the fusion network using the loss function was also examined. These six metrics were used as objective evaluation indicators for the ablation experiments. Experimental results show that the model achieved the best performance when all modules were used simultaneously. The results are shown in Table 4. Figure 8 As shown,

[0108] Table 4. Quantitative evaluation results of the ablation studies. (Bold: Best)

[0109]

[0110] (1). CWA-Net loss

[0111] The purpose of CWA-Net's cross-entropy loss is to measure the difference between the predicted results and the true labels produced by the fusion process guided by class weights. For example... Figure 8 As shown in c, removing the cross-entropy loss weakens the model's ability to handle class imbalance, reduces its focus on minority classes, and consequently affects the model's overall performance and generalization ability. On the other hand, the large changes in MI and EN values ​​indicate that the fused image contains less information and has lower contrast.

[0112] (2) Significance loss

[0113] The saliency loss-guided branch network preserves the high contrast of infrared targets, aiming to compensate for salient target features by shifting the feature mainstream. For example... Figure 8 As shown in d, after removing significant losses, the contrast of the target marked in the red box is significantly reduced. The SD value (evaluation contrast) in Table 4 is reduced, indicating that there are no significant losses and the infrared target of the network is weakened.

[0114] (3) Content loss

[0115] Content loss utilizes both intensity loss and gradient loss to constrain the network, preserving rich texture details while maintaining the optimal intensity distribution. For example... Figure 8 As shown in Figure e, after removing the content loss, it is evident that the background texture details of the fused image are significantly reduced, and all indicators show a marked decline, leaning towards the infrared image. This demonstrates that content loss plays a crucial role in the feature synthesis of infrared and visible light images throughout the entire network.

[0116] (4). Multi-scale edge gradient blocks (MEGB)

[0117] Multi-scale edge gradient blocks (MEGB) can fully extract the texture information of an image through multi-scale feature extraction by embedding gradient operators. For example... Figure 8 As shown in f, when multi-scale edge gradient blocks are excluded, the entire scene is relatively smooth with small gradient changes. Furthermore, AG and Q in Table 4... AB / F The significant decrease in the value indicates that the module does indeed enhance the representation of network texture details.

[0118] (5). Significantly Dense Residual Blocks (SDRB)

[0119] Significantly Dense Residual Blocks (SDRBs) utilize attention to extract pixel distributions of network features within the attention channel. For example... Figure 8 As shown in g, after removing the saliency-dense residual blocks, the target saliency of the fused image is reduced. The MI values ​​in Table 4 are significantly reduced, indicating that the attention blocks are crucial to pixel distribution.

[0120] (6). Spatial Offset Module (SBM)

[0121] SBM effectively accomplishes the gradual fusion process by adding information from prominent tributaries to the mainstream. From Figure 8 As shown in h and Table 4, the overall brightness of the fused image is darker, and the target contrast is reduced after removing the spatial bias block (SBM). On the other hand, the VIF, SCD, and MI values ​​are significantly reduced, indicating that adding the spatial bias module to the attached figure can effectively enhance the infrared target and make the fused image more consistent with the human visual system.

[0122] In summary, the module designed in this embodiment not only facilitates image fusion visually, but also significantly improves performance metrics. Therefore, the designed module facilitates both texture maintenance and the maintenance of prominent targets.

[0123] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An infrared and visible light image fusion network, characterized in that, The system includes a dual-branch feature coding unit, a fusion unit, an image reconstruction unit, and a category weight allocation unit. The dual-branch feature coding unit includes a multi-scale edge gradient module, a salient dense residual module, and a spatial bias module. The infrared and visible light images are processed by the multi-scale edge gradient module to obtain enhanced texture features, and the infrared image is processed by the salient dense residual module to obtain enhanced salient features. The enhanced texture features and the enhanced salient features are fused with global and local information by the spatial bias module. The features processed by the fusion unit are input into the image reconstruction unit, where the fused feature information is reconstructed, and the final fused image is output. The category weight allocation unit includes a SegFormer module, a pooling module, and a linear module. Infrared and visible light images are processed by the SegFormer module, the pooling module, and the linear module to obtain the category weights of the two images. The category weights are input into the fusion unit to guide image fusion. The class activation weight obtained by the category weight allocation unit for: ； in, For classification functions, Let c be the input image of class c, where c∈{ir, vi}, representing the infrared image and the visible light image respectively. express The probability of belonging to class c is: ； Where GAP(·) represents global average pooling, This represents the softmax function; The fusion rules of the fusion unit are as follows: ； in, Indicates channel weighting operation; and These are the feature maps and class activation weights obtained from the category weight allocation unit; This is the normalization function.

2. The infrared and visible light image fusion network according to claim 1, characterized in that, The significant dense residual module includes a dense network module, a residual module, and a channel attention module. The infrared image is used to obtain features ϕ through convolution. C , characteristic ϕ C In the dense network module, features are obtained after feature reuse. The features Features are obtained through the channel attention module. .

3. The infrared and visible light image fusion network according to claim 2, characterized in that, The features Features are obtained through the channel attention module. The methods include: feature After passing through a convolutional kernel, global average pooling is performed to obtain a global feature vector; the importance of each channel is learned through a fully connected layer; and then... The activation function assigns weights, with higher weights given to features with higher contrast. These weights are then multiplied by the features obtained from the original input via skip connections to yield the infrared feature V containing significant information. C ,for: ； The salient features are added to the residual branch features to obtain the final output salient features. ,for: ; Where Conv() represents the convolution operation, Indicates global average pooling. Indicates a fully connected layer. This represents the activation function. This is a significant feature of the output of the significantly dense residual module.

4. The infrared and visible light image fusion network according to claim 3, characterized in that, The operation of the spatial offset module includes: Significant features of the output of the significantly dense residual module Adding a spatial offset, we get: ； in, This indicates spatial offset, while and These represent batch normalization and non-linear activation layers, respectively. The texture features output by the multi-scale edge gradient module and the salient features Fusion characteristics are obtained after gradual fusion. ,for: ； Where Conv() represents convolution operation, and C() represents concatenation along the channel dimension. This represents the output of significant features after adding a spatial bias term.

Images