An infrared and visible light fusion method based on multiple feature extraction

Abstract

The application discloses an infrared and visible light fusion method based on multiple feature extraction, and the multiple feature extraction network is composed of a fusion network, a semantic segmentation network, a content loss function and a semantic loss function; the fusion network comprises a feature extraction part and a feature reconstruction part, the feature extraction part comprises a convolution layer with an activation function, an adaptive gradient feature extraction module, a spatial channel reconstruction convolution and a residual gradient extraction module; the feature reconstruction part is composed of an attention mechanism and four convolution layers connected in series; the steps are as follows: S1, inputting an infrared image and a visible light image into the fusion network, generating a fusion image through feature extraction and feature reconstruction; S2, delivering the fusion image to the segmentation network for segmentation; S3, comparing the segmentation result with a semantic label, and evaluating the richness of semantic information contained in the fusion image. The image after fusion is clearer in outline and clearer in target, and the index is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an infrared and visible light fusion method based on multiple feature extraction. BACKGROUND

[0002] The infrared image has the advantages of strong penetration and being not affected by light, but lacks texture information. The visible light image has rich structure information and good visual effect, but is easily affected by weather and light conditions and has poor anti-interference ability. Therefore, a method of fusing the two is in urgent need.

[0003] Methods of image fusion are mainly divided into two categories, traditional fusion methods and deep learning-based fusion methods. Traditional fusion methods mainly include multi-scale transformation, sparse representation, saliency fusion, subspace decomposition, and hybrid model. At present, the methods of infrared and visible image fusion based on deep learning mainly include three kinds: based on auto-encoder (AE), based on convolutional neural network (CNN) and based on generative adversarial network (GAN). Tang et al. (TANG L F, YUAN J T, MA J Y. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42) combined the image fusion problem and the semantic segmentation problem, and designed a semantic-aware fusion framework SeAFusion. The Gan network mainly includes a generator and a discriminator, and the loss function is formulated to constrain the adversarial game between the two to make the fused image more approximate to the original image. Ma et al. (MA J Y, ZHANG H, SHAO Z F, et al. GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14) first introduced Gan into image fusion, but the fused image obtained by single adversarial training is too smooth and fuzzy and the details are unstable, so intensity loss and gradient loss are introduced. Although the current neural network can perform end-to-end image tasks, there are still some shortcomings in the fusion method:

[0004] 1) The fused image is not clear, the background is blurred, and there is a lot of redundant information;

[0005] 2) The original image is ignored, and too much reliance is placed on feature maps for image reconstruction, resulting in structural differences between the fused image and the original visible light image;

[0006] 3) The texture information is easily lost in the fused image when the background is complex. SUMMARY

[0007] The application aims to provide an infrared and visible light fusion method based on residual feature extraction, so as to improve the extraction capability of the fusion network for image fine-grained features, highlight target information and reserve texture information.

[0008] The technical scheme is an infrared and visible light fusion method based on multiple feature extraction, which uses an adaptive gradient feature extraction module containing a gradient operator to perceive fine texture changes in an image and calculate gradient information of the image in different directions, and then obtains shallow features containing rich edge information; a residual gradient extraction module with a residual connection is used to make the shallow features perform information fusion at different scales and generate deep features; the shallow features are combined with the deep features through a spatial channel reconstruction convolution SCConv, the infrared light image and the visible light image are input into a multi-feature extraction network, and real-time fusion of the infrared light image and the visible light image is realized.

[0009] The multi-feature extraction network is composed of a fusion network, a semantic segmentation network, a content loss function and a semantic loss function.

[0010] The fusion network includes a feature extraction part and a feature reconstruction part, the feature extraction part includes a convolution layer with an activation function, an adaptive gradient feature extraction module, a spatial channel reconstruction convolution and a residual gradient extraction module; the feature reconstruction part is composed of an attention mechanism and four convolution layers connected in series.

[0011] The implementation steps are as follows:

[0012] S1, input the infrared image and the visible light image into the fusion network, and generate a fusion image through feature extraction and feature reconstruction;

[0013] S2, pass the fusion image to the segmentation network for segmentation;

[0014] S3, compare the segmentation result with the semantic label to evaluate the richness of the semantic information contained in the fusion image.

[0015] Further, the implementation steps of the feature extraction part for feature extraction are as follows:

[0016] S11, input the infrared image and the visible light image into a convolution layer with a convolution kernel size of 1x1;

[0017] S12, cross input the infrared and visible light feature maps after convolution into a convolution layer with a 3x3 convolution kernel;

[0018] S13, extract the features through the adaptive gradient feature extraction module and the residual gradient extraction module to deeply extract texture and background information;

[0019] S14, multi-scale feature extraction is performed through spatial channel reconstruction convolution, and feature reconstruction is performed by splicing and combining.

[0020] Further, the adaptive gradient feature extraction module includes three flat branches, each layer branch including a Zpool layer, a convolution layer and a Sigmoid function activation layer; the first two layer branches respectively perform rotation on the input tensor in the H axis and the W axis, and then the shape HxCxW of the tensor is transformed into 2xCxW through the Zpool layer, and then the intermediate dimension tensor 1xCxW used for synthesizing attention weight is obtained through the convolution layer with batch normalization, and finally the attention weight is generated through the Sigmoid function and used for the input tensor, and the rotated tensor is rotated counterclockwise once; for the third layer branch, the input tensor is directly subjected to the Zpool layer, the convolution layer and the Sigmoid function to obtain the spatial weight and be used for the input tensor; finally, the three branch results are added and averaged; the obtained feature map is sent to the dense connection network with a convolution kernel of 3x3, and the Sobel gradient operator is introduced to perform fine-grained extraction on the image, and then channel dimension reduction is performed through the convolution with a convolution kernel of 1x1, and the original feature map is spliced after residual connection through the 5x5 convolution layer.

[0021] Further, the spatial channel reconstruction module utilizes the spatial redundancy of features by introducing a spatial reconstruction unit, and utilizes the channel redundancy of features by introducing a channel reconstruction unit, and the image channels are divided into multiple units, each unit is fused after standard transformation.

[0022] Further, the residual gradient extraction module includes three branch channels, the uppermost branch convolution channel performs edge feature extraction through a 5x5 convolution kernel, and then changes the output dimension of the feature map through a 1x1 convolution; the middle branch channel extracts texture information of the feature map and changes the channel dimension through two 3x3 convolution layers and a 1x1 convolution; the lowermost branch channel includes a jump connection identity mapping convolution layer with a 1x1 convolution kernel; except for the last 1x1 convolution of each layer channel, the remaining convolution layers are all provided with a linear rectifier unit activation function; finally, the outputs of the three branch channels are added.

[0023] Further, the attention mechanism is sequentially connected by a channel attention module and a spatial attention module, and the two perform attention mechanism in the channel dimension and the spatial dimension respectively; the processing process of the feature map is as follows:

[0024] The input feature map is first processed by two parallel MaxPool layers and AvgPool layers, so that the size of the feature map changes from CxHxW to Cx1x1; then, the feature map is transmitted into a shared MLP module, which first reduces the number of channels to 1 / r of the original number, then expands back to the original number of channels, and obtains two activated outputs through the ReLU activation function; then, the two outputs are added element by element, and the final channel attention feature map is generated through the Sigmoid activation function; finally, the final channel attention feature map is multiplied point by point with the original feature map to restore the size of CxHxW.

[0025] Further, the processing process of the spatial attention module on the feature map is as follows:

[0026] Firstly, the output result of the channel attention is processed in the spatial dimension by using the maximum pooling and average pooling operations, and two feature maps with the size of 1xHxW are obtained;

[0027] Then, it is spliced and sent into a 7x7 convolution layer, and the single-channel feature map generated by the convolution layer is processed by the Sigmoid activation function to obtain a spatial attention feature map;

[0028] Finally, the obtained spatial attention feature map is multiplied with the original feature map to restore the original size of CxHxW.

[0029] Further, the content loss function includes intensity loss l int and texture loss l texture , which are defined as follows:

[0030] l content =l int +αl texture

[0031]

[0032]

[0033] Wherein, the intensity loss l int can constrain the overall intensity of the fused image, the texture loss l texture can promote the fused image to contain more detailed texture details; alpha is used to effectively balance between intensity loss and texture loss; H and W represent the height and width of the image respectively, ||.||1 represents the l1 norm, and max(,) represents the maximum selection element by element; is the Sobel gradient operator used to measure the fine-grained texture information of the image; |. | represents the absolute value operation.

[0034] Further, the semantic loss includes subject semantic loss l mainand auxiliary semantic loss l aux , defined as follows:

[0035]

[0036] wherein, L so ∈R H×W×C represents the single-hot vector converted by the segmentation symbol L s ∈(1,C) H×W×C transformed by the segmentation symbol L

[0037] Finally, the semantic loss l semantic is expressed as:

[0038] l semantic =l main +λl aux

[0039] wherein, λ is a constant for balancing the main semantic loss and the auxiliary semantic loss.

[0040] Compared with the prior art, the present application has the following remarkable effects:

[0041] 1. The present application introduces a segmentation network to segment the fused image, and guides the training of the fusion network through the constructed semantic loss to make the fused image contain more semantic information; not only adaptive gradient feature extraction module (AGFE, Adaptive Gradient Feature Extraction Block) and residual gradient extraction module (RGE, Residual gradient extraction) are designed in the fusion network to improve the image fusion quality, but also the source image is cross-processed instead of simply splicing the feature maps in the subsequent stage, so as to improve the complementarity and prevent information redundancy;

[0042] 2. The present application extracts the local important features of the source image through the design of the adaptive gradient feature extraction module AGFE and the residual gradient extraction module RGE, improves the extraction ability of the fusion network for the fine-grained features of the image, and effectively realizes the highlighting of the target information and the reservation of the texture information;

[0043] 3. The present application introduces CBAM attention mechanism and spatial channel reconstruction convolution SCConv which can replace traditional convolution to solve the problem of semantic information loss that may occur in the fusion process, and connects the source image through the spatial channel reconstruction convolution SCConv to ensure the structural similarity of the fusion result, so that the fused image greatly retains more texture details of the source image; finally, through the subjective and objective evaluation, it is proved that the present application can better maintain the target information of the infrared image and the detail information of the visible light image in the complex background. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The overall architecture diagram of the multi-feature extraction network of the application is shown in the figure;

[0045] Figure 2 The architecture diagram of the fusion network is shown in the figure;

[0046] Figure 3 The structure diagram of the adaptive gradient feature extraction module is shown in the figure;

[0047] Figure 4 The structure diagram of the spatial channel reconstruction convolution is shown in the figure;

[0048] Figure 5 The structure diagram of the residual gradient extraction module is shown in the figure;

[0049] Figure 6 The structure diagram of the attention mechanism is shown in the figure;

[0050] Figure 7 The structure diagram of the channel attention mechanism is shown in the figure;

[0051] Figure 8 The structure diagram of the spatial attention mechanism is shown in the figure;

[0052] Figure 9 The fusion image schematic diagram of the street under different algorithms is shown in the figure;

[0053] Figure 10 The fusion image schematic diagram of the house under different algorithms is shown in the figure;

[0054] Figure 11 The fusion image schematic diagram of the running pedestrian under different algorithms is shown in the figure;

[0055] Figure 12 The fusion image schematic diagram of the outdoor under different algorithms is shown in the figure. DETAILED DESCRIPTION

[0056] The application will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

[0057] In view of the problems of information redundancy, artifacts and distortion existing in the current infrared and visible light image fusion algorithm, the application proposes an image fusion algorithm based on multi-feature extraction.

[0058] Firstly, the adaptive gradient feature extraction module containing the gradient operator is used to perceive the subtle texture changes in the image and calculate the gradient information of the image in different directions, and then the shallow features containing rich edge information are obtained.

[0059] Then, the residual gradient extraction module with residual connection is used to make the shallow features in different scales to fuse the information, so as to generate deep features.

[0060] Next, the shallow features are combined with the deep features through spatial channel reconstruction convolution SCConv, so that the final features not only retain detailed information, but also integrate higher-level semantic features.

[0061] Finally, the attention mechanism CBAM is introduced during feature reconstruction to improve the efficiency of extracting salient information.

[0062] Experiments on the TNO dataset show that the images fused by this invention have clearer contours, more distinct targets, and significantly improved metrics.

[0063] (I) Constructing a multi-feature extraction network

[0064] Depend on Figure 1 As can be seen, the multi-feature extraction network of this invention mainly consists of a fusion network (RFEFN, Residual Feature Extraction Fusion Network), a semantic segmentation network, and content loss functions and semantic loss functions. Vis represents a visible light image, and Ir represents an infrared light image.

[0065] First, infrared and visible light images are input into a fusion network to generate a fused image. Then, the fused image is segmented using a segmentation network. The segmentation results are compared with semantic labels to determine the richness of semantic information contained in the fused image. The purpose of the fusion network is to integrate background information from the infrared image and texture information from the visible light image; therefore, a content loss function is designed to make the fused image more closely approximate the source images. Furthermore, to enable the fused image to be used for advanced visual tasks, a semantic loss function is constructed to represent the image's contribution to the advanced visual task. Therefore, the multi-feature extraction network of this invention can better adapt to image fusion for various tasks.

[0066] like Figure 2 The diagram shows the structure of the fusion network (RFEFN), which is mainly divided into two parts: feature extraction and feature reconstruction. The feature extraction part consists primarily of convolutional layers with activation functions and 1×1 and 3×3 kernels, as well as spatial channel reconstruction convolutions (SCConv), adaptive gradient feature extraction (AGFE), and residual gradient extraction (RGE). The feature reconstruction part consists of an introduced attention mechanism (CBAM), three cascaded 3×3 convolutional layers, and one 1×1 convolutional layer. The 3×3 convolutional layers all use LReLU as the activation function, while the 1×1 convolutional layer uses Tanh as the activation function. In this embodiment, the feature extraction input I... vis For visible light images, I ir This is an infrared image.

[0067] likeFigure 2 In order to solve the problems of gradient disappearance, gradient explosion and fusion image artifacts, the residual connection and cross feature extraction are adopted in the feature extraction part.

[0068] Firstly, the infrared image and the visible light image are input into a convolution layer with a convolution kernel size of 1*1, so as to optimize the channel dimension, network parameters and load, and provide sufficient feature pattern samples for subsequent extraction of shallow layer features. Then, the infrared and visible light feature maps after convolution are cross input into a convolution layer with a convolution kernel size of 3*3. Compared with the traditional double-branch feature extraction, the cross convolution can not only fuse the features of different modalities in the early stage, but also utilize the complementary information of the two modalities in the initial stage, so as to improve the feature extraction effect, and reduce the number of network parameters and the calculation complexity. Then, the extracted features are respectively input into an adaptive gradient feature extraction module (AGFE) and a residual gradient extraction module (RGE), so as to deeply extract texture and background information. Finally, multi-scale feature extraction is carried out through a spatial channel reconstruction convolution (SCConv), and the features are spliced and combined for feature reconstruction.

[0069] Figure 3 The structure diagram of the AGFE module is shown in the figure. The input tensor is firstly input into three flat branches. Each layer branch includes a Zpool layer, a convolution layer and a Sigmoid function activation layer. The first two layer branches rotate the input tensor on the H axis and the W axis respectively, and then the shape (HxCxW) of the tensor is transformed into (2xCxW) through the Zpool layer. Then, the convolution layer with batch normalization is used to obtain the intermediate dimension (1xCxW) for synthesizing the attention weight. Finally, the Sigmoid function is used to generate the attention weight and is used for the input tensor. In order to keep the shape of the tensor unchanged, the rotated tensor is rotated counterclockwise again. For the third layer, the input tensor is directly input into the Zpool layer, the convolution layer and the Sigmoid function to obtain the spatial weight and is used for the input tensor. Finally, the results of the three branches are added and averaged. The obtained feature map is input into a dense connection network with a convolution kernel of 3*3, and the Sobel gradient operator is introduced to perform fine-grained extraction on the image. Finally, a convolution with a convolution kernel of 1*1 is used for channel dimension reduction, and the features are spliced. In order to better extract the fine-grained features of the image, the source feature map is further connected through a 1*1 convolution layer, and the outputs of the main dense flow and the residual gradient flow are added through element-by-element addition, so as to integrate the deep features and the fine-grained detailed features.

[0070] In the application, the AGFE module not only considers the lack of information interaction in the extraction of feature channel dimension and spatial dimension, but also realizes cross-dimension interaction of triple attention to obtain important parts of the image, fully extracts fine-grained features of the image, and makes the texture information of the fused image more sufficient.

[0071] Spatial Channel Reconstruction Module SCConv Figure 4 As shown, to utilize the spatial redundancy of features, a Spatial Reconstruction Unit (SRU) is introduced. The SRU achieves this through separation and reconstruction operations. The separation operation aims to separate information-rich feature maps from those with less spatial content. The information content of different feature maps is evaluated using a scaling factor in the Group Normalization (GN) layer. Specifically, given an intermediate feature map X∈R... N×C×H×W Where N is the batch dimension, C is the channel dimension, and H and W are the spatial height and width dimensions, respectively. The input feature X is standardized by subtracting the mean μ and dividing by the standard deviation σ. To utilize the channel redundancy of the features, a Channel Reconstruction Unit (CRU) is introduced, employing a split-transform-fusion strategy. For the intermediate input feature X in the bottleneck residual block, spatial refinement features are first obtained through SRU operations, and then channel refinement features Y are obtained using CRU operations.

[0072] RGE module, such as Figure 5 As shown, to prevent overfitting and gradient explosion, the three branch channels are convolved, summed, and then a linear rectified activation function is applied. The top branch channel uses a 5×5 convolutional kernel for edge feature extraction, followed by a 1×1 convolution to change the output dimension of the feature map. The middle branch channel uses two 3×3 kernels and one 1×1 convolution to extract and fuse texture information from the feature map and change the channel dimension. The bottom branch channel uses a skip-connected identity mapping convolutional layer, called the identity conv, with a 1×1 kernel. Except for the last 1×1 convolution of each channel, the remaining convolutional layers all have a linear rectified activation function. Summing the outputs of the three branch channels ensures that the fused image retains the salient features and texture details of the source image. Compared to traditional convolutional networks for feature extraction, it also fully utilizes the information from all feature maps to achieve semantic information sharing and retain more useful details.

[0073] Depend on Figure 2 As can be seen, the feature reconstruction part consists of three cascaded 3×3 convolutional layers and one 1×1 convolutional layer. The 3×3 convolutional layers all use LReLU as the activation function, while the 1×1 convolutional layer uses Tanh as the activation function. Information loss is a serious problem in image fusion tasks. Therefore, in addition to using 1×1 convolutional layers, the feature reconstruction network also uses the same padding settings and sets the stride to 1. Since no downsampling is introduced, the size of the fused image remains consistent with the original image. Furthermore, a CBAM attention mechanism is introduced in the feature reconstruction part. Before image reconstruction, the image is fed into this attention mechanism to ensure that the fusion result focuses more on important semantic information. The CBAM attention mechanism structure diagram is as follows. Figure 6 As shown.

[0074] CBAM is composed of two independent sub-modules: Channel Attention Module (CAM) and Spatial Attention Module (SAM), which perform attention mechanism in channel and space respectively. This design not only saves parameters and computing resources, but also ensures that CBAM can be integrated into existing network architectures as a plug-and-play module.

[0075] As shown in Figure 7 , the input feature map is first passed through two parallel MaxPool layers and AvgPool layers, so that the size of the feature map changes from CxHxW to Cx1x1. Then, the feature map is transmitted into a shared MLP module, which mainly compresses the channel number to 1 / r (reduction rate) of the original, then expands it back to the original channel number, and obtains two activated results through the ReLU activation function. Subsequently, the two results are added element by element, and the output of channel attention is generated through the Sigmoid activation function. Finally, this output is multiplied with the original feature map to restore the size to CxHxW.

[0076] Figure 8 The structure diagram of spatial attention mechanism. First, the output result of Channel Attention is passed through maximum pooling and average pooling to obtain two feature maps with a size of 1xHxW. Then, the feature maps are spliced together through the Contact splicing operation, and they are input into a 7x7 convolution layer. The convolution layer outputs a single-channel feature map, which is then passed through the Sigmoid activation function to obtain the spatial attention feature map. Finally, the spatial attention feature map is multiplied with the original feature map to restore the size to CxHxW.

[0077] In order to improve the semantic information in the fused image while improving the visual quality of the image, the loss function of the present application is composed of a content loss function and a semantic loss function: the content loss function is used to fully integrate the semantic information of the source image in the fusion network, i.e. the background information of the infrared image and the texture information of the visible light image, and the semantic loss function is used to constrain the contribution of the fused image to high-level visual tasks.

[0078] In order to make the fusion model integrate more meaningful information and improve the visual quality and quantitative indicators, the present application designs a content loss function. The content loss is composed of intensity loss l int and texture loss l texture Two parts. Its specific definition is as follows:

[0079] l content =l int +αl texture (1)

[0080] where l int is the intensity loss for constraining the overall intensity of the fused image, l texture indicates that it can encourage the fused image to contain more detailed texture details. The parameter a is used to find a balance between the intensity loss l int and the texture loss l texture . The intensity loss is used to measure the difference between the fused image and the source images at the pixel level. Therefore, the intensity loss of the infrared image and the visible light image is defined as:

[0081]

[0082] where H, W represent the height and width of the image respectively, ||.||1 represents the l1 norm, and max(,) represents the element-wise maximum selection.

[0083] By adopting the maximum selection strategy, the pixel intensity distribution of the infrared image and the visible light image is combined, and then the integral distribution is used to limit the pixel intensity distribution of the fused image. In order to achieve the final fused image, not only the best intensity distribution can be maintained, but also the rich texture details in the source images can be preserved. However, the intensity loss only provides a rough distribution constraint for model learning. Therefore, the texture loss is introduced to ensure that the fused image contains more detailed texture information. The definition of the texture loss is as follows:

[0084]

[0085] where, is the Sobel gradient operator used to measure the fine-grained texture information of the image. |. | represents the absolute value operator, and the best texture of the fused image is the maximum combination of the textures of the infrared image and the visible light image.

[0086] In summary, the content loss can effectively ensure that the model achieves the first goal, that is, to improve the visual quality and statistical evaluation index of the fused image.

[0087] In order to fully enhance the semantic information of the fused image, the invention designs a semantic loss to achieve this goal. The semantic loss includes two parts, the main semantic loss and the auxiliary semantic loss. More specifically, a real-time semantic segmentation model is introduced to segment the fused image, and the segmentation network outputs the segmentation result I s ∈R H×W×C and the auxiliary segmentation result I sa ∈R H×W×CThe segmentation network outputs a segmentation result, which is mainly used to complete the overall image segmentation task, and is the final segmentation image generated by the network model. The auxiliary segmentation result refers to the auxiliary output result generated by the segmentation network in the training process to help the network learn and optimize better. The main semantic loss and the auxiliary semantic loss are defined as:

[0088]

[0089] wherein, L so ∈R H×W×C represents a one-hot vector converted from the segmentation symbol L s ∈(1,C) H×W×C .

[0090] The main semantic loss and the auxiliary semantic loss reflect the semantic information contained in the fusion image from different angles.

[0091] Finally, the semantic loss is represented as:

[0092] l semantic =l main +λl aux (6)

[0093] wherein, λ is a constant for balancing the main semantic loss and the auxiliary semantic loss.

[0094] In order to maximize the performance of the network, λ is selected as 0.01, 0.05, 0.1, 0.25, and 0.5 for testing in this embodiment, and the mloU index of the network is tested. When λ = 0.1 is set, the network can obtain 76.0% of mloU. Compared with the baseline network, this is a significant improvement. Therefore, in the following experiments, λ = 0.1 will be adopted. It is worth noting that in addition to the constraint fusion network, the semantic loss is also used to train the segmentation model. Finally, a joint loss is constructed to guide the training of the fusion model, which is defined as:

[0095] l joint =l content +βl semantic (7)

[0096] wherein, β is a hyperparameter, which is used to represent the importance of the semantic loss function l semantic .

[0097] It is emphasized that as the fusion model adapts, the segmentation network will also adjust accordingly in the training process. Therefore, according to the joint adaptive training strategy, the time delay will gradually increase with the low-level and high-level adaptation.

[0098] (II) Experimental results and analysis

[0099] Ablation experiments are conducted on the introduced attention mechanism and the designed modules, fully proving the feasibility of the application. In addition, five algorithms published in recent years are compared, and detailed comparative analysis is carried out through experiments. The five algorithms are: gradient transfer and total variation minimization fusion algorithm (GTF), unsupervised misregistration infrared and visible image fusion based on cross-modal image generation and registration (UMF-CMGR), the first image fusion method combined with advanced visual tasks (SeaFusion), a novel fusion method combining convolutional layer fusion layer and dense block (DenseFuse), and unified unsupervised image fusion network (U2Fusion). The evaluation indexes include information entropy (EN), spatial frequency (SF), average gradient (AG), correlation coefficient (CC), visual fidelity (VIF), mean square error (MSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0100] The TNO dataset contains rich military scenes such as helicopters, houses, tanks, people, forests and vehicles, so it is very suitable for infrared and visible image fusion research, and is one of the most authoritative datasets in this field. Through the method of expanding the dataset, 45000 pairs of infrared and visible light images are obtained for the training set. In order to ensure the authority of the test results, 42 pairs of images are selected from the TNO dataset as the test set, and the test results are averaged. The algorithm building environment is windows 11, the training environment is NVIDIA RTX 2080TI GPU, the initial learning rate is 0.001, the batch size and epoch are 1 and 10 respectively.

[0101] In order to verify the effectiveness of the residual gradient extraction module RGE, the adaptive gradient feature extraction module AGFE and the cross feature extraction, as well as the effect of the introduced module on the fusion of the application, the modules RGE, AGFE, CBAM, SCConv and the cross feature extraction method are verified through ablation experiments. The results are shown in Table 1.

[0102] Table 1 Ablation experiment results

[0103]

[0104] The black font in Table 1 indicates the best data for this index. It can be seen that the RGE module has significantly improved the eight indexes, especially the structural similarity SSIM, which has been improved by 5.3%. Although the AGFE module and the cross feature extraction have some negative effects on the spatial frequency (SF), they have greatly improved the other seven indexes. Overall, the attention mechanism introduced in the application and the designed fusion network structure play an important role in improving the image fusion quality.

[0105] Figure 9 This is a street photograph selected from the TNO dataset, which includes not only infrared and visible light images, but also the resulting image after fusion using the five algorithms mentioned above and the algorithm proposed in this invention. Figure 9 It is quite evident that while infrared images effectively highlight target features such as pedestrians, cars, and traffic lights, their texture information is very blurry, making it difficult to discern detailed features. Visible light images clearly show the English letters on the shop roofs, and the background texture information is well preserved, but the recognition of other target features is very low.

[0106] Depend on Figure 9 It can be seen that there are significant differences between the five comparison algorithms and the algorithm proposed in this invention. The GTF algorithm can display the English letters on the shop ceiling, but it lacks texture information and the background is dark, resulting in poor detail display and preserving more background information from the infrared image. The UMF-CMGR algorithm retains rich texture information and can clearly see details such as letters and lights on the ceiling, but the background information and target outlines are blurred. Under the SeaFusion algorithm, the overall image is dark, the letters on the ceiling are blurred, and the lights are dim, but the background information is still discernible. The DenseFuse algorithm retains good background information, but lacks detail, and the letters on the ceiling are not clear. The U2Fusion algorithm has a dark background, loses a lot of background information such as traffic lines, vehicles, and people, and has ghosting, but the details of the letters on the ceiling are complete. The image fused by this invention retains good background texture information. Target information such as letters on the ceiling, traffic lines, pedestrians, vehicles, and streetlights are clear, without ghosting, and the background is brighter and clearer. Overall, this invention performs excellently in extracting image texture and background information.

[0107] Figure 10 This image shows fused images of houses obtained using different algorithms. It compares the results of five different algorithms and the fusion method described in this invention. Figure 10 It is evident that the images fused by the GTF, UMF-CMGR, and DenseFuse algorithms suffer from a darkened, blurred background and significant loss of texture information, exhibiting insufficient structural similarity compared to visible light images. The SeaFusion fusion result is overexposed, leading to loss of detail. In the U2Fusion algorithm-fused image, the target information of the houses is well highlighted, but the background is blurred, and the rooftops and the grass in front of the houses are too dim, resulting in severe loss of texture information. In contrast, the image fused by this invention not only has sufficiently clear outlines but also retains a relatively high level of texture information for the houses and the background in front of them.

[0108] To more fully illustrate the broad applicability of this invention, two different algorithm fusion graphs from the TNO dataset were selected for comparison, as shown below. Figure 11 and Figure 12The results are shown in the figure. The results can be better compared from the subjective level, thereby proving that the application has certain advantages in improving the fusion quality, retaining semantic information and highlighting target information.

[0109] To better prove the effectiveness of the application, 42 different infrared images and visible light images were selected from the TNO data set, fused under five contrast algorithms, and eight indicators were selected for evaluation. The differences between the fused images were finally evaluated by taking the average value. Such experimental comparison makes the evaluation result more objective, greatly eliminates the subjective factors of people, and makes the result more reliable.

[0110] As shown in Table 2, the best data of the experimental results are also represented in bold. It can be seen that the application has improved in five of the eight evaluation indicators compared with the other five algorithms, and the data of the remaining three indicators are also higher than the comprehensive average. In addition, SF and VIF have also greatly improved compared with the second-ranked data. Therefore, the application has better superiority based on the objective evaluation of the experiment on the TNO data set.

[0111] Table 2 Fusion effect evaluation of TNO data set

[0112]

[0113] In summary, the application extracts multiple features of the source image through the fusion network MFEFN of the fusion adaptive gradient feature extraction module AGFE and the residual gradient extraction module RGE. This multiple feature extraction method greatly improves the image fine-grained feature capturing ability and information retention ability of the fusion network, ensuring the high quality and high distinguishability of the fused image. In addition, in view of the possible semantic information loss problem in the fusion process, the CBAM attention mechanism and the spatial channel reconstruction convolution SCConv which can replace the traditional convolution are introduced, so that the fused image greatly retains more texture details of the source image. Finally, through subjective and objective evaluation, it is proved that compared with the classical algorithm, the application can effectively realize the maintenance of the intensity of the salient target and the preservation of the texture details in the complex background.

Claims

1. A method for fusing infrared and visible light based on multiple feature extraction, characterized in that, An adaptive gradient feature extraction module with gradient operators is used to detect subtle texture changes in an image and calculate gradient information in different directions to obtain shallow features with rich edge information. A residual gradient extraction module with residual connections is used to fuse the shallow features at different scales to generate deep features. The shallow features are combined with the deep features through spatial channel reconstruction convolution. Infrared and visible light images are input into the multi-feature extraction network to achieve real-time fusion of infrared and visible light images. The multi-feature extraction network consists of a fusion network, a semantic segmentation network, a content loss function, and a semantic loss function. The fusion network consists of a feature extraction part and a feature reconstruction part. The feature extraction part includes convolutional layers with activation functions, an adaptive gradient feature extraction module, a spatial channel reconstruction convolution, and a residual gradient extraction module. The feature reconstruction part consists of an attention mechanism and four cascaded convolutional layers. The implementation steps are as follows: S1, Infrared and visible light images are input into the fusion network, and a fused image is generated through feature extraction and feature reconstruction; S2, the fused image is passed to the segmentation network for segmentation; S3 compares the segmentation results with the semantic labels to evaluate the richness of semantic information contained in the fused image.

2. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The steps for feature extraction are as follows: S11, input the infrared image and the visible light image into a convolutional layer with a kernel size of 1×1; S12, cross-input the convolutional infrared and visible light feature maps into a convolutional layer with a 3×3 convolutional kernel; S13, the extracted features are further extracted through the adaptive gradient feature extraction module and the residual gradient extraction module to extract texture and background information; S14 extracts multi-scale features through spatial channel reconstruction convolution and then splices and combines them for feature reconstruction.

3. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The adaptive gradient feature extraction module comprises three parallel branches. Each branch includes a Z-pooling layer, a convolutional layer, and a sigmoid activation layer. The first two branches rotate the input tensor along the H-axis and W-axis, respectively. Then, the Z-pooling layer transforms the tensor's shape from H×C×W to 2×C×W. After passing through a convolutional layer with batch normalization, an intermediate-dimensional tensor of 1×C×W is obtained for synthesizing attention weights. Finally, the sigmoid function generates attention weights, which are then used as input tensors. The rotated tensor is then rotated counterclockwise by one... Next, for the third branch, the input tensor is directly passed through the Zpool layer, convolutional layer, and Sigmoid function to obtain spatial weights, which are then used as input tensors. Finally, the results of the three branches are summed and averaged. The resulting feature map is then fed into a densely connected network with a 3×3 convolutional kernel, and the Sobel gradient operator is introduced to perform fine-grained extraction of the image. Then, a 1×1 convolutional kernel is used for channel dimensionality reduction, and the features are concatenated. The original feature map is passed through a 5×5 convolutional layer with residual connections. Finally, the outputs of the main dense flow and the residual gradient flow are added element-wise.

4. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The spatial channel reconstruction convolution utilizes spatial redundancy of features by introducing spatial reconstruction units. By introducing a channel reconstruction unit and utilizing the channel redundancy of features, the image channel is divided into multiple units, and each unit is fused after standard conversion.

5. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The residual gradient extraction module consists of three branch channels. The top branch convolutional channel uses a 5×5 convolutional kernel to extract edge features, followed by a 1×1 convolution to change the output dimension of the feature map. The middle branch channel uses two 3×3 convolutional layers and a 1×1 convolution to extract texture information from the feature map and change the channel dimension. The bottom branch channel uses a skip-connected identity mapping convolutional layer with a 1×1 kernel. Except for the last 1×1 convolution of each channel, the remaining convolutional layers all have linear rectified unit activation functions. Finally, the outputs of the three branch channels are summed.

6. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The attention mechanism consists of a channel attention module and a spatial attention module connected in sequence, which execute the attention mechanism in the channel dimension and the spatial dimension, respectively; the processing of the feature map is as follows: The input feature map first passes through two parallel MaxPool and AvgPool layers, reducing its size from C×H×W to C×1×1. Next, the feature map is fed into a shared MLP module. The MLP module first reduces the number of channels to 1 / r of the original number, then expands it back to the original number of channels, and obtains two activated outputs using the ReLU activation function. These two outputs are then added element-wise, and the final channel attention feature map is generated using the Sigmoid activation function. Finally, the final channel attention feature map is multiplied point-by-point with the original feature map to restore the size to C×H×W.

7. The infrared and visible light fusion method based on multiple feature extraction according to claim 6, characterized in that, The spatial attention module processes the feature map as follows: First, the output of the channel attention is processed in the spatial dimension using max pooling and average pooling operations, resulting in two feature maps of size 1×H×W. Subsequently, the features are concatenated and fed into a 7×7 convolutional layer. The single-channel feature map generated by this convolutional layer is then activated by the Sigmoid activation function to obtain the spatial attention feature map. Finally, the obtained spatial attention feature map is multiplied by the original feature map to restore the original C×H×W size.

8. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The content loss function includes intensity loss l int and texture loss texture The definition is as follows: l content ＝l int +αl texture Among them, I vis For visible light images, I ir Infrared image; intensity loss l int It can constrain the overall intensity of the fused image, and reduce texture loss. texture It can enable the fused image to contain more fine texture details; α is used to achieve an effective balance between intensity loss and texture loss; H and W represent the height and width of the image, respectively; ||.||1 represents the l1 norm; max(,) represents the element-wise maximum selection. is the Sobel gradient operator, used to measure fine-grained texture information of an image; |.| represents absolute value operation.

9. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that, The semantic loss includes the main semantic loss. main and auxiliary semantic loss aux The definition is as follows: Among them, L so ∈R H×W×C Indicated by the separator L s ∈(1,C) H×W×C The transformed single-heat vector; Ultimately, semantic loss l semantic Represented as: the semantic =l main +λl aux Where λ is a constant that balances the main semantic loss and the auxiliary semantic loss.

Images