Infrared and visible light fusion method based on multi-feature extraction

Abstract

The invention discloses an infrared and visible light fusion method based on multi-feature extraction. A multi-feature extraction network is composed of a fusion network, a semantic segmentation network, a content loss function and a semantic loss function. The fusion network comprises a feature extraction part and a feature reconstruction part, and the feature extraction part comprises a convolutional layer with an activation function, an adaptive gradient feature extraction module and a space channel reconstruction convolution and residual gradient extraction module; the feature reconstruction part is composed of an attention mechanism and four convolution layers connected in series; the method comprises the following steps: S1, inputting an infrared image and a visible light image into a fusion network, and generating a fusion image through feature extraction and feature reconstruction; s2, transmitting the fused image to a segmentation network for segmentation; and S3, comparing the segmentation result with a semantic tag, and evaluating richness of semantic information contained in the fused image. The contour of the fused image is clearer, the target is clearer, and the index is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an infrared and visible light fusion method based on multiple feature extraction. Background Art

[0002] Infrared images have the advantages of strong penetration and are not affected by light, but lack texture information. Visible light images have rich structural information and good visual effects, but are easily affected by weather and light conditions and have poor anti-interference ability. Therefore, a method to fuse the two has emerged.

[0003] Image fusion methods are mainly divided into two categories: traditional fusion methods and deep learning-based fusion methods. Traditional fusion methods mainly include multi-scale transformation, sparse representation, saliency fusion, subspace decomposition, hybrid models, etc. At present, infrared and visible light image fusion methods based on deep learning mainly include: autoencoder (AE)-based method, convolutional neural network (CNN)-based method, and generative adversarial neural network (GAN)-based method. Tang et al. (TANG LF, YUAN JT, MA JY. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network [J]. Information Fusion, 2022, 82: 28-42) combined image fusion problems with semantic segmentation problems and designed a semantic-aware fusion framework SeAFusion. The GAN network mainly includes a generator and a discriminator. By formulating a loss function to constrain the adversarial game between the two, the fused image is closer to the original image. Ma et al. (MA JY, ZHANG H, SHAO ZF, et al. GANMcC: A Generative Adversarial Network With Multiclassification Constraints for Infrared and Visible Image Fusion [J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14) first introduced GAN into image fusion, but the fused image obtained by single adversarial training was too smooth and blurry and the details were unstable, so intensity loss and gradient loss were introduced. Although current neural networks are already able to perform end-to-end image tasks, the fusion method still has some shortcomings:

[0004] 1) The fused image is not clear, the background is blurred, and there is a lot of redundant information;

[0005] 2) The original image is ignored and the image reconstruction is overly dependent on the feature map, resulting in structural differences between the fused image and the original visible light image;

[0006] 3) When the background is complex, the fused image is prone to lose texture information. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to provide an infrared and visible light fusion method based on residual feature extraction to improve the ability of the fusion network to extract fine-grained features of the image, to highlight the target information and to retain the texture information.

[0008] Technical solution: A method for fusion of infrared and visible light based on multiple feature extraction, which uses an adaptive gradient feature extraction module containing a gradient operator to sense subtle texture changes in the image and calculate the gradient information of the image in different directions, thereby obtaining shallow features containing rich edge information; through a residual gradient extraction module with a residual connection, shallow features are fused at different scales to generate deep features; shallow features are combined with deep features through spatial channel reconstruction convolution SCConv, and infrared images and visible light images are input into a multi-feature extraction network to achieve real-time fusion of infrared images and visible light images;

[0009] The multi-feature extraction network consists of a fusion network, a semantic segmentation network, a content loss function and a semantic loss function;

[0010] The fusion network includes a feature extraction part and a feature reconstruction part. The feature extraction part includes a convolution layer with an activation function, an adaptive gradient feature extraction module, a spatial channel reconstruction convolution, and a residual gradient extraction module; the feature reconstruction part consists of an attention mechanism and four convolution layers in series.

[0011] The implementation steps are as follows:

[0012] S1, input the infrared image and the visible light image into the fusion network, and generate the fused image through feature extraction and feature reconstruction;

[0013] S2, passing the fused image to the segmentation network for segmentation;

[0014] S3, compares the segmentation results with the semantic labels to evaluate the richness of the semantic information contained in the fused image.

[0015] Furthermore, the feature extraction part performs the following steps to implement feature extraction:

[0016] S11, input the infrared image and the visible light image into a convolution layer with a convolution kernel size of 1×1;

[0017] S12, cross-input the convolved infrared and visible light feature maps into the convolution layer of the 3×3 convolution kernel;

[0018] S13, the extracted features are passed through an adaptive gradient feature extraction module and a residual gradient extraction module to further extract texture and background information;

[0019] S14, multi-scale features are extracted through spatial channel reconstruction convolution, and they are spliced ​​and combined for feature reconstruction.

[0020] Furthermore, the adaptive gradient feature extraction module includes three flat branches, each of which includes a Zpool layer, a convolution layer and a Sigmoid function activation layer; the first two branches rotate the input tensor on the H axis and W axis respectively, and then transform the shape of the tensor H×C×W into 2×C×W through the Zpool layer, and then pass through the convolution layer with batch normalization to obtain the intermediate dimensional tensor 1×C×W for synthesizing attention weights, and finally generate attention weights through the Sigmoid function and use them for the input tensor, and then rotate the rotated tensor counterclockwise. Transform once; for the third layer branch, the input tensor is directly passed through the Zpool layer, the convolution layer and the Sigmoid function to obtain the spatial weight and use it as the input tensor; finally, the results of the three branches are added and the average is taken; the obtained feature map is then sent to a densely connected network with a convolution kernel of 3×3, and the Sobel gradient operator is introduced to perform fine-grained extraction of the image, and then a convolution with a convolution kernel of 1×1 is used for channel dimensionality reduction and splicing; among them, the original feature map is residually connected through a 5×5 convolution layer, and finally the outputs of the main dense flow and the residual gradient flow are added by element-by-element addition.

[0021] Furthermore, the spatial channel reconstruction module utilizes the spatial redundancy of features by introducing a spatial reconstruction unit; and utilizes the channel redundancy of features by introducing a channel reconstruction unit, thereby dividing the image channel into multiple units, and each unit is fused after standard transformation.

[0022] Furthermore, the residual gradient extraction module includes three branch channels. The top branch convolution channel uses a 5×5 convolution kernel to extract edge features, and then changes the output dimension of the feature map through a convolution with a 1×1 convolution kernel; the middle branch channel uses two convolution layers with 3×3 convolution kernels and a 1×1 convolution to extract texture information from the feature map and change the channel dimension; the bottom branch channel consists of a jump-connected identity mapping convolution layer with a convolution kernel size of 1×1; except for the last 1×1 convolution of each layer, the remaining convolution layers all have linear rectifier unit activation functions; finally, the outputs of the three branch channels are added together.

[0023] Furthermore, the attention mechanism is composed of a channel attention module and a spatial attention module connected in series, which respectively perform attention mechanisms in the channel dimension and the spatial dimension; the processing process of the feature map is as follows:

[0024] The input feature map first passes through two parallel MaxPool layers and AvgPool layers, so that the size of the feature map changes from C×H×W to C×1×1; then, the feature map is passed into a shared MLP module, which first reduces the number of channels to 1 / r times the original number, and then expands it back to the original number of channels, and obtains two activated outputs through the ReLU activation function; then, the two outputs are added element by element, and the final channel attention feature map is generated through the Sigmoid activation function; finally, the final channel attention feature map is multiplied point by point with the original feature map to restore it to the size of C×H×W.

[0025] Furthermore, the spatial attention module processes the feature map as follows:

[0026] First, the output of channel attention is processed in the spatial dimension using maximum pooling and average pooling operations, and two feature maps of size 1×H×W are obtained;

[0027] Then, they are concatenated and sent to a 7×7 convolutional layer. The single-channel feature map generated by the convolutional layer is activated by a Sigmoid function to obtain a spatial attention feature map.

[0028] Finally, the obtained spatial attention feature map is multiplied with the original feature map to restore it to its original size of C×H×W.

[0029] Furthermore, the content loss function includes the intensity loss l int and texture loss l texture , defined as follows:

[0030] l content = l int +αl texture

[0031]

[0032]

[0033] Among them, the strength loss l int Can constrain the overall intensity of the fused image, texture loss l texture It can make the fused image contain more delicate texture details; α is used to achieve an effective balance between intensity loss and texture loss; H and W represent the height and width of the image respectively, .1 represents the l1 norm, and max(,) represents the maximum selection of each element; is the Sobel gradient operator, which is used to measure the fine-grained texture information of an image; . represents the absolute value operation.

[0034] Furthermore, the semantic loss includes the main semantic loss l main and auxiliary semantic loss laux , defined as follows:

[0035]

[0036]

[0037] Among them, L so ∈R H×W×C Indicated by the segmentation symbol L s ∈(1,C) H×W×C The transformed one-hot vector;

[0038] Finally, the semantic loss l semantic It is expressed as:

[0039] l semantic = l main +λl aux

[0040] Among them, λ is a constant that balances the main semantic loss and the auxiliary semantic loss.

[0041] Compared with the prior art, the present invention has the following significant effects:

[0042] 1. The present invention introduces a segmentation network to perform segmentation processing on the fused image, and guides the training of the fusion network through the constructed semantic loss so that the fused image contains more semantic information; not only an adaptive gradient feature extraction module (AGFE) and a residual gradient extraction module (RGE) are designed in the fusion network to improve the image fusion quality, but also the source images are cross-processed instead of simply splicing the feature maps in the subsequent stage, so as to improve complementarity and prevent information redundancy;

[0043] 2. The present invention designs an adaptive gradient feature extraction module AGFE and a residual gradient extraction module RGE to extract important local features of the source image, thereby improving the ability of the fusion network to extract fine-grained features of the image, and effectively achieving the highlighting of target information and the retention of texture information;

[0044] 3. In order to solve the problem of semantic information loss that may occur in the fusion process, the present invention introduces the CBAM attention mechanism and the spatial channel reconstruction convolution SCConv that can replace the traditional convolution. The source image is residually connected through the spatial channel reconstruction convolution SCConv to ensure the structural similarity of the fusion result, so that the fused image greatly retains more texture details of the source image. Finally, it is proved through subjective and objective evaluation that the present invention can better maintain the target information of infrared images and the detail information of visible light images under complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is the overall architecture diagram of the multi-feature extraction network of the present invention;

[0046] Figure 2 This is a diagram of the converged network architecture;

[0047] Figure 3 This is the structure diagram of the adaptive gradient feature extraction module;

[0048] Figure 4 Reconstruct the convolutional structure map for the spatial channel;

[0049] Figure 5 This is the structure diagram of the residual gradient extraction module;

[0050] Figure 6 This is the structural diagram of the attention mechanism;

[0051] Figure 7 This is the structure diagram of the channel attention mechanism;

[0052] Figure 8 This is the structure diagram of the spatial attention mechanism;

[0053] Fig. 9 Schematic diagram of fused images of streets under different algorithms;

[0054] Fig.10 This is a schematic diagram of the fused images of houses under different algorithms;

[0055] Fig.11 Schematic diagram of fusion images of people running under different algorithms;

[0056] Fig.12 Schematic diagram of outdoor fusion images under different algorithms. DETAILED DESCRIPTION

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] Aiming at the problems of information redundancy, artifacts and distortion existing in the current infrared and visible light image fusion algorithms, the present invention proposes an image fusion algorithm based on multiple feature extraction.

[0059] Firstly, an adaptive gradient feature extraction module containing gradient operators is used to sense subtle texture changes in the image and calculate the gradient information of the image in different directions, thereby obtaining shallow features containing rich edge information.

[0060] Then, through the residual gradient extraction module with residual connection, the shallow features are fused at different scales to generate deep features.

[0061] Then, the shallow features are combined with the deep features through spatial channel reconstruction convolution SCConv, so that the final features not only retain the detail information but also integrate higher-level semantic features.

[0062] Finally, the attention mechanism CBAM is introduced during feature reconstruction to improve the efficiency of extracting significant information.

[0063] Experiments conducted on the TNO dataset show that the image fused by the present invention has clearer contours, clearer targets and significantly improved indicators.

[0064] (I) Building a multi-feature extraction network

[0065] Depend on Figure 1 It can be seen that the multi-feature extraction network of the present invention is mainly composed of a fusion network (RFEFN, Residual feature extraction fusion network), a semantic segmentation network, a content loss function, and a semantic loss function. Vis represents a visible light image, and Ir represents an infrared light image.

[0066] First, the infrared image and the visible light image are input into the fusion network to generate a fused image, and then the fused image is segmented through the segmentation network. The segmentation result is compared with the semantic label to obtain the richness of the semantic information contained in the fused image. The purpose of the fusion network is to integrate the background information in the infrared image and the texture information in the visible light image, so the content loss function (Content Loss) is designed to make the fused image closer to the source image. In addition, in order to allow the fused image to be used for advanced visual tasks, a semantic loss function (Semantic Loss) is constructed to represent the contribution of the image to the advanced visual tasks. Therefore, the multi-feature extraction network of the present invention can better adapt to the image fusion of various tasks.

[0067] like Figure 2The figure shows the structure of the fusion network (RFEFN), which is mainly divided into two parts: feature extraction and feature reconstruction. The feature extraction part is mainly composed of convolution layers with activation functions and convolution kernels of 1×1 and 3×3, as well as spatial channel reconstruction convolution SCConv, adaptive gradient feature extraction module (AGFE) and residual gradient extraction module (RGE). The feature reconstruction part consists of the introduced attention mechanism CBAM and three series-connected 3×3 convolution layers and one 1×1 convolution layer. The 3×3 convolution layers all use LReLU as the activation function, while the activation function of the 1×1 convolution layer is Tanh. In this embodiment, the feature extraction input I vis is the visible light image, I ir Infrared image.

[0068] like Figure 2 In the feature extraction part shown in the figure, in order to solve the problems of gradient disappearance, gradient explosion and fusion image artifacts, residual connection and cross feature extraction are used in the feature extraction part.

[0069] First, the infrared image and visible light image are input into a convolution layer with a convolution kernel size of 1×1 to optimize the channel dimension, network parameters and load, and provide sufficient feature map samples for the subsequent shallow feature extraction. Then, the convolved infrared and visible light feature maps are cross-input into the convolution layer with a 3×3 convolution kernel. Compared with the traditional two-branch feature extraction, cross convolution can not only fuse the features of different modalities in the early stage, but also use the complementary information of the two modalities in the initial stage to improve the feature extraction effect, while reducing the number of network parameters and computational complexity. Then, the extracted features are input into the adaptive gradient feature extraction module (AGFE) and the residual gradient extraction module (RGE) respectively to further extract texture and background information. Finally, multi-scale feature extraction is performed through spatial channel reconstruction convolution (SCConv), and the splicing and combination are combined for feature reconstruction.

[0070] Figure 3The following is the structure diagram of the AGFE module. The input tensor first passes through three flat branches. Each branch includes a Zpool layer, a convolution layer, and a Sigmoid function activation layer. The first two branches rotate the input tensor on the H and W axes respectively, and then transform the shape of the tensor (H×C×W) into (2×C×W) through the Zpool layer. After that, it passes through the convolution layer with batch normalization to obtain the intermediate dimension (1×C×W) for synthesizing attention weights. Finally, the attention weights are generated by the Sigmoid function and used for the input tensor. In order to keep the tensor unchanged from the original shape, the rotated tensor is rotated counterclockwise again. For the third layer, the input tensor is directly passed through the Zpool layer, convolution layer, and Sigmoid function to obtain the spatial weight and use it as the input tensor. Finally, the results of the three branches are added and averaged. The obtained feature map is then input into a densely connected network with a convolution kernel of 3×3, and the Sobel gradient operator is introduced to extract fine-grained images. Finally, a convolution with a convolution kernel of 1×1 is used to reduce the channel dimension and splice them. In order to better extract the fine-grained features of the image, the source feature map is then connected through a 1×1 convolutional layer residual connection, and the outputs of the main dense flow and the residual gradient flow are added by element-by-element addition to integrate the deep features and fine-grained detail features.

[0071] In the present invention, the AGFE module not only takes into account the lack of information interactivity in the feature channel dimension and the spatial dimension, but also realizes cross-dimensional interaction of triple attention to obtain the important parts of the image, fully extracts the fine-grained features of the image, and makes the texture information of the fused image more complete.

[0072] The spatial channel reconstruction module SCConv is as follows Figure 4 As shown in Figure 1, in order to exploit the spatial redundancy of features, a spatial reconstruction unit (SRU) is introduced. SRU achieves this goal through separation and reconstruction operations. The separation operation aims to separate feature maps with rich information from feature maps with less spatial content. The information content of different feature maps is evaluated by the scaling factor in the group normalization (GN) layer. Specifically, given an intermediate feature map X∈R N×C×H×W , where N is the batch dimension, C is the channel dimension, H and W are the spatial height and width dimensions, respectively. The input feature X is normalized by subtracting the mean μ and dividing by the standard deviation σ. In order to exploit the channel redundancy of features, a channel reconstruction unit (CRU) is introduced, which utilizes a split-transform-fusion strategy. For the intermediate input feature X in the bottleneck residual block, the spatially refined features are first obtained by the SRU operation, and then the channel-refined features Y are obtained by the CRU operation.

[0073] RGE module such as Figure 5As shown in the figure, in order to prevent overfitting and gradient explosion, the three branch channels are added after convolution and then a linear rectification function is used. The top branch convolution channel uses a 5×5 convolution kernel to extract edge features, and then changes the output dimension of the feature map through a convolution with a convolution kernel of 1×1. The middle branch channel uses two convolution kernels of 3×3 and one 1×1 to extract and fuse the texture information of the feature map and change the channel dimension. The bottom branch channel consists of a jump-connected identity mapping convolution layer called identity conv with a convolution kernel size of 1×1. Except for the last 1×1 convolution of each channel, the remaining convolution layers are all activated with a linear rectification unit. Adding the outputs of the three branch channels can not only ensure that the fused image has the significant features and texture details of the source image, but also make full use of the information of all feature maps to realize semantic information sharing and retain more useful details compared to the traditional convolutional network feature extraction.

[0074] Depend on Figure 2 It can be seen that the feature reconstruction part consists of three serially connected 3×3 convolutional layers and one 1×1 convolutional layer. The 3×3 convolutional layers all use LReLU as the activation function, while the activation function of the 1×1 convolutional layer is Tanh. Information loss is a serious problem in image fusion tasks. Therefore, in addition to using a 1×1 convolutional layer, the feature reconstruction network also uses the same padding settings and sets the step size to 1. Since no downsampling is introduced, the size of the fused image is consistent with the original image. In addition, the attention mechanism CBAM is introduced in the feature reconstruction part. Before image reconstruction, the image is sent to the attention mechanism to ensure that the fusion result focuses more on important semantic information. The structure diagram of the CBAM attention mechanism is as follows Figure 6 shown.

[0075] CBAM consists of two independent sub-modules: Channel Attention Module (CAM) and Spatial Attention Module (SAM), which perform attention mechanisms on channels and spaces respectively. This design not only saves parameters and computing resources, but also ensures that CBAM can be integrated into existing network architectures as a plug-and-play module.

[0076] like Figure 7As shown in the figure, the input feature map first passes through two parallel MaxPool layers and AvgPool layers, so that the size of the feature map changes from C×H×W to C×1×1. Then, the feature map is passed into a shared MLP module, which mainly compresses the number of channels to 1 / r (reduction rate) times the original number of channels, and then expands it back to the original number of channels, and obtains two activated results through the ReLU activation function. Subsequently, the two results are added element by element, and the channel-attention output is generated through the Sigmoid activation function. Finally, this output is multiplied with the original feature map to restore it to the size of C×H×W.

[0077] Figure 8 Figure 1 is a diagram of the spatial attention mechanism. First, the output of Channel Attention is subjected to maximum pooling and average pooling to obtain two feature maps of size 1×H×W. The feature maps are then concatenated together through the Contact concatenation operation and input into a 7×7 convolutional layer. The convolutional layer outputs a single-channel feature map, which is then activated by the Sigmoid function to obtain the feature map of spatial attention. Finally, this feature map of spatial attention is multiplied with the original feature map to restore it to the size of C×H×W.

[0078] In order to improve the visual quality of the image while improving the semantic information in the fused image, the loss function of the present invention is composed of a content loss function and a semantic loss function: the content loss function is used to fully integrate the semantic information of the source image in the fusion network, namely the background information of the infrared image and the texture information of the visible light image, and the semantic loss function is used to constrain the contribution of the fused image to the high-level visual task.

[0079] In order to enable the fusion model to integrate more meaningful information and improve visual quality and quantitative indicators, the present invention designs a content loss function. The content loss is composed of the intensity loss l int and texture loss l texture It consists of two parts. Its specific definitions are as follows:

[0080] l content = l int +αl texture (1)

[0081] Among them, l int Used to constrain the overall intensity of the fused image, l texture It can make the fused image contain more detailed texture details. The parameter α is used to adjust the intensity loss l int and texture loss l texture The intensity loss is used to measure the difference between the fused image and the source image at the pixel level. Therefore, the intensity loss of the infrared image and the visible light image is defined as:

[0082]

[0083] Among them, H and W represent the height and width of the image respectively, .1 represents the l1 norm, and max(,) represents the maximum selection of each element.

[0084] By adopting the maximum selection strategy, the pixel intensity distributions of the infrared image and the visible light image are combined, and then the integral distribution is used to limit the pixel intensity distribution of the fused image. In order to achieve the final fused image, not only the optimal intensity distribution can be maintained, but also the rich texture details in the source image can be retained. However, the intensity loss only provides a rough distribution constraint for the learning of the model. For this reason, the texture loss is introduced to ensure that the fused image contains more detailed texture information. The definition of texture loss is as follows:

[0085]

[0086] in, is the Sobel gradient operator, which is used to measure the fine-grained texture information of the image. . represents the absolute value operator, and the best texture of the fused image is the maximum combination of the textures of the infrared image and the visible light image.

[0087] In summary, content loss can effectively ensure that the model achieves the first goal, which is to improve the visual quality and statistical evaluation indicators of the fused image.

[0088] In order to fully enhance the semantic information of the fused image, the present invention designs a semantic loss to achieve this goal. The semantic loss includes two parts: the main semantic loss and the auxiliary semantic loss. More specifically, a real-time semantic segmentation model is introduced to segment the fused image, and the segmentation network outputs the segmentation result I s ∈R H×W×C And auxiliary segmentation result I sa ∈R H×W×C The segmentation result output by the segmentation network is mainly used to complete the overall image segmentation task. It is the segmented image finally generated by the network model. The auxiliary segmentation result refers to the auxiliary output result generated by the segmentation network during the training process to help the network learn and optimize better. The main semantic loss and auxiliary semantic loss are defined as:

[0089]

[0090] Among them, L so ∈R H×W×C Indicated by the segmentation symbol L s ∈(1,C) H×W×C The transformed one-hot vector.

[0091] The main semantic loss and auxiliary semantic loss reflect the semantic information contained in the fused image from different perspectives.

[0092] Finally, the semantic loss is expressed as:

[0093] l semantic = l main +λl aux (6)

[0094] Among them, λ is a constant that balances the main semantic loss and the auxiliary semantic loss.

[0095] In order to maximize the performance of the network, this embodiment selected λ as 0.01, 0.05, 0.1, 0.25, and 0.5 for testing, and tested the network's mloU indicator. When λ = 0.1 is set, the network can obtain 76.0% mloU. This is a significant improvement compared to the baseline network. Therefore, in the following experiments, λ = 0.1 will be used. It is worth noting that in addition to constraining the fusion network, semantic loss is also used to train the segmentation model. Finally, a joint loss is constructed to guide the training of the fusion model, defined as:

[0096] l joint = l content +βl semantic (7)

[0097] Among them, β is a hyperparameter used to characterize the semantic loss function l semantic The importance of.

[0098] It is important to emphasize that as the fusion model adapts, the segmentation network will also be adjusted accordingly during the training process. Therefore, according to the joint adaptive training strategy, the latency will gradually increase with low-level and high-level adaptation.

[0099] (II) Experimental results and analysis

[0100] Ablation experiments were carried out on the introduced attention mechanism and the designed modules, which fully demonstrated the feasibility of the present invention. In addition, five algorithms published in recent years were compared, and a detailed comparative analysis was carried out through experiments. The five algorithms are: gradient transfer and total variation minimization fusion algorithm (GTF), unsupervised misaligned infrared and visible light image fusion based on cross-modal image generation and registration (UMF-CMGR), the first image fusion method combining advanced visual tasks (SeaFusion), a novel fusion method combining convolutional layer fusion layer and dense block (DenseFuse), and a unified unsupervised image fusion network (U2Fusion). The evaluation indicators include information entropy (EN), spatial frequency (SF), average gradient (AG), correlation coefficient (CC), visual fidelity (VIF), mean square error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity (SSIM).

[0101] The TNO dataset contains a wealth of military scenes, such as helicopters, houses, tanks, people, forests, and vehicles, so it is very suitable for infrared and visible light image fusion research and is one of the most authoritative datasets in this field. By expanding the dataset, 45,000 pairs of infrared and visible light images were obtained for the training set. In order to ensure the authority of the test results, 42 pairs of images were selected from the TNO dataset as the test set, and the test results were averaged. The algorithm was built in Windows 11, the training environment was NVIDIA RTX 2080TI GPU, the initial learning rate was 0.001, and the batch size and epoch were 1 and 10 respectively.

[0102] In order to verify the effectiveness of the residual gradient extraction module RGE, the adaptive gradient feature extraction module AGFE and the cross-feature extraction, as well as the effect of the introduced modules on the fusion of the present invention, the modules RGE, AGFE, CBAM, SCConv and the cross-feature extraction method are verified by ablation experiments. The results are shown in Table 1.

[0103] Table 1 Ablation experiment results

[0104]

[0105] The bold fonts in Table 1 indicate the data with the best effect on this indicator. It can be seen that the RGE module has significantly improved the eight indicators, especially the structural similarity SSIM, which has increased by 5.3%. Although the AGFE module and cross feature extraction have some negative effects on spatial frequency (SF), they have greatly improved the other seven indicators. Overall, the attention mechanism introduced by the present invention and the designed fusion network structure play an important role in improving the image fusion quality.

[0106] Fig. 9 This is a street photo selected from the TNO dataset, which includes not only infrared images and visible light images, but also the image obtained by fusing the above five algorithms with the algorithm proposed in this invention. Fig. 9 It can be clearly seen that although the infrared image can well highlight the target features including pedestrians, cars and traffic lights, its texture information is very fuzzy and the details cannot be seen clearly. The visible light image can clearly see the English letters on the roof of the shop, and the background texture information is well preserved, but the recognition of other target features is very low.

[0107] Depend on Fig. 9It can be seen that there are obvious differences between the five comparison algorithms and the algorithm proposed in the present invention. The fusion result of the GTF algorithm can display the English letters on the roof of the shop, but lacks texture information and the background is dark, the details are not well displayed, and more background information in the infrared image is retained. The UMF-CMGR algorithm retains rich texture information, and can clearly see details such as the letters and lights on the roof, but the background information and the outline of the target object are blurred. Under the SeaFusion algorithm, the image is dark as a whole, the letters on the roof are blurred, the lights are dim, but the background information is still recognizable. The DenseFuse algorithm retains better background information, but the details are insufficient, and the letters on the roof are not clear. The background of the U2Fusion algorithm is dark, and the background information such as lanes, vehicles and characters is lost and there are ghosting, but the details of the letters on the roof are complete. The image fused by the present invention retains good background texture information. The target information such as the letters on the roof, lanes, pedestrians, vehicles and street lights is clear, there is no ghosting, and the background is brighter and clearer. In general, the present invention performs well in extracting image texture and background information.

[0108] Fig.10 The fused images of the house under different algorithms compare the fused images of the house under five comparison algorithms and the fused images of the method of the present invention. Fig.10 It can be clearly seen that the images fused by the three algorithms of GTF, UMF-CMGR and DenseFuse have a dark background blur and serious loss of texture information. Compared with the visible light image, the structural similarity is not high enough. The SeaFusion fusion result is overexposed, resulting in loss of details. In the result image fused under the U2Fusion algorithm, the target information of the house is well highlighted, but the background is blurred and the brightness of the roof and the grass in front of the house is too low, resulting in serious loss of texture information. In contrast, the picture fused by the present invention not only has a clear enough outline, but also retains the texture information of the house and the background in front of the house more fully.

[0109] In order to more fully illustrate the universality of the present invention, two algorithm fusion diagrams in different scenarios are selected from the TNO dataset for comparison, as shown in Figure 2. Fig.11 and Fig.12 The result graph allows for better comparison from a subjective level, thus proving that the present invention has certain advantages in improving fusion quality, retaining semantic information, and highlighting target information.

[0110] In order to better demonstrate the effectiveness of the present invention, 42 different sets of infrared images and visible light images were selected from the TNO dataset, fused under five contrast algorithms, and 8 sets of indicators were selected for evaluation. The difference of the fused images was finally evaluated by taking the average value. Such experimental comparison makes the evaluation results more objective, and greatly eliminates the subjective factors of humans, making the results more credible.

[0111] As shown in Table 2, the best experimental results are also indicated by bold fonts. It can be clearly seen that compared with the other five algorithms, the present invention has improved five of the eight evaluation indicators, and the data of the remaining three indicators are also higher than the comprehensive average. In addition, SF and VIF are also greatly improved compared with the second-ranked data. Therefore, the present invention has better superiority based on the objective evaluation of the experiment on the TNO dataset.

[0112] Table 2 Evaluation of fusion effect of TNO dataset

[0113]

[0114] In summary, the present invention extracts multiple features of the source image by fusing the fusion network MFEFN of the adaptive gradient feature extraction module AGFE and the residual gradient extraction module RGE. This multiple feature extraction method greatly improves the fusion network's ability to capture fine-grained features of the image and retain information, ensuring the high quality and high recognizability of the fused image. In addition, in order to address the problem of semantic information loss that may occur during the fusion process, the CBAM attention mechanism and the spatial channel reconstruction convolution SCConv that can replace the traditional convolution are introduced, so that the fused image greatly retains more texture details of the source image. Finally, it is proved through subjective and objective evaluation that compared with the classical algorithm, the present invention can effectively maintain the intensity of significant targets and preserve texture details under complex backgrounds.

Claims

1. A method for fusion of infrared and visible light based on multiple feature extraction, characterized in that: An adaptive gradient feature extraction module containing a gradient operator is used to sense subtle texture changes in the image and calculate the gradient information of the image in different directions, thereby obtaining shallow features containing rich edge information. A residual gradient extraction module with a residual connection is used to fuse shallow features at different scales to generate deep features. The shallow features are combined with deep features through spatial channel reconstruction convolution SCConv, and the infrared image and visible light image are input into a multi-feature extraction network to achieve real-time fusion of infrared and visible light images. The multi-feature extraction network consists of a fusion network, a semantic segmentation network, a content loss function and a semantic loss function; The fusion network includes a feature extraction part and a feature reconstruction part. The feature extraction part includes a convolution layer with an activation function, an adaptive gradient feature extraction module, a spatial channel reconstruction convolution, and a residual gradient extraction module; the feature reconstruction part consists of an attention mechanism and four convolution layers in series. The implementation steps are as follows: S1, input the infrared image and the visible light image into the fusion network, and generate the fused image through feature extraction and feature reconstruction; S2, passing the fused image to the segmentation network for segmentation; S3, compares the segmentation results with the semantic labels to evaluate the richness of the semantic information contained in the fused image.

2. The infrared and visible light fusion method based on multiple feature extraction according to claim 1 is characterized in that: The steps for feature extraction in the feature extraction part are as follows: S11, input the infrared image and the visible light image into a convolution layer with a convolution kernel size of 1×1; S12, cross-input the convolved infrared and visible light feature maps into the convolution layer of the 3×3 convolution kernel; S13, the extracted features are passed through an adaptive gradient feature extraction module and a residual gradient extraction module to further extract texture and background information; S14, multi-scale features are extracted through spatial channel reconstruction convolution, and they are spliced ​​and combined for feature reconstruction.

3. The infrared and visible light fusion method based on multiple feature extraction according to claim 1 is characterized in that: The adaptive gradient feature extraction module consists of three flat branches, each of which includes a Zpool layer, a convolution layer, and a Sigmoid function activation layer. The first two branches rotate the input tensor on the H and W axes respectively, and then transform the shape of the tensor H×C×W into 2×C×W through the Zpool layer. After that, it passes through a convolution layer with batch normalization to obtain an intermediate dimensional tensor 1×C×W for synthesizing attention weights. Finally, the attention weights are generated by the Sigmoid function and used for the input tensor, and then the rotated tensor is rotated counterclockwise by one. times; for the third layer branch, the input tensor is directly passed through the Zpool layer, the convolution layer and the Sigmoid function to obtain the spatial weight and use it as the input tensor; finally, the results of the three branches are added and the average is taken; the obtained feature map is then sent to a densely connected network with a convolution kernel of 3×3, and the Sobel gradient operator is introduced to perform fine-grained extraction of the image, and then a convolution with a convolution kernel of 1×1 is used for channel dimensionality reduction and splicing; among them, the original feature map is residually connected through a 5×5 convolution layer, and finally the outputs of the main dense flow and the residual gradient flow are added by element-by-element addition.

4. The infrared and visible light fusion method based on multiple feature extraction according to claim 1 is characterized in that: The spatial channel reconstruction module utilizes the spatial redundancy of features by introducing a spatial reconstruction unit; By introducing the channel reconstruction unit and utilizing the channel redundancy of features, the image channel is divided into multiple units, and each unit is fused after standard transformation.

5. The infrared and visible light fusion method based on multiple feature extraction according to claim 1 is characterized in that: The residual gradient extraction module includes three branch channels. The top branch convolution channel uses a 5×5 convolution kernel to extract edge features, and then changes the output dimension of the feature map through a convolution with a 1×1 convolution kernel. The middle branch channel uses two convolution layers with 3×3 convolution kernels and a 1×1 convolution to extract texture information from the feature map and change the channel dimension. The bottom branch channel consists of a jump-connected identity mapping convolution layer with a convolution kernel size of 1×1. Except for the last 1×1 convolution of each layer, the remaining convolution layers all have linear rectifier unit activation functions. Finally, the outputs of the three branch channels are added together.

6. The infrared and visible light fusion method based on multiple feature extraction according to claim 1 is characterized in that: The attention mechanism is composed of a channel attention module and a spatial attention module connected in series, which perform attention mechanisms in the channel dimension and spatial dimension respectively; the processing process of the feature map is as follows: The input feature map first passes through two parallel MaxPool layers and AvgPool layers, so that the size of the feature map changes from C×H×W to C×1×1; then, the feature map is passed into a shared MLP module, which first reduces the number of channels to 1 / r times the original number, and then expands it back to the original number of channels, and obtains two activated outputs through the ReLU activation function; then, the two outputs are added element by element, and the final channel attention feature map is generated through the Sigmoid activation function; finally, the final channel attention feature map is multiplied point by point with the original feature map to restore it to the size of C×H×W.

7. The infrared and visible light fusion method based on multiple feature extraction according to claim 6 is characterized in that: The spatial attention module processes the feature map as follows: First, the output of channel attention is processed in the spatial dimension using maximum pooling and average pooling operations, and two feature maps of size 1×H×W are obtained; Then, they are concatenated and sent to a 7×7 convolutional layer. The single-channel feature map generated by the convolutional layer is activated by a Sigmoid function to obtain a spatial attention feature map. Finally, the obtained spatial attention feature map is multiplied with the original feature map to restore it to its original size of C×H×W.

8. The infrared and visible light fusion method based on multiple feature extraction according to claim 1, characterized in that: The content loss function contains the intensity loss l int and texture loss l texture , defined as follows: l content ＝l int +αl texture Among them, the strength loss l int Can constrain the overall intensity of the fused image, texture loss l texture It can make the fused image contain more delicate texture details; α is used to achieve an effective balance between intensity loss and texture loss; H and W represent the height and width of the image respectively, ||.||1 represents the l1 norm, and max(,) represents the maximum selection of each element; It is the Sobel gradient operator, which is used to measure the fine-grained texture information of the image; |.| represents the absolute value operation.

9. The infrared and visible light fusion method based on residual feature extraction according to claim 1 is characterized in that: The semantic loss includes the main semantic loss l main and auxiliary semantic loss l aux , defined as follows: Among them, L so ∈R H×W×C Indicated by the segmentation symbol L s ∈(1,C) H×W×C The transformed one-hot vector; Finally, the semantic loss l semantic It is expressed as: the semantic =l main +l aux Among them, λ is a constant that balances the main semantic loss and the auxiliary semantic loss.

Images