Infrared image and visible light image fusion method based on deep learning

Abstract

The invention relates to the technical field of image fusion, in particular to an infrared image and visible light image fusion method based on deep learning, and the method comprises the steps: constructing a multi-scale feature extraction module which is used for extracting the features of a visible light image and an infrared image, and further extracting the features from a feature map outputted by a feature interaction module, the extracted features are input into the feature interaction module to continue feature interaction; an information aggregation module and a multi-level fusion module are constructed, the information aggregation module is used for performing information aggregation on output features of different feature interaction modules, and the multi-level fusion module is used for fusing deep features of visible light and infrared images to obtain a feature map required by image fusion; the infrared and visible light image fusion network is constructed, the fused image contains more key information of the infrared image and the visible light image, a better visual effect is achieved, and the problem that after the infrared image and the visible light image are fused, the key information of the image is seriously lost can be solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and particularly relates to a method for fusing infrared images and visible light images based on deep learning. Background Art

[0002] Infrared thermal imaging technology uses the thermal radiation properties of objects to form images. The infrared images formed have prominent thermal target information and are not affected by lighting conditions. However, the spatial resolution and contrast are often low, and the ability to present target details is weak. Visible light imaging technology uses the reflection of visible light to form images. The visible light images formed have high spatial resolution and can display very rich detail information. However, the imaging quality is easily affected by factors such as lighting conditions. Infrared images and visible light images contain different information about objects respectively. Fusing the two can provide more comprehensive and accurate information, which is beneficial to improving the accuracy of target tracking, recognition, and detection, etc.

[0003] The methods for fusing infrared images and visible light images can be roughly divided into traditional methods and deep learning-based methods. Traditional image fusion methods mainly include image fusion methods based on multi-scale transformation, image fusion methods based on sparse representation, and image fusion methods based on subspace, etc. The performance of traditional image fusion methods highly depends on manually designed feature extractors and fusion rules. Although they show good fusion performance in specific scenarios, they are not applicable to the fusion of infrared and visible light images in variable scenarios and have weak generalization ability.

[0004] The deep learning-based image fusion method uses a large number of training images to train a deep convolutional network, enabling it to actively learn feature extraction and image fusion strategies to achieve the fusion of visible light and infrared images. Compared with traditional methods, it requires no manual intervention and effectively improves the image fusion performance and generalization ability.

[0005] Although the deep learning-based method for fusing infrared images and visible light images has become one of the important methods for image fusion, the existing methods only use convolutional operations to extract features of each modality, without considering the interaction between modality features and ignoring the complementary information between contexts. Moreover, they do not fully explore the important information of the source images and do not well fuse the useful features between complementary modalities, resulting in a large amount of key information being lost in the fused images and affecting the accuracy of subsequent image processing methods. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and propose a method for fusing infrared images and visible light images based on deep learning, which can solve the problem of serious loss of key information in the images after fusing infrared images and visible light images.

[0007] To achieve the above object, the present invention adopts the following specific technical solutions:

[0008] The infrared image and visible light image fusion method based on deep learning provided by the present invention includes the following steps:

[0009] S1. Obtain visible light images and infrared images of the same scene at the same time. Through image cropping and registration processing, obtain an image pair including a visible light image and the corresponding infrared image; fuse the visible light image and the infrared image in the image pair to obtain a fused image; divide the fused image and the corresponding image pair into a training set and a test set;

[0010] S2. Construct a multi-scale feature extraction module for extracting features of infrared images and visible light images;

[0011] S3. Construct a feature interaction module for mining the difference information between infrared images and visible light images, complementing the information features of each other's modalities, and enriching the texture information and structural information in the fused features through information interaction;

[0012] S4. Construct an information aggregation module for obtaining an aggregated feature map by integrating features from different depths and associating important information at each level;

[0013] S5. Construct a multi-level fusion module for adaptively deep fusing the features of infrared and visible light images, so that the final image can simultaneously retain the clear texture details from the visible light image and the thermal radiation information from the infrared image, and improve the visual quality of the image;

[0014] S6. Establish an infrared and visible light image fusion network model, including two feature extraction networks, a Gaussian filter, a feature interaction network, two information aggregation modules, a multi-level fusion module, and an image reconstruction network;

[0015] The two feature extraction networks are respectively used for extracting features of visible light images and infrared images;

[0016] The Gaussian filter processes the visible light image to obtain a Gaussian mask image;

[0017] The feature interaction network includes three feature interaction modules for adjusting and correcting the correlation and difference between different modality features according to the importance distribution of pixel information in different regions of the Gaussian mask image;

[0018] The two information aggregation modules are used for aggregating the visible light image features and infrared image features output by the three feature interaction modules in the feature interaction network to achieve aggregation of features of the same attribute at different depths;

[0019] The multi-level fusion module is used to fuse the aggregated features of the infrared image and the aggregated features of the visible light image, providing a feature map containing rich features for image fusion;

[0020] The image reconstruction network is used to reconstruct the fused image from the fused feature map;

[0021] S7. Use the training set in S1 to train the infrared and visible light image fusion network model, and then use the test set in S1 to test the trained model. Use the trained model that meets the expected requirements as the final infrared and visible light image fusion model for image fusion.

[0022] Further, in step S2, the multi-scale feature extraction module includes a first branch, a second branch, and a third branch;

[0023] The first branch includes a gradient operation and a 1×1 convolutional layer. The gradient operation uses the sobel operator to calculate the gradient magnitude of the input feature to obtain fine-grained detail features; the sobel operator includes two 3×3 convolutional kernels for calculating the gray-scale weighted difference in the neighborhood of the central pixel; the two 3×3 convolutional kernels are the sobel filter G x in the horizontal direction and the sobel filter G y in the vertical direction, specifically as follows:

[0024]

[0025] The feature map F sobel obtained after being processed by the sobel operator has the following expression:

[0026] F sobel = G x *A + G y *A;

[0027] In the formula, A represents the image feature map to be processed, and * represents ordinary convolution;

[0028] The second branch includes three parallel 3×3 dilated convolutions and a LeakyRelu activation function. The dilation rates of the three parallel 3×3 dilated convolutions are 1, 2, and 4 respectively. After the input feature is output by the three parallel 3×3 dilated convolutions, it is concatenated at the channel dimension and then passes through a 1×1 convolutional layer to extract more feature information of the image;

[0029] The third branch includes a 3×3 convolution and a 1×1 convolution for obtaining supplementary information of the image;

[0030] All three branches of the multi-scale feature extraction module adopt 1×1 convolutional layers to eliminate the differences in the channel dimension of each branch. And the output feature maps of the three branches are subjected to element-wise addition operation and the LeakyRelu activation function to obtain the output feature map.

[0031] Furthermore, in step S3, the feature interaction module includes channel feature interaction and spatial feature interaction;

[0032] The channel feature interaction concatenates the visible light image feature map F v ′ i ∈R H×W×C and the infrared image feature map F i ′ r ∈R H×W×C along the channel dimension, then performs max pooling and average pooling respectively to retain the significant features and average feature information of the feature map. The two result vectors of the significant feature and the average feature information are concatenated together and applied to the MLP to split it, and the infrared channel feature weight and the visible light channel feature weight are obtained respectively. The expressions are as follows:

[0033]

[0034] where F mlp () represents the first multi-layer perceptron, C() represents concatenation along the channel dimension, F v ′ i and F i ′ r are the visible light image feature map and the infrared image feature map respectively;

[0035] The infrared channel feature weight information is weighted by element-wise multiplication with the infrared image feature, and the visible light channel weight information is weighted by element-wise multiplication with the visible light image feature to obtain the information volume of the infrared feature and the visible light feature in the channel dimension. The expressions are as follows:

[0036]

[0037] where, represents element-wise multiplication;

[0038] In the channel feature interaction, the bimodal feature F v ′ i and F i ′ rConcatenate along the channel dimension, and input the concatenated feature map into another multi-layer perceptron to obtain spatial feature weights and The process is as follows:

[0039]

[0040] Among them, F ′ mlp () represents the second multi-layer perceptron, and the spatial integration formula is as follows:

[0041]

[0042] Take half of the infrared feature maps containing channel features and spatial feature weight information as supplementary information, add it to the visible light features, and correct the visible light image feature map; take half of the visible light feature maps containing channel features and spatial feature weight information as supplementary information, add it to the infrared features, and correct the infrared image feature map; enable the complementary features of different modalities to interact in the channel and spatial dimensions to achieve adaptive supplementation and adjustment of modality features, and obtain the visible light features after comprehensive interaction and infrared features The expression is as follows:

[0043]

[0044] Furthermore, in step S3, the feature interaction module further includes taking the inverse of the Gaussian mask map F vmask of the visible light image to obtain the Gaussian mask map F imask of the infrared image, fusing the Gaussian mask map of the visible light image with the visible light image feature map using the concatenation operation, fusing the Gaussian mask map of the infrared image with the infrared image feature map using the concatenation operation, and inputting the fused feature map into two residual feature extraction modules with the same structure for feature extraction;

[0045] The main branch of the residual feature extraction module includes three convolutional layers and a Relu activation function. The residual branch of the residual feature extraction module includes a 1×1 convolution, and this 1×1 convolution is used to adjust the number of channels of the input feature map so that the features output by the residual branch can be merged with the features output by the main branch, respectively obtaining the visible light image feature map F v ′ i ∈R H ×W×C and the infrared image feature map F i ′ r ∈R H×W×C , specifically as follows:

[0046]

[0047] Among them, F vi ∈R H×W×C and F ir ∈R H×W×C represent the visible light image feature and the infrared image feature respectively. F vmask represents the visible light image mask map, and F imask represents the infrared image Gaussian mask map. C() represents concatenation along the channel dimension, represents element-wise addition, and Conv 1×1 () represents a convolutional layer with a kernel size of 1×1. Conv() represents a convolutional block composed of 2 1×1 convolutions, 1 3×3 convolution, batch normalization, and ReLu activation function.

[0048] Furthermore, in step S4, the information aggregation module passes the features of each layer of the infrared or visible light image through a convolutional layer with a kernel size of 3×3, batch normalization, and LeakyRelu activation function respectively to capture the effective information of each layer of features, concatenates the features of the first two layers along the channel dimension to merge their associated information, and obtains the concatenated feature

[0049]

[0050] where x represents the infrared image or visible light image, represents the output feature map of the first FIM1 in the complete network, represents the output feature map of the second FIM1 in the complete network. Conv BN () represents a convolutional block composed of a 3×3 convolutional layer, batch normalization, and LeakyRelu activation function. C() represents concatenation along the channel dimension;

[0051] Input the concatenated feature into a convolutional block composed of a convolutional layer with a kernel size of 3×3, batch normalization, and LeakyRelu activation function, use the splicing operation to fuse the output feature map with , and perform a 1×1 convolution to obtain the aggregated feature map. The expression is as follows:

[0052]

[0053] where represents the output feature map of FIM2, and Conv 1×1 () is a convolutional layer with a kernel size of 1×1.

[0054] Further, in step S5, the multi-level fusion module performs multi-level information fusion on the features output by the information aggregation module and performs multi-level information fusion;

[0055] First, four parallel dilated convolutions are embedded to obtain multi-scale feature information and output parallel visible light feature maps and infrared image feature maps The expression is as follows:

[0056]

[0057] where represents a dilated convolution with a convolution kernel size of 3×3 and a dilation rate of j (j = 1, 3, 5, 7);

[0058] Use addition, multiplication, and subtraction in three dimensions to mine the complementary and unique information of the fusion channels. The integrated result is denoted as: F Xi (i = 1, 2, 3) ∈ R H×W×1 , and the specific formula is as follows:

[0059]

[0060] Map the obtained feature maps to spatial attention for compression to obtain spatial information weights φ(F Xi )(i = 1, 2, 3), and reload them into the corresponding fusion features F Xi (i = 1, 2, 3). Subsequently, splice the three types of features in the channel dimension and extract the global information weight W G and local information weight sum W L :

[0061]

[0062] where, represents spatial attention, AvgPool() represents adaptive average pooling, and MaxPool() represents adaptive max pooling;

[0063] Add the global and local information weights to obtain the final channel weight W, then evenly divide W into three equal parts, and multiply them by the initially obtained multi-level information F Xi (i = 1, 2, 3) respectively. Finally, add the three-layer feature information obtained to get the fusion information The process is as follows:

[0064] W = W L + W G ;

[0065] f i = W [0:C] × F X1 + W [C:2C] × F X2 + W [2C:] × F X3 ;

[0066] Merge the extracted fusion information, and obtain the final fusion feature F through a module composed of convolutional layers with convolutional kernel sizes of 1×1 and 3×3, batch normalization, and activation functions fu ∈ R H×W×C , and the expression is as follows:

[0067] F fu = Re(BN(Conv 3×3 (Conv 1×1 (C(f1, f2, f3, f4))))) ;

[0068] Among them, C() represents concatenation along the channel dimension, Conv n×n () represents a convolutional layer with a convolutional kernel size of n×n, and BN() and Re() represent batch normalization and ReLu activation functions respectively.

[0069] Furthermore, in step S6, the feature extraction network includes a 1×1 convolution and three multi-scale feature extraction modules with the same structure; the image reconstruction network includes four cascaded 3×3 convolutions and a 1×1 convolution, and each 3×3 convolution is followed by a batch normalization layer and a LeakyRelu activation function.

[0070] The present invention can achieve the following technical effects:

[0071] The infrared image and visible light image fusion method based on deep learning provided by the present invention includes constructing a multi-scale feature extraction module for extracting features of visible light images and infrared images, further extracting features from the feature maps output by the feature interaction module, and inputting the extracted features into the feature interaction module to continue feature interaction; also constructing an information aggregation module and a multi-level fusion module. The information aggregation module is used to aggregate the output features of different feature interaction modules, and the multi-level fusion module is used to fuse the deep features of visible light and infrared images to obtain the feature maps required for image fusion; constructing an infrared and visible light image fusion network guided by Gaussian filtering. The fused image contains more key information of infrared images and visible light images, has a better visual effect, and can solve the problem of serious loss of key information in the image after infrared image and visible light image fusion. Description of the Drawings

[0072] Figure 1It is a schematic structural diagram of a multi-scale feature extraction module provided according to an embodiment of the present invention;

[0073] Figure 2 It is a schematic structural diagram of a two-stage feature interaction module provided according to an embodiment of the present invention;

[0074] Figure 3 It is a schematic structural diagram of a one-stage feature interaction module provided according to an embodiment of the present invention;

[0075] Figure 4 It is a schematic structural diagram of an information aggregation module provided according to an embodiment of the present invention;

[0076] Figure 5 It is a schematic structural diagram of a multi-level fusion module provided according to an embodiment of the present invention;

[0077] Figure 6 It is a schematic structural diagram of an infrared and visible light image fusion network model provided according to an embodiment of the present invention;

[0078] Figure 7 It is a pair of source images, a fused image, and a locally enlarged image obtained by testing the method proposed in this embodiment and other methods on the MSRS dataset according to an embodiment of the present invention;

[0079] Figure 8 It is another pair of source images, a fused image, and a locally enlarged image obtained by testing the method proposed in this embodiment and other methods on the MSRS dataset according to an embodiment of the present invention;

[0080] Figure 9 It is a comparison result data graph of seven metrics obtained from all image pairs in the MSRS test dataset according to an embodiment of the present invention;

[0081] Figure 10 It is a pair of source images, a fused image, and a locally enlarged image obtained by testing the method proposed in this embodiment and other methods on the TNO dataset according to an embodiment of the present invention;

[0082] Figure 11 It is another pair of source images, a fused image, and a locally enlarged image obtained by testing the method proposed in this embodiment and other methods on the TNO dataset according to an embodiment of the present invention;

[0083] Figure 12 It is a comparison result data graph of seven metrics obtained from all image pairs in the TNO test dataset according to an embodiment of the present invention;

[0084] Figure 13The method proposed in this embodiment and other methods are tested on the M3FD dataset provided by the embodiments of the present invention, including a pair of source images, a fused image, and a locally magnified image;

[0085] Figure 14 The method proposed in this embodiment and other methods are tested on the M3FD dataset provided by the embodiments of the present invention, including another pair of source images, a fused image, and a locally magnified image;

[0086] Figure 15 It is a comparison result data graph of seven metrics obtained from all image pairs in the M3FD test dataset provided by the embodiments of the present invention. Detailed implementation manners

[0087] In the following, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, their detailed descriptions will not be repeated.

[0088] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

[0089] The infrared image and visible light image fusion method based on deep learning provided by the embodiments of the present invention includes the following steps:

[0090] S1. Obtain visible light images and infrared images of the same scene at the same time, and process them using image cropping and registration techniques to obtain an image pair composed of a visible light image and the corresponding infrared image. Use a fusion method to fuse the paired visible light image and infrared image to obtain a fused image. Use 80% of the fused images and the corresponding image pairs as the training set, and 20% of the fused images and the corresponding image pairs as the test set.

[0091] S2. Construct a multi-scale feature extraction module (MFEM) for extracting features of infrared images and visible light images.

[0092] The multi-scale feature extraction module consists of three branches. Figure 1 The structure of the multi-scale feature extraction module is shown.

[0093] The first branch consists of a gradient operation and a 1×1 convolutional layer. The gradient operation uses the sobel operator to calculate the gradient magnitude of the input feature to obtain fine-grained detail features. The sobel operator consists of two 3×3 convolutional kernels for calculating the gray-scale weighted difference in the neighborhood of the central pixel. The two convolutional kernels are the sobel filters G in the horizontal direction.x and the Sobel filter G in the vertical direction y , which is specifically expressed as follows:

[0094]

[0095] The expression of the feature map obtained after being processed by the Sobel operator is as follows:

[0096] F sobel = G x * A + G y * A (3);

[0097] In the formula, A represents the image feature map to be processed, and * represents ordinary convolution.

[0098] The second branch consists of three parallel 3×3 dilated convolutions and the LeakyRelu activation function. The dilation rates of the dilated convolutions are 1, 2, and 4 respectively. After the input features are output by these three parallel dilated convolutions, they are concatenated at the channel dimension, and then pass through a 1×1 convolutional layer, which is mainly used to extract more multi-scale feature information of the image.

[0099] The third branch consists of a 3×3 convolution and a 1×1 convolution to obtain complementary information of the image. All three branches use 1×1 convolutional layers to eliminate the differences in the channel dimension of each branch. And the output feature maps of the three branches pass through an element-wise addition operation and the LeakyRelu activation function to obtain the final output feature map.

[0100] Since each branch uses different feature extraction operations, the extracted feature information is different. Therefore, different scale features in the source image and the global context are effectively extracted, which improves the performance of the network when processing two-modal features.

[0101] S3. Construct a Feature Interaction Module (FIM) to effectively mine the difference information between infrared images and visible light images, and then complement the information features of each other's modalities. Through information interaction, the texture information and structural information in the fused features can be further enriched.

[0102] In order to enable different-modal features to interact according to the information weight distribution in the Gaussian mask map and different-modal features to adjust each other, this embodiment provides a two-stage feature interaction module and a one-stage feature interaction module. The two-stage feature interaction module (IFM1) consists of stage 1 and stage 2, and the one-stage feature interaction module (IFM2) consists only of stage 2.

[0103] In stage 1, considering that the information contained in the Gaussian mask map of the infrared image is complementary to that in the Gaussian mask map of the visible light image, in order to make full use of the infrared and visible light image information, the Gaussian mask map F of the visible light image vmask is inverted to obtain the Gaussian mask map F of the infrared image imask . The Gaussian mask map of the infrared image contains the important information distribution of each region in the infrared image. First, the Gaussian mask map of the visible light image is fused with the feature map of the visible light image by using the splicing operation, and the Gaussian mask map of the infrared image is also fused with the feature map of the infrared image by using the splicing operation. The fused feature maps are respectively input into two residual feature extraction modules with the same structure for feature extraction. The main branch of this residual feature extraction module consists of three convolutional layers and a Relu activation function, and the residual branch consists of a 1×1 convolution, which is mainly used to adjust the number of channels of the input feature map so that the features output by the residual branch can be merged with the features output by the main branch, respectively obtaining the feature map F of the visible light image v ′ i ∈R H×W×C and the feature map F of the infrared image i ′ r ∈R H×W×C , which are specifically expressed as follows:

[0104]

[0105] where F vi ∈R H×W×C and F ir ∈R H×W×C respectively represent the visible light image feature and the infrared image feature, F vmask represents the visible light image mask map, F imask represents the Gaussian mask map of the infrared image, C() represents concatenation along the channel dimension, represents element-wise addition, Conv 1×1 () represents a convolutional layer with a kernel size of 1×1, and Conv() represents a convolutional block composed of 2 1×1 convolutions, 1 3×3 convolution, batch normalization, and ReLu activation function.

[0106] In stage 2, considering that there is usually a complementary relationship between the infrared and visible light image feature maps from different sensors. This embodiment designs stage 2 to integrate the complementary modal information, so that the two modal features can adaptively exchange complementary information and calibrate features according to the modal information weights, in order to achieve the purpose of full interaction of complementary features. Channel feature interaction and spatial feature interaction are included in stage 2.

[0107] In terms of channel feature interaction, the visible light image feature map F output in stage 1 v ′ i ∈R H×W×C and the infrared image feature map F i ′ r ∈R H×W×C are concatenated along the channel dimension, and then the joint features are respectively subjected to max pooling and average pooling to retain the significant features and average feature information of the feature map. Then, the two resulting vectors are concatenated together and applied to an MLP to split it, so as to obtain the channel feature weights of infrared and visible light respectively and The generation process of the weights can be expressed as:

[0108]

[0109] where F mlp () represents the first multi-layer perceptron, C() represents concatenation along the channel dimension, F v ′ i and F i ′ r are the visible light image feature map and the infrared image feature map output in stage 1 respectively. Subsequently, the infrared channel weight information is weighted by element-wise multiplication with the infrared image features. Similarly, the visible light channel weight information is weighted by element-wise multiplication with the visible light image features to obtain the information content of the infrared features and the visible light features in the channel dimension, which is specifically expressed as follows:

[0110]

[0111] where, represents element-wise multiplication.

[0112] In terms of channel feature interaction, the bimodal features F v ′ i and F i ′ r are concatenated along the channel dimension, and the concatenated feature map is input into another multi-layer perceptron to obtain the spatial feature weights and The process is as follows:

[0113]

[0114] where, F ′ mlp() represents the second multi-layer perceptron. Similar to channel integration, the spatial integration formula is as follows:

[0115]

[0116] Finally, take half of the infrared feature maps containing the channel feature and spatial feature weight information respectively as supplementary information, add it to the visible light features, and correct the visible light image feature maps. Similarly, the same operation is performed on the visible light feature maps containing the channel feature and spatial feature weight information. This enables the complementary features of different modalities to interact in the channel and spatial dimensions, thereby realizing the adaptive supplementation and adjustment of modal features, and finally obtaining the visible light features after comprehensive interaction and infrared features The specific expression is as follows:

[0117]

[0118] The structure of the two-stage feature interaction module is as Figure 2 shown. The structure of the one-stage feature interaction module is as Figure 3 shown. The one-stage feature interaction module and Figure 2 have the same network structure in stage 2. The only difference between the two is that Figure 3 the inputs in Figure 2 are the visible light image features and infrared image features respectively, while the inputs of the stage 2 network in

[0119] are the visible light image features and infrared image features output in stage 1 respectively.

[0119] The feature interaction module enhances the interaction between different modal features through adaptive complementary information exchange. The design of this module ensures that infrared and visible light images can effectively exchange information during the fusion process, enhancing the thermal radiation targets and texture details in the fused image.

[0120] S4. Construct an Information Aggregation Module (IFAM). Considering that there is a correlation between the feature information of each layer, not only do they all contain rich modal details and structural information, but also the deep features of each modality are obtained through deep interaction with the complementary features of the other modality. Therefore, the context information has similar spatial attributes, which is conducive to highlighting the importance of modal features. In view of this, an information aggregation module is designed to integrate the correlation between multiple-level features in each modality. The structure of the information aggregation module is as Figure 4 shown.

[0121] First, the features of each layer of the infrared or visible light image The effective information of each layer's features is captured through a convolutional layer with a kernel size of 3×3, batch normalization, and the LeakyRelu activation function respectively. Then, the features of the first two layers are concatenated in the channel dimension to merge their associated information, obtaining the concatenated features.

[0122]

[0123] Among them, x represents an infrared image or a visible light image. represents the output feature map of the first FIM1 in the complete network. represents the output feature map of the second FIM1 in the complete network. Conv BN () represents a convolutional block composed of a 3×3 convolutional layer, batch normalization, and the LeakyRelu activation function. C() represents concatenation along the channel dimension.

[0124] Then, the concatenated features are input into a convolutional block composed of a convolutional layer with a kernel size of 3×3, batch normalization, and the LeakyRelu activation function. Finally, the output feature map and are fused using a splicing operation and passed through a 1×1 convolution to obtain the final aggregated feature map.

[0125] Specifically, it is expressed as follows:

[0126]

[0127] Among them, represents the output feature map of FIM2. Conv 1×1 () is a convolutional layer with a kernel size of 1×1. The information aggregation module integrates features from different depths and correlates important information at each level. It aggregates information on the spatial attributes between context information to facilitate subsequent deep fusion of the two modalities, ultimately enhancing the overall performance of the fused image.

[0128] S5. Construct a multi-level fusion module (MLFM). Since the features of visible light images and infrared images come from images of different modalities, they have complementary information. Only by fully fusing the complementary information between the two modalities can it be ensured that the fused image contains rich visible light texture information and infrared thermal radiation information. To better fuse the bimodal information, a multi-level fusion module is designed to perform multi-level information fusion on the output features and of the information aggregation module. The structure of the multi-level fusion module is as Figure 5 shown.

[0129] First, four parallel dilated convolutions are embedded to obtain multi-scale feature information and output parallel visible light feature maps and infrared image feature maps Dilated convolutions can expand the receptive field without increasing the computational cost, which is beneficial to capturing the global information of both modalities. The specific expression is as follows:

[0130]

[0131] where represents a dilated convolution with a kernel size of 3×3 and a dilation rate of j (j = 1, 3, 5, 7).

[0132] The dual-modal features corresponding to each parallel branch are fused in depth from multiple levels. Taking the first branch as an example, specifically, three dimensions of addition, multiplication, and subtraction are used to fully explore the complementary and unique information of the fusion channels. The integrated result is denoted as: F Xi (i = 1, 2, 3) ∈ R H×W×1 , and the specific formula is as follows:

[0133]

[0134] The obtained feature maps are mapped to spatial attention for compression to obtain spatial information weights φ(F Xi )(i = 1, 2, 3), and then reloaded into the corresponding fused features F Xi (i = 1, 2, 3). Subsequently, the three types of features are concatenated in the channel dimension, and global information weights W G and local information weights and W L are extracted at the channel level:

[0135]

[0136] where represents spatial attention, AvgPool() represents adaptive average pooling, and MaxPool() represents adaptive max pooling.

[0137] Subsequently, the global and local information weights are added to obtain the final channel weight W, and then W is evenly divided into three equal parts and multiplied by the initially obtained multi-level information F Xi (i = 1, 2, 3). Finally, the three-layer feature information obtained is added to obtain the fused information The process is as follows:

[0138] W = W L + W G (23);

[0139] f i =W [0:C] ×F X1 +W [C:2C] ×F X2 +W [2C:] ×F X3 (24);

[0140] Execute multi - angle information fusion through channel weights. The relevant weights reflect the relative contributions of two modal features to the overall fusion feature in the channel dimension. By adopting this method to achieve information fusion, parallel bimodal information can be adaptively and deeply fused according to the complementary features of different channels.

[0141] Finally, in order to obtain multi - scale and multi - level modal fusion features, the extracted fusion information is merged, and the final fusion feature F is obtained through a module composed of convolutional layers with kernel sizes of 1×1 and 3×3, batch normalization, and activation functions. fu ∈R H×W×C , and the expression is as follows:

[0142] F fu =Re(BN(Conv 3×3 (Conv 1×1 (C(f1,f2,f3,f4))))) (25);

[0143] Among them, C() represents concatenation along the channel dimension, Conv n×n () represents a convolutional layer with a kernel size of n×n, and BN() and Re() represent batch normalization and ReLu activation functions respectively.

[0144] The multi - level fusion module adaptively and deeply fuses the features of infrared and visible - light images through multiple levels. This method performs information fusion from multiple angles, enabling the final image to better fuse the information of the two modalities, ensuring that the final image can simultaneously retain the clear details from the visible - light image and the thermal radiation information from the infrared image, thereby improving the visual quality of the image.

[0145] S6. Establish an infrared and visible - light image fusion network model. The structure of the infrared and visible - light image fusion network model is as Figure 6 shown, including two feature extraction networks, a Gaussian filter, a feature interaction network, two information aggregation modules, a multi - level fusion module, and an image reconstruction network.

[0146] The Gaussian filter consists of Gaussian functions and is mainly used to process the visible - light image to obtain a Gaussian mask image that can highlight the important information of the visible - light image.

[0147] The feature extraction network consists of a 1×1 convolution and three multi-scale feature extraction modules with the same structure. Two feature extraction networks with the same structure are respectively used to extract visible light image features and infrared image features. The first residual module of the feature extraction network is mainly used to extract shallow features from the source image, and the second and third residual modules are mainly used to extract deep features from the feature interaction module.

[0148] The feature interaction network consists of three feature interaction modules. The first feature interaction module and the second feature interaction module have the same structure, and the third feature interaction module is only composed of a part of the first feature interaction module. The feature interaction network is mainly used to adjust and correct the correlation and difference between different modality features according to the importance distribution of pixel information in different regions of the Gaussian mask map. Since the Gaussian mask itself represents spatial attention, and the spatial features in the deep features decrease with the increase of the network depth, the Gaussian mask map is not introduced in the third feature interaction module.

[0149] Two information aggregation modules with the same structure are respectively used to aggregate the visible light image features and infrared image features output by the three feature interaction modules, so as to achieve the aggregation of different depth features of the same attribute.

[0150] The multi-level fusion module is mainly used to fuse the infrared image aggregation features and the visible light image aggregation features, so as to provide a feature map containing rich features for image fusion.

[0151] The image reconstruction network consists of four cascaded 3×3 convolutions and a 1×1 convolution. A batch normalization layer and a LeakyRelu activation function are connected after each 3×3 convolution. The image reconstruction network is mainly used to reconstruct the fused image from the fused feature map.

[0152] S7. Use the training set in S1 to train the infrared and visible light image fusion network model, and then use the test set in S1 to test the trained model. Use the trained model that meets the expected requirements as the final infrared and visible light image fusion model for image fusion.

[0153] The infrared image and visible light image fusion method based on deep learning provided by the embodiments of the present invention includes constructing a multi-scale feature extraction module for extracting the features of visible light images and infrared images, further extracting features from the feature maps output by the feature interaction module, and inputting the extracted features into the feature interaction module for continuous feature interaction; an information aggregation module and a multi-level fusion module are also constructed. The information aggregation module is used for aggregating the output features of different feature interaction modules, and the multi-level fusion module is used for fusing the deep features of visible light and infrared images to obtain the feature maps required for image fusion; a Gaussian filter-guided infrared and visible light image fusion network is constructed. The fused image contains more key information of infrared images and visible light images, has a better visual effect, and can solve the problem of serious loss of key information in the image after the fusion of infrared images and visible light images.

[0154] The method proposed by the present invention and other methods are tested on the MSRS dataset, and two pairs of source images are randomly selected from the MSRS dataset as test images. Figure 7 and Figure 8 Two pairs of source images, the fused images and their locally enlarged images are shown. Figure 7 The first pair of source images, the fused image and the locally enlarged image are shown. Figure 8 The second pair of source images, the fused image and the locally enlarged image are shown. For better comparison, the areas containing texture details and the areas of infrared targets are respectively indicated by green frames and red frames in each picture.

[0155] From Figure 7 It can be seen that the method of FusionGAN cannot well retain the useful information in the source image, resulting in unclear infrared target information and scene details in the generated fused image. The infrared targets in the images fused by PMGI, RFN-Nest, and CSF are relatively clear, but some texture details are lost. The images fused by GANMcC and UMF-CMGR are relatively dim, and both the texture details and the infrared targets are blurred. The images fused by CUFD, LRRNet, DATfuse, and the method of the present invention effectively retain the texture detail information of visible light, but LRRNet has limited ability in representing infrared salient targets and has a lower contrast. Only CUFD, DATfuse, and the method of the present invention well maintain the contrast of the thermal targets in the infrared image.

[0156] In Figure 8It can be seen that for the method of FusionGAN, the texture details of the generated fused image scene are relatively blurred, and there is a certain distortion in the infrared targets of the fused image. The infrared thermal targets in the image fused by the PMGI method have a higher contrast, and the scene details are also relatively clear. The images fused by the RFN-Nest and CSF methods cannot maintain relatively clear texture details and infrared thermal targets with high contrast. The image fused by the GANMcC method has relatively clear scene details and significant thermal targets. The texture clarity and infrared thermal target contrast of the fused image generated by UMF-CMGR are relatively low. CUFD can maintain thermal targets with high contrast, but the scene details are not clear. The ability of the fused image generated by LRRNet to maintain texture details and the contrast of human targets is limited. Both DATfuse and the method of the present invention can maintain clear texture details, but in terms of the infrared thermal target contrast, the method of the present invention has a higher infrared thermal target contrast.

[0157] To objectively verify the effectiveness of the method provided by the present invention, all test set images in the MSRS dataset are input into the image fusion model for performance testing. The performance metrics of the fused images are as Figure 9 shown. As can be seen from Figure 9 , DATfuse has the highest MI value, followed by the method of the present invention and the CUFD method. The method of the present invention has the highest SF value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest AG value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest VIF value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest Qabf value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest SSIM value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest SNR value, followed by the LRRNet method and the CUFD method. In summary, except that the algorithm of the present invention is lower than the DATfuse method in the MI metric, since the present invention adopts the way of feature interaction, the complex feature interaction will cause a certain information distortion, resulting in a reduction in the amount of information transmitted from the source image to the fused image, and thus the MI metric is relatively low. However, it does not affect the other six metrics, and the present invention is higher than other methods in the other six metrics. This means that the method of the present invention has better image fusion ability compared to other methods.

[0158] The method proposed in this paper and other methods are tested on the TNO dataset. Two pairs of source images are randomly selected from the TNO dataset as test images. Figure 10 and Figure 11 show two pairs of source images, the fused images and their local enlarged images. Figure 10Shown are the first pair of source images, fused images, and locally magnified images. Figure 11 Shown are the second pair of source images, fused images, and locally magnified images. For better comparison, green and red frames are used in each picture to represent the regions containing texture details and the regions of infrared targets, respectively.

[0159] From Figure 10 It can be seen that the fused images generated by the FusionGAN method cannot well retain the visible light texture information and infrared thermal radiation information. The fused images generated by PMGI can well highlight the thermal radiation targets but lose some texture information. RFN-Nest and CSF show clear scene texture details but have insufficient ability to preserve infrared target information. The human target in RFN-Nest is relatively blurred, and compared with the method proposed in this paper, the contrast of the human target in CSF is lower. The fused images generated by GANMcC cannot well display clear texture information, and at the same time, the thermal radiation targets in the fused images are not clear enough. The UMF-CMGR and CUFD methods effectively retain the thermal targets of the infrared images but still have the problem of blurred texture details. The fused images generated by the LRRNet method have blurred texture details and low-contrast thermal target information. The fused images generated by DATfuse have relatively clear infrared thermal target details but still have the problem of unclear scene texture. The method of the present invention has better performance in simultaneously maintaining the thermal radiation information of the infrared image and capturing the scene texture details of the visible light image.

[0160] From Figure 11 It can be seen that the fused images generated by the FusionGAN method still cannot well retain the visible light texture information and infrared thermal radiation information. PMGI can better retain the infrared image thermal radiation information but has limited ability to retain visible light texture details. The fused images generated by the RFN-Nest, CSF, GANMcC, and UMF-CMGR methods cannot well capture the visible light image texture detail information and at the same time cannot effectively maintain the infrared image thermal radiation information. The fused images generated by RFN-Nest, CSF, and GANMcC have relatively low-clear infrared targets, while the fused images generated by UMF-CMGR have relatively low contrast of infrared thermal targets. CUFD has better performance in retaining the infrared image thermal targets, but its scene texture is relatively blurred. The fused images generated by LRRNet cannot well retain the scene texture information and infrared thermal radiation information. DATfuse has better ability to preserve the visible light image texture information, but compared with the method of the present invention, the contrast of the infrared thermal targets is lower. The fused images generated by the method of the present invention have both clear texture detail information and high-contrast thermal radiation target information.

[0161] To objectively verify the effectiveness of the method of the present invention, all image pairs in the TNO dataset are input into the image fusion model for performance testing. The average performance metrics of the fused images are as Figure 12 shown. It can be seen from Figure 12 that CUFD has the highest MI value, followed by DATfuse and the method of the present invention. The method of the present invention has the highest SF value, followed by the CUFD method and the DATfuse method. The method of the present invention has the highest AG value, followed by the CUFD method and the LRRNet method. The method of the present invention has the highest VIF value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest Qabf value, followed by the DATfuse method and the PMGI method. The UMF-CMGR method has the highest SSIM value, followed by the method of this article and the PMGI method. The method of the present invention has the highest SNR value, followed by the UMF-CMGR and DATfuse methods. The method of the present invention is lower than the CUFD method and the DATfuse method in terms of the MI metric, and lower than the UMF-CMGR method in terms of the SSIM metric. The reason for the lower MI metric is that the feature interaction module causes a certain amount of information distortion when processing features, resulting in a reduction in the amount of information transmitted from the source image to the fused image, thus leading to a lower MI metric. For the SSIM metric, Gaussian filtering pays more attention to the background and low-contrast regions when processing the low-resolution dataset images, so less attention is paid to the structure and quality of the images, thus leading to a lower SSIM metric. However, this does not affect the other five metrics, and the algorithm of this article is higher than other methods in the other five metrics. Generally speaking, compared with other methods, the method of the present invention has better image fusion performance.

[0162] The method proposed in this article and other methods were tested on the M3FD dataset. Two pairs of source images were randomly selected from the M3FD dataset as test images. Figure 13 and Figure 14 show two pairs of source images, the fused images and their locally enlarged images. Figure 13 shows the first pair of source images, the fused image and the locally enlarged image. Figure 14 shows the second pair of source images, the fused image and the locally enlarged image. For better comparison, green frames and red frames are used in each picture to represent the regions containing texture details and the regions of infrared targets respectively.

[0163] From Figure 13It can be seen that the fused images generated by the methods of FusionGAN, PMGI, RFN-Nest, CSF, GANMcC, and UMF-CMGR can better preserve the detailed textures, but the image scenes are darker, and there are also problems to varying degrees in preserving the infrared salient targets. The infrared thermal targets of the FusionGAN and RFN-Nest methods are not clear, and the contrast of the infrared thermal targets of the PMGI, CSF, GANMcC, and UMF-CMGR methods is relatively low. The CUFD method can effectively capture the texture information and infrared thermal target information in visible light images. The fused images generated by the LRRNet method contain rich texture details, but there is a problem of unclear infrared thermal radiation targets. The DATfuse and the method of this paper can effectively preserve the visible light texture information and infrared image thermal radiation information.

[0164] From Figure 14 It can be seen that the method of FusioGAN cannot well preserve the visible light texture details, and there is a problem of blurred infrared thermal radiation targets. PMGI can effectively preserve the thermal radiation information of infrared images, but the texture details of the fused images are not clear. The RFN-Nest, CSF, GANMcC, and UMF-CMGR methods have better capabilities in extracting infrared thermal target information, but the capabilities in capturing visible light texture information are insufficient. The infrared targets in the fused images generated by the CUFD and LRRNet methods are relatively clear, but there are still blurred problems in the scene details. Only the DATfuse and the method of this paper can effectively preserve the infrared thermal radiation information and well capture the visible light texture detail information.

[0165] Similarly, in order to objectively verify the effectiveness of the method of the present invention, all image pairs in the M3FD dataset are input into the image fusion model for performance testing. The average performance indicators of the fused images are as Figure 15 shown. From Figure 15It can be seen that DATfuse has the highest MI value, followed by CUFD and the method of the present invention. The method of the present invention has the highest SF value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest SF value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest AG value, followed by the DATfuse method and the CUFD method. The method of the present invention has the highest VIF value, followed by the DATfuse method and the CSF method. The method of the present invention has the highest Qabf value, followed by the DATfuse method and the LRRNet method. The method of the present invention has the highest SSIM value, followed by the UMF-CMGR method and the CSF method. The method of the present invention has the highest SNR value, followed by the UMF-CMGR method and the PMGI method. In summary, except that the algorithm of the present invention is lower than the DATfuse method and the CUFD method in terms of the MI index for the same reason as above, it is higher than other methods in the other six indexes. Generally speaking, compared with other methods, the method of the present invention has better image fusion performance.

[0166] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0167] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0168] The above specific implementation manners of the present invention do not constitute a limitation to the protection scope of the present invention. Any other corresponding changes and deformations made according to the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for fusing infrared images and visible light images based on deep learning, characterized in that: The steps include: S1. Obtain a visible light image and an infrared image of the same scene at the same time, and obtain an image pair including a visible light image and a corresponding infrared image through image cropping and registration processing; fuse the visible light image and the infrared image in the image pair to obtain a fused image; and divide the fused image and the corresponding image pair into a training set and a test set; S2, construct a multi-scale feature extraction module for extracting features of infrared images and visible light images; S3. Construct a feature interaction module to mine the difference information between infrared images and visible light images, complement the information features of each modality, and enrich the texture information and structural information in the fusion features through information interaction; S4, constructing an information aggregation module, which is used to associate important information at each level by integrating features from different depths to obtain an aggregated feature map; S5. Construct a multi-level fusion module to perform adaptive deep fusion of the features of infrared and visible light images, so that the final image can retain both the clear texture details from the visible light image and the thermal radiation information from the infrared image, thus improving the visual quality of the image; S6. Establish an infrared and visible light image fusion network model, including two feature extraction networks, Gaussian filter, feature interaction network, two information aggregation modules, multi-level fusion module, and image reconstruction network; The two feature extraction networks are used to extract visible light image features and infrared image features respectively; The Gaussian filter processes the visible light image to obtain a Gaussian mask image; The feature interaction network includes three feature interaction modules, which are used to adjust and correct the correlation and difference between different modality features according to the importance distribution of pixel information in different regions of the Gaussian mask image; Two information aggregation modules are used to aggregate the visible light image features and infrared image features output by the three feature interaction modules in the feature interaction network to achieve aggregation of different depth features of the same attribute; The multi-level fusion module is used to fuse the infrared image aggregation features and the visible light image aggregation features to provide a feature map containing rich features for image fusion; The image reconstruction network is used to reconstruct a fused image from the fused feature map; S7. Use the training set in S1 to train the infrared and visible light image fusion network model, and then use the test set in S1 to test the trained model, and use the trained model that meets the expected requirements as the final infrared and visible light image fusion model for image fusion.

2. The infrared image and visible light image fusion method based on deep learning according to claim 1 is characterized in that: In step S2, the multi-scale feature extraction module includes a first branch, a second branch and a third branch; The first branch includes a gradient operation and a 1×1 convolution layer, wherein the gradient operation uses a sobel operator to calculate the gradient size of the input feature to obtain fine-grained detail features; The sobel operator includes two 3×3 convolution kernels, which are used to calculate the grayscale weighted difference of the central pixel neighborhood; the two 3×3 convolution kernels are divided into a horizontal sobel filter G x And the vertical sobel filter G y , as follows: The feature map F obtained after processing by the sobel operator sobel The expression is as follows: F sobel =G x *A+G y *A; In the formula, A represents the image feature map to be processed, and * represents ordinary convolution; The second branch includes three parallel 3×3 dilated convolutions and a LeakyRelu activation function, wherein the dilated holes of the three parallel 3×3 dilated convolutions are 1, 2, and 4 respectively. After the input features are output by the three parallel 3×3 dilated convolutions, they are cascaded in the channel dimension and then pass through a 1×1 convolution layer to extract more feature information of the image. The third branch includes a 3×3 convolution and a 1×1 convolution, which are used to obtain supplementary information of the image; The three branches of the multi-scale feature extraction module all use a 1×1 convolution layer to eliminate the differences between the branches in the channel dimension, and the output feature maps of the three branches are subjected to element-wise addition operation and LeakyRelu activation function to obtain the output feature maps.

3. The infrared image and visible light image fusion method based on deep learning according to claim 1, characterized in that: In step S3, the feature interaction module includes channel feature interaction and spatial feature interaction; The channel feature interaction transforms the visible light image feature map F′ vi ∈R H×W×C and infrared image feature map F′ ir ∈R H×W×C Connect along the channel dimension, and then perform maximum pooling and average pooling respectively to retain the significant features and average feature information of the feature map. Connect the two result vectors of significant features and average feature information together and apply MLP to split them to obtain the infrared channel feature weights respectively. Channel feature weights with visible light The expression is as follows: Among them, F mlp () represents the first multi-layer perceptron, C() represents the cascade along the channel dimension, and F′ vi and F′ ir They are visible light image feature map and infrared image feature map respectively; The infrared channel feature weight information and the infrared image feature are weighted by element-wise multiplication, and the visible light channel weight information and the visible light image feature are weighted by element-wise multiplication to obtain the information amount of the infrared feature and the visible light feature in the channel dimension. The expression is as follows: in, Represents element-wise multiplication; In the channel feature interaction, the bimodal feature F′ vi and F′ ir Concatenate along the channel dimension and input the concatenated feature map into another multi-layer perceptron to obtain spatial feature weights and The process is as follows: Among them, F′ mlp () represents the second multi-layer perceptron, and the spatial integration formula is as follows: Take half of the infrared feature map containing channel feature and spatial feature weight information as supplementary information, add it to the visible light feature, and correct the visible light image feature map; take half of the visible light feature map containing channel feature and spatial feature weight information as supplementary information, add it to the infrared feature, and correct the infrared image feature map; make the complementary features of different modalities interact in the channel and spatial dimensions to achieve adaptive complementation and adjustment of modal features, and obtain the visible light features after comprehensive interaction and infrared characteristics The expression is as follows:

4. The infrared image and visible light image fusion method based on deep learning according to claim 3 is characterized in that: In step S3, the feature interaction module further includes a Gaussian mask image F of the visible light image vmask Perform the inversion operation to obtain the infrared image Gaussian mask image F imask , using a splicing operation to fuse the visible light image Gaussian mask map and the visible light image feature map, using a splicing operation to fuse the infrared image Gaussian mask map and the infrared image feature map, and inputting the fused feature maps into two residual feature extraction modules with the same structure to perform feature extraction; The main branch of the residual feature extraction module includes three convolutional layers and a Relu activation function. The residual branch of the residual feature extraction module includes a 1×1 convolution, which is used to adjust the number of channels of the input feature map so that the features output by the residual branch can be merged with the features output by the main branch to obtain the visible light image feature map F′. vi ∈R H ×W×C and infrared image feature map F′ ir ∈R H×W×C , as follows: Among them, F vi ∈R H×W×C and F ir ∈R H×W×C Represent the visible light image features and infrared image features respectively, F vmask Represents the visible light image mask, F imask represents the infrared image Gaussian mask map, C() represents the cascade along the channel dimension, Indicates element-by-element addition, Conv 1×1 () represents a convolutional layer with a convolution kernel of size 1×1, and Conv() represents a convolutional block, which consists of 2 1×1 convolutions, 1 3×3 convolution, batch normalization, and ReLu activation function.

5. The infrared image and visible light image fusion method based on deep learning according to claim 1, characterized in that: In step S4, the information aggregation module aggregates the features of each layer of the infrared or visible light image The convolutional layer with a kernel size of 3×3, batch normalization and LeakyRelu activation function are used to capture the effective information of the features of each layer. The features of the first two layers are cascaded in the channel dimension to merge their associated information to obtain the cascaded features. Where x represents an infrared image or a visible light image, represents the output feature map of the first FIM1 in the complete network, Represents the output feature map of the second FIM1 in the complete network, Conv BN () represents a convolutional block consisting of a 3×3 convolutional layer, batch normalization, and LeakyRelu activation function, and C() represents cascading along the channel dimension; Cascade Features The input is a convolutional block consisting of a convolutional layer with a kernel size of 3×3, batch normalization, and a LeakyRelu activation function. The output feature map is concatenated with After fusion and 1×1 convolution, the aggregated feature map is obtained, which is expressed as follows: in, Represents the output feature map of FIM2, Conv 1×1 () is a convolutional layer with a kernel size of 1×1.

6. The method for fusion of infrared image and visible light image based on deep learning according to claim 1, characterized in that: In step S5, the multi-level fusion module outputs features to the information aggregation module. and Conduct multi-level information fusion; First, four parallel dilated convolutions are embedded to obtain multi-scale feature information and output parallel visible light feature maps. and infrared image feature map The expression is as follows: in It represents a dilated convolution with a kernel size of 3×3 and a dilation rate of j (j=1, 3, 5, 7); The complementary and unique information of the fusion channels is mined using the three dimensions of addition, multiplication and subtraction. The integrated result is recorded as: F Xi (i=1,2,3)∈R H×W×1 , the specific formula is as follows: The obtained feature map is mapped to the spatial attention for compression to obtain the spatial information weight φ(F Xi )(i=1,2,3), and reload it into the corresponding fusion feature F Xi (i=1,2,3), the three types of features are then concatenated in the channel dimension, and the global information weight W is extracted at the channel level G and local information weight and W L : in, Represents spatial attention, AvgPool() represents adaptive average pooling, and MaxPool() represents adaptive maximum pooling; The global and local information weights are added together to obtain the final channel weight W, and then W is evenly divided into three equal parts and multiplied by the multi-level information F obtained initially. Xi (i=1,2,3), and finally add the obtained three layers of feature information to get the fusion information The process is as follows: W=W L +W G ; f i =W [0:C] ×F X1 +W [C:2C] ×F X2 +W [2C:] ×F X3 ; The extracted fusion information is merged, and the final fusion feature F is obtained through a module consisting of a convolution layer with a convolution kernel size of 1×1 and 3×3, batch normalization, and an activation function. fu ∈R H×W×C , the expression is as follows: F fu =Re(BN(Conv 3×3 (Conv 1×1 (C(f1,f2,f3,f4))))); Among them, C() represents the cascade along the channel dimension, Conv n×n () represents a convolutional layer with a kernel size of n×n, while BN() and Re() represent batch normalization and ReLu activation functions, respectively.

7. The infrared image and visible light image fusion method based on deep learning according to claim 1, characterized in that: In step S6, the feature extraction network includes a 1×1 convolution and three multi-scale feature extraction modules with the same structure; the image reconstruction network includes four serially connected 3×3 convolutions and one 1×1 convolution, and each 3×3 convolution is followed by a batch normalization layer and a LeakyRelu activation function.

Images