Infrared and visible light image fusion method and device

Abstract

The application discloses an infrared and visible light image fusion method and device, comprising: acquiring an infrared image and a visible light image of a target object; performing a pretreatment operation of gray scale and data enhancement on the infrared image and the visible light image to obtain a pretreated infrared image and a pretreated visible light image; inputting the pretreated infrared image and the pretreated visible light image into a pre-trained infrared and visible light image fusion model based on feature interaction and an autoencoder to obtain a fusion image; and the infrared and visible light image fusion model based on feature interaction and the autoencoder comprises an encoder, a stepped fusion layer and a one-way cascaded decoder. Advantages: the method can retain significant thermal radiation information, rich texture detail information and background features in the source image, and the fusion image has high contrast and is more in line with the visual effect of the human eye.

Description

TECHNICAL FIELD

[0001] The present application relates to an infrared and visible light image fusion method and device, belonging to the technical field of image processing. BACKGROUND

[0002] Due to the imaging characteristics of infrared sensors, infrared images have strong anti-interference ability, but the structure texture is not clear, and the spatial resolution is low, which does not conform to the human visual perception. On the contrary, visual sensors capture light reflection from objects to form visible light images with rich texture details and high resolution, but they are greatly affected by lighting conditions, especially in low light environments, and cannot accurately identify important targets. Therefore, the fusion of visible light images and infrared images can present images with rich texture details and significant contrast, thereby solving the limitations of single sensor imaging. This method has broad application prospects in target recognition, video monitoring, military applications, medical diagnosis and other fields.

[0003] Existing infrared and visible light image fusion techniques mainly include traditional fusion methods and deep learning fusion methods. Traditional fusion methods mainly include multi-scale transformation, sparse representation, subspace, saliency-based, hybrid model and other methods. These methods are only suitable for fusion between specific modalities, have limited ability to mine complex image features, usually require manual design of fusion strategies, have low computational efficiency, and the visual effect of the fused image is poor.

[0004] In recent years, deep learning has rapidly developed in the field of computer vision, and methods based on deep learning have achieved good results in image fusion. However, existing deep learning methods usually use convolution to extract image features, without fully considering the complementary characteristics and redundancy of information between different modalities, resulting in problems such as loss of image background detail information and unclear thermal radiation targets. In addition, the fusion strategies of these methods usually adopt methods such as feature map addition, mean value taking and splicing, which are difficult to adaptively fuse semantic features, and have limited extraction of image fine-grained information, resulting in unclear edges and loss of local detail information, affecting the fusion performance. SUMMARY

[0005] The technical problem to be solved by the present application is to overcome the defects of the prior art and provide an infrared and visible light image fusion method and device.

[0006] To solve the above technical problems, the present application provides an infrared and visible light image fusion method, comprising:

[0007] obtaining an infrared image and a visible light image of a target object;

[0008] performing a preprocessing operation of gray scale and data enhancement on the infrared image and the visible light image to obtain a preprocessed infrared image and a preprocessed visible light image;

[0009] input the preprocessed infrared image and the visible light image into the pre-trained infrared and visible light image fusion model based on feature interaction and autoencoder to obtain a fused image;

[0010] The infrared and visible light image fusion model based on feature interaction and autoencoder comprises an encoder, a stepped fusion layer and a one-way cascaded decoder.

[0011] The encoder is configured to extract infrared features, visible light features and complementary features of the preprocessed infrared image and the visible light image.

[0012] The stepped fusion layer is configured to obtain information weight maps of the respective features by subjecting the extracted infrared features, visible light features and complementary features to a sigmoid activation function, and adaptively fuse the infrared features, visible light features and complementary features by corresponding weight coefficients to obtain fused image features.

[0013] The one-way cascaded decoder is configured to obtain a fused image by subjecting the fused image features to four residual blocks.

[0014] Further, the encoder comprises an infrared path, a visible light path and an interaction compensation path.

[0015] The infrared path and the visible light path each comprise a first convolution group, a second convolution group, a first gradient residual block, a second gradient residual block, a third gradient residual block and a fourth gradient residual block.

[0016] The first convolution group comprises four convolutional layers connected in sequence, namely Conv1, Conv2, Conv3 and Conv4, wherein the input of Conv1 is the visible light image I v , and Conv4 outputs shallow visible light image features, and the shallow visible light image features are sequentially subjected to the first gradient residual block and the second gradient residual block to output visible light image features.

[0017] The second convolution group comprises four convolutional layers connected in sequence, namely Conv2, Conv4, Conv6 and Conv8, wherein the input of Conv5 is the infrared image I i , Conv8 outputs shallow infrared image features, and the shallow infrared image features are sequentially subjected to the third gradient residual block and the fourth gradient residual block to output infrared image features.

[0018] The interaction compensation path comprises four interactive feature perfecting modules and three cross-level feature aggregation modules, which are respectively denoted as: a first interactive feature perfecting module, a second interactive feature perfecting module, a third interactive feature perfecting module, a fourth interactive feature perfecting module, a first cross-level feature aggregation module, a second cross-level feature aggregation module, and a third cross-level feature aggregation module;

[0019] The first interactive feature perfecting module, the second interactive feature perfecting module, the first cross-level feature aggregation module, the third interactive feature perfecting module, the second cross-level feature aggregation module, the fourth interactive feature perfecting module, and the third cross-level feature aggregation module are sequentially connected; the output of the first cross-level feature aggregation module is further connected to the first cross-level feature aggregation module, the output of the first cross-level feature aggregation module is further connected to the second cross-level feature aggregation module, and the output of the second cross-level feature aggregation module is further connected to the third cross-level feature aggregation module.

[0020] The input of the first interactive feature perfecting module is connected to the output of Conv2 and Conv6, respectively; the input of the second interactive feature perfecting module is connected to the output of Conv4 and Conv8, respectively; the input of the third interactive feature perfecting module is connected to the first gradient residual block and the third gradient residual block, respectively; and the input of the fourth interactive feature perfecting module is connected to the second gradient residual block and the fourth gradient residual block, respectively.

[0021] Further, the input channel numbers of Conv1, Conv3, Conv5, and Conv7 in the first convolution group are 1, 16, 1, and 16, respectively, and the output channel numbers are 1, 16, 1, and 16, respectively; each convolution block comprises a convolution kernel of 1x1, BN, and LeakyReLU activation function; and the input features are sequentially output after 1x1 convolution, BN, and LeakyReLU operation.

[0022] The input channel numbers of Conv2, Conv4, Conv6, and Conv8 in the second convolution group are 1, 16, 1, and 16, respectively, and the output channel numbers are 16, 16, 16, and 16, respectively; each convolution block comprises a convolution kernel of 3x3, BN, and LeakyReLU activation function; and the input features are sequentially output after 3x3 convolution, BN, and LeakyReLU operation.

[0023] The first gradient residual block and the third gradient residual block, the second gradient residual block and the third gradient residual block each include a main path and a residual path, wherein the input and output channel numbers of the first gradient residual block and the third gradient residual block are 16, 32 respectively; the input and output channels of the second gradient residual block and the third gradient residual block are 32, 64 respectively; the main path includes a convolution kernel of 1*1, BN, a LeakyReLU activation function, a convolution kernel of 3*3, and the residual path includes a Scharr gradient operator and a DSConv; first, the input feature sequentially passes through 1*1 convolution, BN, a LeakyReLU activation function, a convolution kernel of 3*3, and BN operation to obtain a learnable convolution feature; second, the input feature sequentially passes through a Scharr gradient operator and a DSConv operation to obtain gradient amplitude information feature, and the two features are added and then subjected to LeakyReLU to obtain the output feature.

[0024] Further, the interactive feature perfecting module includes two stages of channel attention feature correction and spatial attention feature aggregation, the channel attention feature correction includes element-wise multiplication operation, splicing operation, MLP, average pooling operation, maximum pooling operation, sigmoid operation, element-wise addition operation, the spatial attention feature aggregation includes element-wise multiplication operation, splicing operation, average pooling operation, maximum pooling operation, element-wise addition operation;

[0025] The channel attention feature correction stage adopts element-wise multiplication method to interact the input infrared feature F ir and the visible light feature F vis , and then splices the infrared feature F ir and the visible light feature F vis to obtain a feature F c ; the feature F c is subjected to global maximum pooling, global average pooling and MLP operation in the channel dimension to obtain an output feature, the output feature is subjected to element-wise addition and sigmoid activation function operation to obtain a channel calibration weighted coefficient, and finally the channel calibration weighted coefficient is multiplied by the spliced feature F c to obtain a channel correction attention map , which is expressed as:

[0026]

[0027] Wherein, concat(·) represents a splicing operation, sigmoid(·) represents an activation function, avg(·) represents global maximum average pooling, and max(·) represents global average pooling.

[0028]

[0029]

[0030] wherein, represents the visible light feature aggregated by the spatial attention, represents the infrared feature aggregated by the spatial attention;

[0031] The two spatial attention maps, the channel attention map and the initial input feature are added to obtain a cross-modal feature perfect attention map F if , which is represented as:

[0032]

[0033] The cross-modal feature perfect attention map is obtained by capturing the complementary information of different modalities through the channel-by-channel feature correction and the spatial-by-spatial feature aggregation.

[0034] Further, the cross-layer feature aggregation module includes two convolution kernels of 1x1 convolution, splicing operation, element-by-element multiplication operation, element-by-element addition operation, two different scale features obtain the correlation feature map through the 1x1 convolution operation, and the correlation feature map is multiplied by the input feature and then the corresponding elements are added, and finally the complementary feature F3 is obtained, wherein one convolution is used to adjust the input channel number, and the other convolution is used to obtain the correlation feature map.

[0035] Further, the stepwise fusion layer includes sigmoid operation, element multiplication operation and element addition operation.

[0036] The input of the stepwise fusion layer is the infrared path feature, the visible light path feature and the complementary feature, the infrared path feature and the visible light path feature are respectively obtained through the sigmoid operation to obtain the weight map, and then the weight coefficients of the infrared and visible light paths are obtained according to the weight maps of the infrared path feature and the visible light path feature and , which is represented as:

[0037]

[0038]

[0039] λ represents a hyperparameter, which is used to prevent the denominator from being 0, F1 represents the infrared feature, and F2 represents the visible light feature.

[0040] The weight coefficients are multiplied by the corresponding path features to obtain the pre-fusion feature F' out , which is represented as:

[0041]

[0042] wherein, is represented as element-by-element multiplication.

[0043] The complementary features F3 output by the cross-level feature aggregation module and the pre-fusion features F' out Fusion is performed to obtain a fused image feature.

[0044] Further, the one-way cascaded decoder includes four residual modules with the same structure connected in sequence, and the input channel number of the four residual modules is 128, 64, 32, and 16 in sequence, and the output channel number is 64, 32, 16, and 1 in sequence.

[0045] The residual module includes a main path and a residual path, the main path includes a convolution with a convolution kernel of 1*1, BN, a LeakyReLU activation function, and a convolution with a convolution kernel of 3*3, and the residual path includes a DSConv.

[0046] First, the input feature is sequentially subjected to a 1*1 convolution, BN, LeakyReLU, 3*3 convolution, and BN operation to obtain a learnable convolution feature; second, the input feature is subjected to a DSConv operation to obtain a gradient amplitude information feature, and the learnable convolution feature and the gradient amplitude information feature are added and subjected to a LeakyReLU operation to obtain an output feature.

[0047] The input channel of the residual module of the first layer inputs the output channel fused image feature, and the fourth layer residual module outputs a fused image.

[0048] Further, the activation function of the fourth layer residual module is a hyperbolic tangent function.

[0049] Further, the training of the infrared and visible light image fusion model based on feature interaction and autoencoder includes:

[0050] 32 pairs of images are selected from the TNO dataset as a dataset, the image gray value is converted to [-1, 1], the image is cropped with a window of 128*128, the step size is set to 32, and finally 6184 pairs of image blocks are obtained as a training set;

[0051] The loss function L is set total , which is represented as:

[0052] L total =λ1L SSIM +λ2L patchNCE +λ3L detail

[0053] Wherein, λ1, λ2 and λ3 are hyperparameters, L SSIM is a structural similarity loss, L patchNCE is a contrast loss, and L detail is a texture detail loss.

[0054] According to the training set and the loss function Ltotal The initial feature interaction and autoencoder-based infrared and visible light image fusion model is trained, and the Adam optimizer is used to update the network model parameters in the training process until the training is completed, and the trained feature interaction and autoencoder-based infrared and visible light image fusion model is obtained.

[0055] An infrared and visible light image fusion device comprises:

[0056] An acquisition module is configured to acquire an infrared image and a visible light image of a target object.

[0057] A preprocessing module is configured to perform grayscale and data enhancement preprocessing operations on the infrared image and the visible light image to obtain preprocessed infrared and visible light images.

[0058] A fusion module is configured to input the preprocessed infrared and visible light images into a pre-trained feature interaction and autoencoder-based infrared and visible light image fusion model to obtain a fusion image.

[0059] The feature interaction and autoencoder-based infrared and visible light image fusion model comprises an encoder, a stepwise fusion layer, and a one-way cascaded decoder.

[0060] The encoder is configured to extract infrared features, visible light features, and complementary features of the preprocessed infrared and visible light images.

[0061] The stepwise fusion layer is configured to obtain information weight maps of the respective features by subjecting the extracted infrared features, visible light features, and complementary features to a sigmoid activation function, and adaptively fuses the infrared features, visible light features, and complementary features by corresponding weight coefficients to obtain a fusion image feature.

[0062] The one-way cascaded decoder is configured to obtain a fusion image by subjecting the fusion image feature to four residual blocks.

[0063] The present application has the following beneficial effects:

[0064] 1. The present application introduces a gradient residual module in the infrared path and the visible light path of the encoder, integrates learnable convolution features and gradient amplitude information, extracts spatial fine-grained detail information, and further improves the representation ability of texture detail information.

[0065] 2. The present application develops an interactive feature improvement module, which corrects information of different modalities from the channel dimension, aggregates interactive information from the spatial dimension, utilizes the relationship between channels and spaces, makes each other pay more attention to the complementary information of the other party, suppresses the influence of noise introduced by multi-modalities, and thus better realizes cross-modal feature interaction.

[0066] 3、The application develops a cross-layer feature aggregation module, gradually extracts the frequency correlation between the image features of adjacent network layers, and uses the relevant feature map to enhance the image features and aggregate the complementary information on the interactive compensation path.

[0067] 4、The application develops a stepwise fusion module, which adaptively fuses the pixel intensity information of the infrared image and the texture detail information of the visible light through the weight coefficients on different paths, and further improves the fusion performance of the model.

[0068] 5、The fusion effect of the application is significantly improved, and the model has good generalization ability, so the method can be applied to multi-modal image and medical image fusion, and has high application value in the field of image fusion. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 The fusion network schematic diagram of the method of the application is shown in the figure;

[0070] Figure 2 The schematic diagram of the residual module in the decoder is shown in the figure;

[0071] Figure 3 The schematic diagram of the interactive feature improvement module in the encoder is shown in the figure;

[0072] Figure 4 The schematic diagram of the cross-layer feature aggregation module in the encoder is shown in the figure;

[0073] Figure 5 The schematic diagram of the stepwise fusion module of the application is shown in the figure;

[0074] Figure 6 It is the first group of normal light conditions under the fusion result comparison figure;

[0075] Figure 7 It is the second group of low light conditions under the fusion result comparison figure. DETAILED DESCRIPTION

[0076] The application will be further described below in combination with the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0077] An infrared and visible light image fusion method, comprising the following steps:

[0078] S1, as Figure 1As shown, an infrared and visible image fusion model framework based on feature interaction and autoencoder is constructed; the model framework mainly consists of an encoder, a stepped fusion layer and a one-way cascaded decoder; the encoder includes three different branches, i.e. an infrared path, a visible path and an interaction compensation path, through which infrared features, visible light features and complementary features required for the fusion image are extracted, so that the fusion image contains richer source image information.

[0079] The encoder includes an infrared path, a visible path and an interaction compensation path;

[0080] The infrared path and the visible path each include a first convolution group, a second convolution group, a first gradient residual block, a second gradient residual block, a third gradient residual block and a fourth gradient residual block;

[0081] The first convolution group includes four convolutional fasts connected in turn, namely Conv1, Conv2, Conv3, Conv4, the input of Conv1 is the visible light image I v , Conv4 outputs shallow visible light image features, and the shallow visible light image features are outputted as visible light image features after passing through the first gradient residual block and the second gradient residual block in turn;

[0082] The second convolution group includes four convolutional fasts connected in turn, namely Conv2, Conv4, Conv6, Conv8, the input of Conv5 is the infrared image I i , Conv8 outputs shallow infrared image features, and the shallow infrared image features are outputted as infrared image features after passing through the third gradient residual block and the fourth gradient residual block in turn;

[0083] The interaction compensation path includes four interactive feature improvement modules and three cross-level feature aggregation modules, which are respectively denoted as: a first interactive feature improvement module, a second interactive feature improvement module, a third interactive feature improvement module, a fourth interactive feature improvement module, a first cross-level feature aggregation module, a second cross-level feature aggregation module and a third cross-level feature aggregation module;

[0084] The first interactive feature improvement module, the second interactive feature improvement module, the first cross-level feature aggregation module, the third interactive feature improvement module, the second cross-level feature aggregation module, the fourth interactive feature improvement module and the third cross-level feature aggregation module are connected in turn; the output of the first cross-level feature aggregation module is also connected to the first cross-level feature aggregation module, the output of the first cross-level feature aggregation module is also connected to the second cross-level feature aggregation module, and the output of the second cross-level feature aggregation module is also connected to the third cross-level feature aggregation module;

[0085] The input of the first interactive feature perfecting module is connected with the output of Conv2 and Conv6 respectively, the input of the second interactive feature perfecting module is connected with the output of Conv4 and Conv8 respectively, the input of the third interactive feature perfecting module is connected with the first gradient residual block and the third gradient residual block respectively, and the input of the fourth interactive feature perfecting module is connected with the second gradient residual block and the fourth gradient residual block respectively.

[0086] The input channel numbers of Conv1, Conv3, Conv5 and Conv7 in the first convolution group are 1, 16, 1, 16 respectively, and the output channel numbers are 1, 16, 1, 16 respectively, each convolution block contains a convolution kernel of 1x1, BN and LeakyReLU activation function, and the input feature is output after passing through 1x1 convolution, BN (Batch Normalization) and LeakyReLU operation in turn;

[0087] The input channel numbers of Conv2, Conv4, Conv6 and Conv8 in the second convolution group are 1, 16, 1, 16 respectively, and the output channel numbers are 16, 16, 16, 16 respectively, each convolution block contains a convolution kernel of 3x3, BN and LeakyReLU activation function, and the input feature is output after passing through 3x3 convolution, BN and LeakyReLU operation in turn;

[0088] The first gradient residual block and the third gradient residual block, and the second gradient residual block and the third gradient residual block all include a main path and a residual path, wherein the input and output channel numbers of the first gradient residual block and the third gradient residual block are 16, 32 respectively; the input and output channels of the second gradient residual block and the third gradient residual block are 32, 64 respectively; the main path contains a convolution kernel of 1x1, BN, LeakyReLU activation function, a convolution kernel of 3x3, and the residual path includes a Scharr gradient operator and a DSConv (Depthwise Separable Convolution), first, the input feature is obtained after passing through 1x1 convolution, BN, LeakyReLU activation function, convolution kernel of 3x3 and BN operation in turn; second, the input feature is obtained after passing through Scharr gradient operator and DSConv operation in turn, the two features are added and then passed through LeakyReLU to obtain the output feature. The main branch of the gradient residual block is composed of 1x1 convolution and 3x3 convolution, and the residual branch is composed of Scharr gradient operator and depthwise separable convolution, which integrates the learnable convolution feature and the gradient amplitude information, and improves the spatial fine-grained information representation ability.

[0089] S2, as Figure 3As shown, the interactive feature perfecting module on the interaction compensation path and the cross-level feature aggregation module are designed. The interaction compensation path consists of 4 interactive feature perfecting modules and 3 cross-level feature aggregation modules. Among them, the feature interaction perfecting module includes two stages of channel attention feature correction and spatial attention feature aggregation, thereby obtaining the complementary information of the two modal images. For the low-level to high-level features of infrared and visible light images, after the channel correction attention, the channel correction attention map of the two modal images is obtained; after the spatial aggregation attention, the spatial aggregation attention map is obtained, the channel and spatial attention features are fused to obtain the complementary features, and after the cross-level feature aggregation module of the adjacent interactive features, the interactive compensation features are finally obtained.

[0090] S21, channel attention feature correction. The input infrared feature F i and the visible light feature F v are multiplied element by element, and then spliced with the input. Then the spliced features are respectively subjected to maximum pooling, average pooling and multilayer perception in the channel dimension, the two output features are added element by element and input into the activation function sigmoid to obtain the channel correction weighting coefficient, and finally multiplied by the spliced features to obtain the channel correction attention map is expressed as Wherein, avg(·) represents global maximum average pooling, and max(·) represents global average pooling.

[0091] S22, spatial attention feature aggregation. F i is respectively subjected to maximum pooling and average pooling operation in the spatial dimension, and then subjected to splicing operation, F v Similarly, the features of the two modalities are spliced, and the spliced features are multiplied with F i and F v to obtain the corresponding spatial aggregation attention map F i s , is respectively expressed as and

[0092] S23, channel and spatial attention feature fusion. The two spatial attention maps and the channel attention map are added to the initial input feature to obtain the cross-modal feature perfecting attention map F f , which is expressed as By channel correction and spatial aggregation, the complementary information of different modalities is captured, and the single modality individual feature representation ability is enhanced.

[0093] S24, as Figure 4As shown, a cross-level feature aggregation module is designed. The cross-level feature aggregation module includes two ordinary convolutions, two different scale features are obtained through 1x1 convolution operation to obtain correlation feature maps, and the correlation feature maps are multiplied with the input features and then added element by element to obtain the aggregated features. One of the convolutions is used to adjust the input channel number, and the other convolution is used to obtain the correlation feature map. By referencing the cross-level feature aggregation module and the interactive feature refinement module, all intermediate layer features are used for feature interaction and refinement, improving the ability to complement information and suppress redundant information.

[0094] S3, as shown, Figure 5 a stepwise fusion module is designed. The fusion layer adaptively fuses the semantic information of the three different branches F n (n = 1, 2, 3) to obtain fusion features.

[0095] S31, stepwise fusion branch. First, the features of the infrared branch and the visible light branch are fused, where the weight coefficients and can be represented as where λ is set to 1e-8.

[0096] Then, the weight coefficients are multiplied by the corresponding branch features to obtain the pre-fusion branch features F' out , which can be represented as where is an element-wise multiplication. The interaction compensation branch F3 and the fusion branch operate in the same way, and finally the fusion feature map is obtained.

[0097] S4, as shown, Figure 1 and Figure 2 a one-way cascaded decoder is designed. The decoder includes four residual modules, and the input channel numbers of the four residual blocks are 128, 64, 32, and 16 in turn, and the output channels are 64, 32, 16, and 1. Among them, the main branch of the residual block is composed of 1x1 convolution and 3x3 convolution, and the residual branch is composed of depth separable convolution, and the two branches are finally output through element-wise addition and activation function LeakyReLU. The fusion features are input into the decoder, sequentially pass through the four residual modules, and complete the fusion image reconstruction. Here, the activation function of the last residual block is the hyperbolic tangent function (Tanh).

[0098] S5, network model training. The TNO image is used as the training data set, the input source images of the two modalities are grayed and data enhanced, the structural similarity, fine-grained detail loss and contrast loss are used to train the network model, and the parameters of the network model are obtained.

[0099] S51, Dataset Preprocessing. 32 pairs of images were selected from the TNO dataset as the dataset. The grayscale values ​​of the images were converted to [-1, 1]. The images were cropped using a 128×128 window with a stride of 32, resulting in 6184 pairs of image patches as the training set.

[0100] S52, Set the loss function. The loss function formula can be expressed as L total =λ1L SSIM +λ2L patchNCE +λ3L detail , where λ1, λ2 and λ3 are all hyperparameters.

[0101] The above structural similarity loss L SSIM It can evaluate the quality of fused images in terms of brightness, contrast, and structure. Structural similarity is expressed as... Where μ represents the pixel mean, σ xy Let C1 represent the covariance and σ represent the variance. C1 and C2 are used to avoid... The minimum value set close to zero is typically 0.01. 2 0.03 2 L SSIM Represented as:

[0102]

[0103] Where I x I y and I f These represent the visible light image, infrared image, and fused image, respectively. W represents the sliding window with a step size of 1, and P... i The value of pixel i is represented by m and n, which represent the size of the sliding window. In this invention, the window is set to 16×16.

[0104] The contrast loss L mentioned above patchNCE Represented as Where k represents the encoded feature sample, k + Let k represent a positive sample similar to k, k- represent a negative sample dissimilar to k, and τ represent the temperature coefficient, typically taken as τ = 0.07. S represents the number of sampling locations in the image feature layer, where s ∈ {1, 2, ..., S}, and D... s This represents any feature sequence in the channel space. By calculating the similarity between a certain region in the space and positive and negative samples, the most similar parts of the fused image and the source image are effectively preserved.

[0105] The aforementioned loss of texture detail L detail This allows the fused image to retain more fine-grained detail information, represented as... in denotes the Scharr gradient operator, H and W denote the image height and width, and ||·||1 denotes the L1 norm.

[0106] The Adam optimizer is used in the training process to update the network model parameters, the learning rate is set to 0.01, the Epoch is set to 20, and the Batchsize is set to 4.

[0107] Further, to verify the image fusion effect of the autoencoder trained by the above method, the trained autoencoder is verified in the embodiment of the application.

[0108] Specifically, in the test phase, 20 groups of images are selected from the M3FD dataset for testing and verification, and 7 typical methods are selected as comparison methods, including DenseFuse, U2Fusion, RFN-Nest, SEDRFuse, IFCNN, GANMcC and CSF. In addition, the quantitative evaluation indexes include information entropy (EN), average gradient (AG), spatial frequency (SF), mutual information (MI), standard deviation (SD) and visual fidelity (VIF), and the verification results include qualitative evaluation and quantitative evaluation.

[0109] (1) Qualitative evaluation. Figure 6 and Figure 7 The qualitative comparison results of two groups of images (road scene images under normal illumination and low illumination conditions) are given. By comparison, it can be found that the fusion method of the application has three advantages. First, the fusion effect can maximize the retention of the brightness information of the thermal radiation target in the infrared image. For typical infrared targets, such as Figure 6 , 7 the person in the present application has higher brightness and clearer edge profile target features than other methods. Second, the fusion result can retain the texture details and background information in the visible light image. For example Figure 6 the license plate of the car and Figure 7 the garbage can, the fusion method of the application can retain clearer detail information and more obvious background information than other methods. Finally, the fusion result has better high contrast and better visual effect. Compared with the source image and other fusion results, the method of the application can better retain prominent target features and rich scene detail information, the image contrast is high, and it is more in line with the visual effect of the human eye.

[0110] (2) Quantitative evaluation. Table 1 shows the objective comparison results of 20 groups of images of the M3FD dataset. The optimal average and the suboptimal average are marked with bold and underlined respectively. It can be seen that the present application achieves the optimal average of EN, MI, SD and VIF, and the suboptimal value of the index SF. The objective experiment shows that the present application has better fusion performance than other methods. The maximum value EN and MI show that the fusion image of the present application effectively obtains a large amount of infrared thermal radiation information and texture detail features from the source images, because the fusion method of the present application constructs an interactive feature perfect module to obtain complementary information from different modalities. The maximum value SD shows that the fusion method of the present application constructs a stepwise fusion strategy to adaptively fuse the features of different modalities. The maximum value VIF shows that the fusion image of the present application contains rich fine-grained information, better contrast and visual effect, because the fusion method of the present application constructs a gradient residual block to establish a long-distance dependency relationship of local features and obtain fine-grained detail information from the image.

[0111] Table 1 Quantitative comparison results of 20 groups of images of the M3FD dataset

[0112]

[0113] Correspondingly, the present application also provides an infrared and visible light image fusion device, comprising:

[0114] An acquisition module is configured to acquire infrared images and visible light images of a target object.

[0115] A preprocessing module is configured to perform grayscale and data enhancement preprocessing operations on the infrared images and the visible light images to obtain preprocessed infrared images and visible light images.

[0116] A fusion module is configured to input the preprocessed infrared images and the visible light images into a pre-trained infrared and visible light image fusion model based on feature interaction and autoencoder to obtain a fusion image.

[0117] The infrared and visible light image fusion model based on feature interaction and autoencoder comprises an encoder, a stepwise fusion layer and a one-way cascaded decoder.

[0118] The encoder is configured to extract infrared features, visible light features and complementary features of the preprocessed infrared images and the visible light images.

[0119] The stepwise fusion layer is configured to obtain information weight maps of the extracted infrared features, visible light features and complementary features through a sigmoid activation function, and adaptively fuse the infrared features, visible light features and complementary features through corresponding weight coefficients to obtain a fusion image feature.

[0120] The one-path cascaded decoder is configured to pass the fused image feature through four residual blocks to obtain a fused image.

[0121] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) embodying computer readable program code.

[0122] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It should be understood that each flow and / or block in the flowchart and / or block diagrams, and a combination of flows and / or blocks in the flowchart and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram.

[0123] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufacture product including instruction means, which implement the functions specified in the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram.

[0124] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operational steps are performed on the computer or other programmable data processing apparatus to generate a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowchart and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the function specified in the flowchart and / or block diagram.

[0125] The above description is only preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.

Claims

1. An infrared and visible image fusion method, characterized by, The method comprises the following steps: acquiring an infrared image and a visible light image of a target object; performing a preprocessing operation of graying and data enhancement on the infrared image and the visible light image to obtain preprocessed infrared and visible light images; inputting the preprocessed infrared and visible light images into a pre-trained infrared and visible light image fusion model based on feature interaction and autoencoder to obtain a fusion image; the infrared and visible light image fusion model based on feature interaction and autoencoder comprises an encoder, a stepwise fusion layer and a one-way cascaded decoder; the encoder is configured to extract infrared features, visible light features and complementary features of the preprocessed infrared and visible light images; the stepwise fusion layer is configured to obtain information weight maps of the extracted infrared features, visible light features and complementary features through a sigmoid activation function, and adaptively fuse the infrared features, visible light features and complementary features through corresponding weight coefficients to obtain a fusion image feature; the one-way cascaded decoder is configured to pass the fusion image feature through four residual blocks to obtain a fusion image; the encoder comprises an infrared path, a visible path and an interactive compensation path; the infrared path and the visible light path each comprise a first convolution group, a second convolution group, a first gradient residual block, a second gradient residual block, a third gradient residual block and a fourth gradient residual block; The first convolution group comprises four convolutional layers connected in sequence, namely Conv1, Conv2, Conv3 and Conv4, wherein the input of Conv1 is the visible light image I v Conv4 outputs a shallow visible light image feature, and the shallow visible light image feature is sequentially input into a first gradient residual block and a second gradient residual block to output a visible light image feature; The second convolution group includes four sequentially connected convolutional layers, Conv2, Conv4, Conv6, and Conv8, and an input of Conv5 is the infrared image I i Conv8 outputs a shallow infrared image feature, and the shallow infrared image feature is sequentially input into a third gradient residual block and a fourth gradient residual block to obtain an infrared image feature. the interactive compensation path comprises four interactive feature improvement modules and three cross-layer feature aggregation modules, which are respectively denoted as a first interactive feature improvement module, a second interactive feature improvement module, a third interactive feature improvement module, a fourth interactive feature improvement module, a first cross-layer feature aggregation module, a second cross-layer feature aggregation module and a third cross-layer feature aggregation module; the first interactive feature improvement module, the second interactive feature improvement module, the first cross-layer feature aggregation module, the third interactive feature improvement module, the second cross-layer feature aggregation module, the fourth interactive feature improvement module and the third cross-layer feature aggregation module are sequentially connected; the output of the first cross-layer feature aggregation module is further connected to the first cross-layer feature aggregation module, the output of the first cross-layer feature aggregation module is further connected to the second cross-layer feature aggregation module, and the output of the second cross-layer feature aggregation module is further connected to the third cross-layer feature aggregation module; the input of the first interactive feature improvement module is connected to the output of Conv2 and Conv6, the input of the second interactive feature improvement module is connected to the output of Conv4 and Conv8, the input of the third interactive feature improvement module is connected to the first gradient residual block and the third gradient residual block, and the input of the fourth interactive feature improvement module is connected to the second gradient residual block and the fourth gradient residual block.

2. The infrared and visible image fusion method of claim 1, wherein, The input channel numbers of Conv1, Conv3, Conv5 and Conv7 in the first convolution group are 1, 16, 1, 16 respectively, and the output channel numbers are 1, 16, 1, 16 respectively, each convolution block contains a convolution kernel of 1*1, BN and LeakyReLU activation function, and the input feature is output after sequentially passing through 1*1 convolution, BN and LeakyReLU operation; The input channel numbers of Conv2, Conv4, Conv6 and Conv8 in the second convolution group are 1, 16, 1, 16 respectively, and the output channel numbers are 16, 16, 16, 16 respectively, each convolution block contains a convolution kernel of 3*3, BN and LeakyReLU activation function, and the input feature is output after sequentially passing through 3*3 convolution, BN and LeakyReLU operation; The first gradient residual block and the third gradient residual block, and the second gradient residual block and the third gradient residual block all include a main path and a residual path, wherein the input and output channel numbers of the first gradient residual block and the third gradient residual block are 16, 32 respectively; the input and output channels of the second gradient residual block and the third gradient residual block are 32, 64 respectively; the main path contains a convolution kernel of 1*1, BN, LeakyReLU activation function, and a convolution kernel of 3*3, and the residual path includes a Scharr gradient operator and a DSConv, first, the input feature is sequentially processed by 1*1 convolution, BN, LeakyReLU activation function, a convolution kernel of 3*3 and BN to obtain a learnable convolution feature; second, the input feature is sequentially processed by the Scharr gradient operator and the DSConv to obtain a gradient amplitude information feature, and the two features are added and then processed by LeakyReLU to obtain an output feature.

3. The method of fusing infrared and visible light images according to claim 1, wherein, The interactive feature improvement module includes two stages of channel attention feature correction and spatial attention feature aggregation, the channel attention feature correction includes element-wise multiplication operation, splicing operation, MLP, average pooling operation, maximum pooling operation, sigmoid operation and element-wise addition operation, and the spatial attention feature aggregation includes element-wise multiplication operation, splicing operation, average pooling operation, maximum pooling operation and element-wise addition operation; The channel attention feature correction stage adopts an element-by-element multiplication method to input the infrared feature F ir and the visible light feature F vis are interacted, and then spliced with the infrared feature F ir and the visible light feature F vis to obtain a feature F c ; after the feature F c undergoes global maximum pooling, global average pooling and MLP operations in the channel dimension, an output feature is obtained, the output feature is element-by-element added and subjected to a sigmoid activation function operation to obtain a channel calibration weighting coefficient, and finally the spliced feature F c is multiplied to obtain a channel correction attention map , which is represented as: ； wherein, represents a concatenation operation, represents an activation function, represents a global max pooling, represents a global average pooling; ； ； wherein, represents the visible light feature aggregated by spatial attention, represents the infrared feature aggregated by spatial attention; The two spatial attention maps, the channel attention map and the initial input feature are added to obtain a cross-modal feature perfect attention map F if is represented as: ； Through channel-by-channel feature correction and spatial feature aggregation, complementary information of different modalities is captured to obtain a cross-modal feature improvement attention map.

4. The method of fusing infrared and visible light images according to claim 3, wherein, The cross-level feature aggregation module includes two convolution kernels of 1*1, splicing operation, element-wise multiplication operation and element-wise addition operation, two different scale features are processed by 1*1 convolution operation to obtain a correlation feature map, and the correlation feature map is multiplied by the input feature and then the corresponding elements are added to finally obtain a complementary feature, wherein one convolution is used to adjust the input channel number, and the other convolution is used to obtain the correlation feature map.

5. The method of fusing infrared and visible light images according to claim 4, wherein, The stepwise fusion layer includes sigmoid operation, element multiplication operation and element addition operation; The input of the stepped fusion layer is infrared path features, visible light path features and complementary features, the infrared path features and the visible light path features are respectively subjected to sigmoid operation to obtain weight maps, and then the weight coefficients of the infrared path and the visible light path are obtained according to the weight maps of the infrared path features and the visible light path features and is expressed as: ； ； ​ represents a hyperparameter for preventing the denominator from being zero, F 1 represents an infrared feature, F 2 represents a visible light feature; The weight coefficient is multiplied by the corresponding path feature to obtain a pre-fusion feature , is expressed as: ； wherein is represented as an element-wise multiplication; complementary features output by the cross-level feature aggregation module F 3 and pre-fusion features perform fusion to obtain a fused image feature.

6. The infrared and visible image fusion method of claim 5, wherein, The one-way cascaded decoder comprises four residual modules with the same structure connected in sequence, the input channel number of the four residual modules is 128, 64, 32 and 16 in sequence, and the output channel number is 64, 32, 16 and 1 in sequence; The residual module comprises a main path and a residual path, the main path comprises a convolution with a convolution kernel of 1*1, BN, a LeakyReLU activation function and a convolution with a convolution kernel of 3*3, and the residual path comprises a DSConv; Firstly, the input feature is sequentially subjected to 1*1 convolution, BN, LeakyReLU, 3*3 convolution and BN operation to obtain a learnable convolution feature; secondly, the input feature is subjected to DSConv operation to obtain a gradient amplitude information feature, and the learnable convolution feature and the gradient amplitude information feature are added and subjected to LeakyReLU operation to obtain an output feature; The input channel of the residual module of the first layer inputs the output channel fusion image feature, and the fourth layer residual module outputs the fusion image.

7. The infrared and visible image fusion method of claim 6, wherein, The activation function of the residual module of the fourth layer is a hyperbolic tangent function.

8. The infrared and visible image fusion method of claim 1, wherein, The training of the infrared and visible image fusion model based on feature interaction and autoencoder comprises: 32 pairs of images are selected from a TNO dataset as a dataset, the image gray value is converted to [-1, 1], the image is cropped with a window of 128*128, the step is set to 32, and finally 6184 pairs of image blocks are obtained as a training set; Setting a loss function L total is represented as: ； wherein, λ 1, λ 2 and λ 3 are hyperparameters, L SSIM is a structural similarity loss, L patchNCE is a contrastive loss, L detail is a texture detail loss; According to the training set and the loss function L total The initial feature interaction and autoencoder-based infrared and visible light image fusion model is trained, and the Adam optimizer is used in the training process to update the network model parameters until the training is completed, and the trained feature interaction and autoencoder-based infrared and visible light image fusion model is obtained.

9. An infrared and visible image fusion apparatus characterized by comprising: Comprise: An acquisition module is configured to acquire an infrared image and a visible light image of a target object; A preprocessing module is configured to perform grayscale and data enhancement preprocessing operations on the infrared image and the visible light image to obtain preprocessed infrared and visible light images; A fusion module is configured to input the preprocessed infrared and visible light images into a pre-trained infrared and visible light image fusion model based on feature interaction and an autoencoder to obtain a fusion image; The infrared and visible image fusion model based on feature interaction and an autoencoder comprises an encoder, a stepwise fusion layer and a one-way cascaded decoder; The encoder is configured to extract infrared features, visible light features and complementary features of the preprocessed infrared and visible light images; The stepwise fusion layer is configured to obtain information weight maps of the respective features by subjecting the extracted infrared features, visible light features and complementary features to sigmoid activation functions, and adaptively fusing the infrared features, visible light features and complementary features by corresponding weight coefficients to obtain a fusion image feature; The one-way cascaded decoder is configured to pass the fusion image feature through four residual blocks to obtain a fusion image; The encoder comprises an infrared path, a visible path and an interaction compensation path; The infrared path and the visible light path each comprise a first convolution group, a second convolution group, a first gradient residual block, a second gradient residual block, a third gradient residual block and a fourth gradient residual block; The first convolution group comprises four convolutional layers connected in sequence, namely Conv1, Conv2, Conv3 and Conv4, wherein the input of Conv1 is the visible light image I v Conv4 outputs a shallow visible light image feature, and the shallow visible light image feature is sequentially input into a first gradient residual block and a second gradient residual block to output a visible light image feature. The second convolution group includes four sequentially connected convolutional layers, Conv2, Conv4, Conv6, and Conv8, and an input of Conv5 is the infrared image I i Conv8 outputs a shallow infrared image feature, and the shallow infrared image feature is sequentially input into a third gradient residual block and a fourth gradient residual block to obtain an infrared image feature. The interaction compensation path comprises four interactive feature perfecting modules and three cross-level feature aggregation modules, which are respectively denoted as: a first interactive feature perfecting module, a second interactive feature perfecting module, a third interactive feature perfecting module, a fourth interactive feature perfecting module, a first cross-level feature aggregation module, a second cross-level feature aggregation module and a third cross-level feature aggregation module; The first interactive feature perfecting module, the second interactive feature perfecting module, the first cross-level feature aggregation module, the third interactive feature perfecting module, the second cross-level feature aggregation module, the fourth interactive feature perfecting module and the third cross-level feature aggregation module are sequentially connected; the output of the first cross-level feature aggregation module is further connected to the first cross-level feature aggregation module, the output of the first cross-level feature aggregation module is further connected to the second cross-level feature aggregation module, and the output of the second cross-level feature aggregation module is further connected to the third cross-level feature aggregation module; The input of the first interactive feature perfecting module is connected to the output of Conv2 and Conv6 respectively, the input of the second interactive feature perfecting module is connected to the output of Conv4 and Conv8 respectively, the input of the third interactive feature perfecting module is connected to the first gradient residual block and the third gradient residual block respectively, and the input of the fourth interactive feature perfecting module is connected to the second gradient residual block and the fourth gradient residual block respectively.

Images