Infrared and visible image fusion method based on feature difference compensation and fusion

Abstract

The application discloses an infrared and visible light image fusion method based on feature difference compensation and fusion. In the feature extraction stage, CNN and Transfomer are used to encode infrared and visible light images respectively. The CNN module is used to extract image pixel intensity distribution information, and the Transfomer module is used to extract image detail texture information. A cross-modal difference compensation and fusion module is designed, complementary information of each stage is extracted through the cross-modal difference compensation module, and then the complementary information is integrated into the context global information by combining the cross-modal feature fusion module. A progressive difference fusion decoder is constructed by using the cross-modal difference compensation and fusion module, different modalities and different scales of the original image are fused through a progressive manner, so that the fused image has strong thermal radiation information and clear texture information. The method has significant effect and excellent generalization ability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion, and in particular to an infrared and visible light image fusion method based on feature difference compensation and fusion. BACKGROUND

[0002] Image fusion aims to generate a fusion image with rich information by combining important information from original images. Among them, infrared and visible light image fusion and medical image fusion are two challenging subcategories of multi-modal image fusion. Infrared and visible light image fusion aims to generate a fusion image that retains the thermal radiation information in the infrared image and the detailed texture information in the visible light image. The fusion image can avoid the shortcomings of visible light images being sensitive to lighting conditions and infrared images having high noise and low resolution. According to certain rules, they are fused into a new image, so that the multi-modal image after fusion can improve the visual perception ability and scene representation ability of target information, so as to better complete tasks such as target detection, tracking, semantic segmentation and multispectral pedestrian recognition.

[0003] Traditional fusion methods include methods based on multi-scale decomposition, sparse representation, total space, optimization and hybrid methods. These methods represent source images and then obtain fusion images according to different fusion strategies. Although these methods have achieved good fusion performance in many fusion tasks, their shortcomings are also very significant, including lack of understanding of modal differences, fusion only in inherent common features and increasing complexity of hand-crafted fusion rules.

[0004] In recent years, due to the powerful feature extraction capability of deep networks, many image fusion methods based on deep learning have been proposed. These methods are divided into automatic encoder (AE) based, convolutional neural network (CNN) based and generative adversarial network (GAN) based methods. They design modules, fusion rules, etc. in the three fusion steps of feature extraction, feature fusion and image reconstruction to obtain fusion images with both thermal radiation information and clear texture. In order to more thoroughly obtain the information extracted by features, the literature extracts different information through a double-path network. The literature fuses the fusion features of different modalities by designing fusion rules. The literature cascades the features output by the encoder and the decoder through a skip connection to strengthen the image reconstruction features.

[0005] In the above method, there are three shortcomings. First, the existing method often only outputs the last layer as the extracted information of the original image when performing feature extraction, and most of them only rely on convolution operation to extract local features without considering their global dependency, which cannot effectively maintain infrared targets and visible light details at the same time. Second, when extracting features from two input original images, since the two modal images have different types of features, most methods often ignore whether complementary features can be extracted and integrated between different modalities when performing feature fusion. Finally, most methods often use simple splicing, addition and other operations when performing feature fusion, and there is no module that can overcome semantic inconsistency, effectively integrate different scale features and continuously improve the quality of fused features in various network scenarios, resulting in a large deviation in the aggregation scale of context information. Therefore, it is necessary to propose an infrared and visible light image fusion method based on feature differential compensation and fusion to solve the above problems. SUMMARY

[0006] The purpose of the present application is to provide an infrared and visible light image fusion method based on feature differential compensation and fusion to solve the problem that the existing image fusion method often cannot effectively maintain infrared targets and visible light details at the same time, ignores whether complementary features can be extracted and integrated between different modalities, and has a large deviation in the aggregation scale of context information.

[0007] The present application provides an infrared and visible light image fusion method based on feature differential compensation and fusion, comprising:

[0008] using CNN and Transformer encoders to extract multi-scale features of infrared and visible light images respectively;

[0009] inputting the multi-scale features into four cross-modal differential compensation and fusion modules for feature fusion;

[0010] using a progressive differential and fusion decoding module to take the output features of the highest level cross-modal differential compensation and fusion module as the input of the lower level cross-modal differential compensation and fusion module, continuously fusing low-level modal features to obtain a fused image.

[0011] Further, using CNN and Transformer encoders to extract multi-scale features of infrared and visible light images respectively, comprising:

[0012] extracting shallow features F vi and F ir from visible light and infrared original images by feature encoders based on CNN encoding and Transformer encoding respectively.; wherein a feature encoder based on CNN coding is used to extract the infrared image branch, and a feature encoder based on Restormer Block coding is used to extract the visible light image branch to obtain the detailed texture information and structural information of the image, represented as follows:

[0013]

[0014] wherein, and respectively represent the deep features extracted from the infrared and visible light images, and the superscript i = 0, 1, 2, 3 corresponds to the output features of the i-th layer; when i = 0, it represents the output features extracted by the shallow layer; when i = 1, 2, 3, it represents the multi-scale features of three stages obtained by CNN coding and Transfomer coding respectively; I vi and I ir respectively represent the visible light image and the infrared image; E(.) represents the original image.

[0015] Further, the multi-scale features are input into the step of feature fusion in the four cross-modal differential compensation and fusion modules, which include a cross-modal differential compensation module and a cross-modal feature fusion module, and the cross-modal differential compensation module is defined as:

[0016]

[0017] wherein, ⊕ represents element-wise summation, represents element-wise multiplication, δ(·) represents the Sigmod function, Conv(.) represents the convolution operation, GMP(·) and GAP(·) represent global maximum pooling and global average pooling respectively; formula (2) represents that after global maximum pooling, global average pooling and convolution, the attention maps are normalized by the sigmoid function to generate corresponding attention weights, the supplementary features are multiplied by the attention weights, and the results are added to the original features to obtain the modal supplementary information.

[0018] Further, the multi-scale features are input into the step of feature fusion in the four cross-modal differential compensation and fusion modules, and the feature fusion includes:

[0019] The output features of the fourth layer and are fused to obtain the initial fusion features

[0020] are respectively subjected to channel attention and spatial attention to obtain initial fusion attention weight maps, and the initial weight maps

[0021] ​The Sigmoid function is used to normalize the generated attention weights;

[0022] The attention weights and are multiplied respectively and then fused to obtain intermediate fusion features Meanwhile, the intermediate fusion features and are taken as the input of the next cross-modal feature fusion module.

[0023] Further, the cross-modal feature fusion module is defined as:

[0024]

[0025] wherein, ⊕ represents element-wise summation, represents element-wise product, p represents prelu activation function, Conv(.) represents convolution operation, GMP(·) represents global maximum pooling, represents intermediate fusion features, represents initial weight map, δ(·) represents Sigmod function, and represent infrared and visible output features respectively.

[0026] Further, the method further comprises:

[0027] The discriminator distinguishes the fusion result and the original image, and two same network structures are taken as the discriminator, the discriminator is composed of four 3x3 convolution layers, Leaky ReLu is used as the activation function after the first three convolution layers, and the hyperbolic tangent function is used in the last layer.

[0028] Further, the generator loss, which is represented as: G

[0029] L G =λ1L content +λ2L ssim +λ3L adv (6)

[0030] wherein, L G represents the total loss function, L content represents the content loss, L adv and L ssim represent the adversarial loss and the structural similarity loss respectively, and parameters λ1, λ2 and λ3 are balance parameters.

[0031] Further, an intensity loss Lint is introduced to constrain the similarity between the pixels of the fusion image and the original image, and the intensity loss L int is represented by equation 7:

[0032] ​

[0033] where I f denotes the fused image, I vi and I ir denote the visible and infrared images respectively; ||·||1denotes the L1 norm; Max(I vi , I ir ) denotes the element-wise maximum selection;

[0034] A texture loss L text is introduced to assist the intensity loss and is denoted as:

[0035]

[0036] where denotes denotes the Sobel operator used to detect the texture details in the image, which is used to calculate the gradient changes in the image; |·| denotes the absolute value; max(.) denotes the maximum value;

[0037] The content loss L content is composed of two parts, the intensity loss L int and the texture loss L text , and is denoted as:

[0038] L content = L int + L text (9)

[0039] The similarity measure L ssim is defined as follows:

[0040] L ssim = a(1-ssim(I f , I vi )) + (1-a)(1-ssim(I f , I ir )) (10)

[0041] where ssim(.) measures the average distance of the structural similarity of two images, a is the balance parameter; x vi and x ir denote the visible and infrared images respectively; I f denotes the fused image, I vi and I ir denote the visible and infrared images respectively;

[0042] L adv denotes the loss of the generator and is defined as:

[0043]

[0044] Further, the loss function of the two discriminator networks is defined as and As follows:

[0045]

[0046] Wherein, the first term and the second term represent Wasserstein distance estimation and gradient penalty respectively, and λ4 is a regularization parameter; D ir (.) and D vi (.) represent the infrared light discriminator and the visible light discriminator respectively; ||·||2 represents the L2 norm; N represents the number of images.

[0047] The beneficial effects of the present application are as follows: the infrared and visible light image fusion method based on feature difference compensation and fusion provided by the present application is composed of three parts of encoding-feature fusion-decoding. In the feature extraction stage, CNN and Transfomer are used to encode the infrared and visible light images respectively. Among them, the CNN module is used to extract the image pixel intensity distribution information, and the Transfomer module is used to extract the image detail texture information. A cross-modal difference compensation and fusion module is designed, the complementary information of each stage is extracted through the cross-modal difference compensation module, and then the complementary information is integrated into the context global information by combining the cross-modal feature fusion module. A progressive difference fusion decoder is constructed by using the cross-modal difference compensation and fusion module, and the cross-layer feature fusion of different modalities and different scales of the original image is performed in a progressive manner, so that the fused image has strong thermal radiation information and clear texture information; the method of the present application has significant effect and shows excellent generalization ability. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiments will be briefly introduced as follows: obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0049] Figure 1 is a flowchart of the infrared and visible light image fusion method based on feature difference compensation and fusion of the present application;

[0050] Figure 2 is an overall framework diagram of infrared and visible light image fusion;

[0051] Figure 3 is a cross-modal difference compensation and fusion module diagram;

[0052] Figure 4 is a comparison diagram of visualization results of different methods on the TNO data set;

[0053] Figure 5 is a comparison chart of the visualization results of different methods on the MRSR dataset;

[0054] Figure 6 is a comparison chart of the visualization results of different methods on the LLVIP dataset. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions will be described below in connection with the specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application. The technical solutions provided by the embodiments of the present application will be described in detail below in connection with the drawings.

[0056] Referring to Figures 1 to 3 , the present application provides an infrared and visible light image fusion method based on feature difference compensation and fusion, comprising:

[0057] S101, using CNN and Transformer encoder to extract multi-scale features of infrared and visible light images respectively.

[0058] Specifically, in the encoding stage, shallow features F vi and F ir are extracted from the visible light and infrared original images through two convolutional layers; then, respectively by CNN-based and Transformer-based feature encoders. Specifically, CNN is used to extract the infrared image branch, which helps to extract global context semantic information, while RestormerBlock-based is used to extract the visible light image branch to obtain image detail texture information and structure information. Denote them as:

[0059]

[0060] wherein, and denote the deep features extracted from the infrared and visible light images, respectively, and the superscript i=0,1,2,3 corresponds to the output features of the i-th layer; when i=0, it represents the output features of the shallow extraction; when i=1,2,3, it represents the multi-scale features of the three stages obtained by CNN encoding and Transformer encoding, respectively. vi and I ir denote the visible light image and the infrared image, respectively; E(.) denotes the original image.

[0061] The overall network framework of this invention consists of a generator and two discriminators. The generator aims to generate a fused image to deceive the discriminators; the discriminators distinguish between the fused image and the original infrared or visible light image; the model is trained repeatedly to make the fused result approximate the original image.

[0062] S102, the multi-scale features are input into four cross-modal difference compensation and fusion modules for feature fusion.

[0063] This invention designs a cross-modal differential compensation and fusion module. This module consists of two parts: a cross-modal differential compensation module and a cross-modal feature fusion module. The core idea of ​​the cross-modal differential compensation module is to fully extract and merge complementary information from multimodal images, while the core idea of ​​the cross-modal feature fusion module is to solve the problem of integrating context and initial features by adding local context information to the global context information within the attention module.

[0064] 1) Cross-modal differential compensation module. Since data from different modalities may contain incomplete or missing information, image complementation is achieved at the pixel level by extracting missing information from one modality and compensating for it in other modalities. Therefore, the cross-modal differential compensation module can be defined as:

[0065]

[0066] Where ⊕ represents element-wise summation, δ(·) represents element-wise multiplication, δ(·) represents the Sigmoid function, Conv(.) represents the convolution operation, and GMP(·) and GAP(·) represent global max pooling and global average pooling, respectively. Equation (2) indicates that after global max pooling, global average pooling, and convolution, these attention maps are normalized by the sigmoid function to generate corresponding attention weights. Then, the supplementary features are multiplied by the attention weights, and finally the result is added to the original features to obtain the modality supplementary information.

[0067] 2) Cross-modal feature fusion module. Simple initial feature integration methods as input significantly impact the quality of the fusion weights. A cross-modal fusion module can improve model performance, especially in tasks requiring joint decision-making across modalities. By fusing features from different modalities, the model can better understand the relationships between multimodal inputs. This invention's module is based on spatial and channel attention mechanisms, using the acquired weight map to measure the activity levels between different modalities at different phase scales.

[0068] First, the output features of the fourth layer will be analyzed. and The initial fusion features are obtained by performing fusion. Then The initial fusion attention weight map is obtained through channel attention and spatial attention respectively, and the initial weight map is obtained through channel addition Then the corresponding attention weights are normalized using the Sigmoid function, and finally After multiplication respectively, the intermediate fusion features are obtained At the same time, the is input into the next cross-modal feature fusion module. Therefore, the cross-modal feature fusion module can be defined as:

[0069]

[0070] Where, ⊕ represents element-wise summation, represents element-wise multiplication, p represents prelu activation function, Conv(.) represents convolution operation, GMP(·) represents global maximum pooling, represents intermediate fusion features, represents initial weight map, and δ(·) represents Sigmod function, and represent infrared and visible output features respectively.

[0071] S103, through the progressive difference and the fusion decoding module, the highest cross-modal difference compensation and the output features of the fusion module are taken as the input of the low-level cross-modal difference compensation and the fusion module, the low-level modal feature is continuously fused into, and the fusion image is obtained.

[0072] The discriminator is used to distinguish the fusion result and the original image, and two same network structures are designed as discriminators. The discriminator is composed of four 3*3 convolution layers, Leaky ReLu is used as the activation function after the first three convolution layers, and the hyperbolic tangent (Tanh) function is used in the last layer.

[0073] The model of the application includes a generator, an infrared discriminator and a visible light discriminator, so the loss function is also composed of three parts.

[0074] 1) Generator loss: in the application, content, structure similarity and adversarial loss are used to constrain the network. Therefore, the loss L G of the generator is expressed as:

[0075] L G =λ1L content +λ2L ssim +λ3L adv (6)

[0076] Where, L G represents the total loss function, L content represents the content loss, Ladv and L ssim respectively represent the adversarial loss and the structural similarity loss, and parameters λ1, λ2 and λ3 are balance parameters.

[0077] 2) For the content loss, an intensity loss Lint is introduced to constrain the similarity of pixels between the fused image and the original image. Therefore, the intensity loss L int is represented by equation 6.

[0078]

[0079] where I f represents the fused image, I vi and I ir represent the visible light image and the infrared image respectively; ||·||1 represents the L1 norm; Max(I vi , I ir ) represents the element-wise maximum selection.

[0080] At the same time, one of the goals of image fusion is to integrate the texture details in the source images into a single fused image. Therefore, in order to preserve more texture information of the modal, a texture loss L text is introduced to assist the intensity loss and is represented as:

[0081]

[0082] where represents represents the Sobel operator used to detect texture details in the image, which calculates the gradient change in the image; |·| represents the absolute value; max(.) represents the maximum value.

[0083] Finally, the content loss L content is composed of two parts, the intensity loss L int and the texture loss L text , and is represented as:

[0084] L content = L int + L text (9)

[0085] At the same time, the present application proposes a constraint fusion algorithm based on the structural similarity between the fused image and the original image to ensure the structural information of the fused image. The similarity measure L ssim is defined as follows:

[0086] L ssim = a(1-ssim(I f ,I vi )) + (1-a)(1-ssim(I f ,I ir )) (10)

[0087] where ssim(.) measures the average distance of structural similarity of two images, a is a balancing parameter; x vi and x ir denote the visible light image and the infrared image, respectively; I f denotes the fused image, I vi and I ir denote the visible light image and the infrared image, respectively.

[0088] L adv denotes the loss of the generator, which is defined by the present application as:

[0089]

[0090] 3) Discriminator loss: the loss function of the discriminator network aims to accurately classify the source images. Therefore, the present application defines the loss function of the two discriminator networks as: and as follows:

[0091]

[0092] where the first and second terms represent the Wasserstein distance estimation and the gradient penalty, respectively, and λ4 is a regularization parameter; D ir (.) and D vi (.) denote the infrared light discriminator and the visible light discriminator, respectively; ||·||2 denotes the L2 norm; N denotes the number of images.

[0093] The following introduces the construction of the data set, the experimental settings and implementation details of the training phase and the test phase. In addition, ablation experiments are conducted to verify the superiority of the proposed algorithm. Finally, the performance of the proposed fusion framework is compared with other existing algorithms.

[0094] Training phase and test phase: in the training phase, 25 image pairs are selected from the TNO data set for training. In order to expand this data set, the original image pairs are divided into 256x256, with a sliding step of 8 to achieve image data augmentation. Therefore, a total of 18204 image pairs can be obtained. At the same time, the present application sets the epoch and batch size to 16 and 8, and uses the Adam optimizer to update the network parameters. The present application sets the initial learning rate of the generator and the discriminator to 1x10-4 and 4x10-4. After training the discriminator twice, the generator is trained.

[0095] During the testing phase, to demonstrate the effectiveness of the method of this invention, qualitative and quantitative analyses were performed on TNO, RoadScence, and MRSR. Nine representative methods were selected for comparison with this invention: CUFD, GANMcc, ICA_fusion, Tarsal, U2Fusion, Swinfusion, CDDfuse, UNFusion, and RFN-nest. In the following experiments, average gradient AG, mutual information MI, entropy EN, spatial frequency SF, standard deviation SD, visual information fidelity VIF, wavelet transform-based feature mutual information FMI_w, and edge information QAB / F were used as evaluation metrics.

[0096] Results on the TNO dataset:

[0097] (1) Qualitative Analysis. First, the visualization results of the method of this invention on the TNO dataset and the nine state-of-the-art methods mentioned above are presented. Figure 4 In this study, three representative images were selected: Nato Camp, Kaptein 1654, and Kaptein_1123. In the first and second rows, the red boxes depict people near the fence, and the green boxes show the tree canopy. In the third and fourth rows, the red boxes depict people holding oil-paper umbrellas, and the green boxes show tree branches. In the fifth and sixth rows, the red boxes depict people beside the lawn, and the green boxes show floor-to-ceiling windows. The fusion results show that ICAFSion, RFN-Nest, and U2Fusion can preserve the texture details of the original images well, but weaken the intensity of infrared targets. CUFD, GANMcc, and UNFusion can preserve the intensity of infrared targets in the original images well, but the target edges are blurred and lack visible details. Tardal, SwinFsuion, and CDDfuse have good fusion performance, but the fused image becomes washed out. This is because the TNO dataset primarily contains daytime scenes, while the aforementioned method, trained on the MRSR dataset which includes both daytime and nighttime scenes, is directly tested on the TNO dataset. This results in the fused image tending to retain the intensity of the visible light image. Overall, the method of this invention can maintain the clarity of both visible light and infrared camouflaged human figures while ensuring that the image contrast meets the requirements of the human visual system.

[0098] (2) Quantitative Analysis. Qualitative analysis is highly subjective, and when the differences between two images are small, the human visual perception system has difficulty distinguishing the quality of the two images. Therefore, this invention uses the aforementioned indicators to quantitatively analyze different fusion results. Table 1 shows that the fusion results of this invention achieved optimal values ​​for EN, MI, SD, VIF, FMI_w, and QAB / F. The optimal metrics MI and EN indicate that this invention transfers most of the information from the source image to the fused image. SD and VIF indicate that the fusion results of this invention have higher contrast information and are more consistent with human visual perception. FMI_w and QAB / F indicate that the fusion of this invention transfers more features and edge information to the fused image. Furthermore, AG and SF also achieved good results. Therefore, compared with other methods, the method of this invention has better performance, which also means that the method of this invention pays more attention to the protection of the original image.

[0099] Table 1 shows the quantitative index values ​​obtained by the method of the present invention and other methods on the TNO dataset.

[0100]

[0101] Generalization experiment: To verify the generalization ability of the model of this invention, the fusion model trained on the TNO dataset was directly tested on the MRSR and LLVIP datasets.

[0102] 1) MRSR dataset

[0103] (1) Qualitative analysis. Visualization results of different methods on the MRSR dataset, such as... Figure 5 The images shown are a daytime scene and a nighttime scene. The red boxes in both scenes depict people, while the green boxes depict the second-floor escalator during the day and the windows outside the building at night. The fusion results show that CUFD, Tarsal, SwinFusion, UNFusion, CDDfuse, and the method of this invention can significantly enhance target information and provide clear texture details. The detail information of GANMcc, RFN-Nest, and Tarsal is not obvious in the magnified local images. While ICAFusion shows significant target and detail information, it suffers from insufficient brightness, similar to GANMcc, RFN-Nest, Tarsal, and U2Fusion. The method of this invention can display significant targets and rich texture details, while the brightness of the fused image is consistent with the visible light image.

[0104] (2) Quantitative analysis. From Table 2, it can be seen that the method of the present application shows the fusion results with higher contrast information and more consistent with human visual perception ability in the four best values of MI, SF, SD, QAB / F, which depends on the cross-modal differential compensation and fusion module of the present application. The second best values of AG, EN and the third best values of VIF, FMI_w. The four indicators are decreased on this data set, which is considered reasonable by the present application, since the TNO data set is almost all daytime scenes, lacking night information, and the method of the present application tends to preserve the intensity information of the daytime visible light image, thus causing the contrast of part of the fusion image to decrease.

[0105] Table 2 Quantitative index values obtained by the method of the present application and other methods on the MRSR data set

[0106]

[0107] 2) LLVIP data set.

[0108] (1) Qualitative analysis. The visualization results of different methods on the LLVIP data set are as shown in Figure 6 From Figure 6 it can be seen that the fusion picture obtained by the method of the present application has rich texture details and significant target structures. In Figure 6 this group of pictures, the method of the present application can clearly observe the levels of walls and floors in the local magnification, which is obviously superior to other methods. The remaining methods are not ideal in texture detail reconstruction or retaining significant target structures, and only weak target saliency and detail texture can be observed.

[0109] (2) Quantitative analysis. From the results of Table 3, it can be seen that the method of the present application obtains the best value in most indicators. Therefore, through experiments on three different benchmarks, the present application can conclude that the fusion performance of the method of the present application has strong generalization ability.

[0110] Table 3 Quantitative index values obtained by the method of the present application and other methods on the LLVIP data set.

[0111]

[0112] Ablation study: The rationality of different module designs is verified by ablation experiments. The present application made the following three groups of experiments to verify the effectiveness of the model of the present application. VIF, MI, QAB / F, SF were used for quantitative experiments, and the experimental group results are shown in Table 4.

[0113] (1) Verify the dual-branch encoding network. The application designs three comparative experiments to verify the method of the application. Details as follows: (a) CNN encoding is adopted for both infrared and visible light images, and other network parts remain unchanged; (b) Transformer: using Transformer encoding for both infrared and visible light images, and other network parts remain unchanged; (c) using CNN encoding for visible light images, and using Transformer encoding for infrared images, and other network parts remain unchanged.

[0114] (2) Verify the designed cross-modal difference compensation and fusion module. The module is composed of a cross-modal difference compensation module and a cross-modal feature fusion module. In order to verify the effectiveness of the model, the following experiments are done: (d) remove the cross-modal feature fusion module, and keep the cross-modal difference compensation module; (e) remove the cross-modal difference compensation module, and keep the cross-modal feature fusion module; (f) remove the cross-modal difference compensation and fusion module.

[0115] (3) Verify the rationality of the loss function used. The application carries out three groups of experiments, details as follows: (g) remove the ssim loss, and keep the intensity loss and detail loss; (h) remove the intensity loss, and keep the structural similarity loss and detail loss; (i) remove the detail loss, and keep the intensity loss and ssim loss.

[0116] Table 4 Ablation experiment on TNO dataset

[0117]

[0118] From Table 4, it can be concluded that the fusion evaluation index of the fusion module using CNN and Transformer as feature extractors is higher than that obtained by using CNN or Transformer single branch as feature extractor alone. It is proved from the side that introducing the transformer branch in the CNN module branch strengthens the feature extraction capability of the network. Secondly, using the CDCFM module in the fusion and decoding stage helps to significantly improve MI, SF, QAB / F and VIF. It shows that the CDCFM module can effectively realize the cross-modal fusion of infrared and visible image features and depth features. Finally, for the loss function in this paper, it can be seen that compared with two-by-two combination, the loss function of the application can achieve better results.

[0119] The application proposes a cross-modal infrared and visible image fusion network based on a generative adversarial network, and obtains original image pixel intensity and detail texture mapping features through CNN and Transformer branches.

[0120] The application also provides a storage medium, which stores a computer program, and the computer program is executed by a processor to realize some or all steps of the cross-modal infrared and visible image fusion method based on feature difference compensation and fusion.

[0121] Those skilled in the art can clearly understand that the technology in the embodiments of the application can be realized by means of software and necessary general hardware platforms. Based on this understanding, the technical solutions in the embodiments of the application can be embodied in the form of a software product, which can be stored in a storage medium such as a ROM / RAM, a magnetic disc, an optical disc, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the method described in the embodiments of the application or some parts of the embodiments.

[0122] The above-mentioned embodiments of the application do not constitute a limitation on the protection scope of the application.

Claims

1. An infrared and visible image fusion method based on feature difference compensation and fusion, characterized in that, The method comprises the following steps: using CNN and Transformer encoder to extract multi-scale features of infrared and visible light images respectively; inputting the multi-scale features into four cross-modal differential compensation and fusion modules for feature fusion; using a progressive differential and fusion decoding module, taking the output features of the highest level cross-modal differential compensation and fusion module as the input of the lower level cross-modal differential compensation and fusion module, and continuously fusing the low-level modal features to obtain a fused image; using CNN and Transformer encoder to extract multi-scale features of infrared and visible light images respectively, comprising: The shallow features F are extracted from the visible light and infrared original images by a feature encoder based on CNN encoding and a feature encoder based on Transfomer encoding respectively vi and F ir ; wherein the feature encoder based on CNN encoding is used to extract the infrared image branch, and the feature encoder based on Restormer Block encoding is used to extract the visible light image branch to obtain the detailed texture information and structural information of the image, expressed as follows: wherein, and denote the deep features extracted from the infrared and visible light images, respectively, and the superscript i = {0, 1, 2, 3} corresponds to the output features of the i-th layer; when i = 0, it represents the output features extracted by the shallow layer, and when i = {1, 2, 3}, it represents the multi-scale features of three stages obtained by CNN encoding and Transfomer encoding, respectively; I vi and I ir denote the visible light image and the infrared image, respectively; E(.) denotes the original image; in the step of inputting the multi-scale features into four cross-modal differential compensation and fusion modules for feature fusion, the cross-modal differential compensation and fusion module comprises a cross-modal differential compensation module and a cross-modal feature fusion module, and the cross-modal differential compensation module is defined as: wherein, denotes element-wise summation, denotes element-wise multiplication, δ(·) denotes a Sigmod function, Conv(.) denotes a convolution operation, GMP(·) and GAP(·) denote global max pooling and global average pooling, respectively; formula (2) indicates that after global max pooling, global average pooling, and convolution, the attention maps are normalized by a sigmoid function to generate corresponding attention weights, the supplementary features are multiplied by the attention weights, and the result is added to the original features to obtain the modal supplementary information.

2. The method for infrared and visible image fusion based on feature difference compensation and fusion according to claim 1, characterized in that, in the step of inputting the multi-scale features into four cross-modal differential compensation and fusion modules for feature fusion, the feature fusion comprises: The output features of the fourth layer are fused to obtain initial fusion features and ​ will be described below. The initial fusion attention weight map is obtained through channel attention and spatial attention respectively, and the initial weight map is obtained by channel addition using a Sigmoid function to normalize to generate corresponding attention weights; The attention weight and the feature of the current frame are multiplied respectively and fused to obtain an intermediate fusion feature The attention weight and the feature of the current frame are multiplied respectively and fused to obtain an intermediate fusion feature The attention weight and the feature of the current frame are multiplied respectively and fused to obtain an intermediate fusion feature as the input of the next cross-modal feature fusion module.

3. The method for infrared and visible image fusion based on feature difference compensation and fusion according to claim 2, characterized in that, the cross-modal feature fusion module is defined as: wherein, denotes element-wise summation, denotes element-wise multiplication, p denotes prelu activation function, Conv(.) denotes convolution operation, GMP(·) denotes global max pooling, denotes intermediate fused feature, denotes initial weight map, δ(·) denotes Sigmod function, and denote infrared and visible output features, respectively.

4. The method for infrared and visible image fusion based on feature difference compensation and fusion according to claim 1, characterized in that, the method further comprises: distinguishing the fusion result and the original image by using a discriminator, using two same network structures as the discriminator, the discriminator is composed of four 3×3 convolution layers, Leaky ReLu is used as the activation function after the first three convolution layers, and the hyperbolic tangent function is used in the last layer.

5. The method for infrared and visible image fusion based on feature difference compensation and fusion according to claim 1, characterized in that, Generator loss, whose loss L G is represented as: L G = λ1L content + λ2L ssim + λ3L adv (6) where L G The total loss function L content represents the content loss L adv and L ssim respectively represent the adversarial loss and the structural similarity loss, and parameters λ1, λ2 and λ3 are balancing parameters.

6. The method for infrared and visible image fusion based on feature difference compensation and fusion according to claim 1, characterized in that, An intensity loss Lint is introduced to constrain the similarity of pixels between the fused image and the original image, and the intensity loss L int is expressed by Equation 7: where I f represents the fused image, I vi and I ir represent the visible and infrared images, respectively; ||·||1represents the L1 norm; Max(I vi , I ir ) represents the element-wise maximum selection; A texture loss L is introduced to assist the strength loss and is represented as: text L = 1 - 1 + 1 wherein represents represents that the Sobel operator is used to detect the texture details in the image, with which the gradient change in the image is calculated; | · | represents taking the absolute value; max(.) represents the maximum value; Content loss L content By intensity loss L int And texture loss L text Two parts, expressed as: L content = L int + L text (9) Defining a similarity measure L ssim As follows: L ssim = a(1 - ssim(I f , I vi )) + (1 - a)(1 - ssim(I f , I ir )) (10) where ssim(.) is the mean distance of structural similarity of two images, a is a balancing parameter; x vi and x ir denote the visible light image and the infrared image, respectively; I f denotes the fused image, I vi and I ir denote the visible light image and the infrared image, respectively; L adv represents the loss of the generator, defined as:

7. The method for infrared and visible image fusion based on feature difference compensation and fusion according to claim 1, characterized in that, Loss function defining 2 discriminator networks and As follows: where the first and second terms represent Wasserstein distance estimation and gradient penalty, respectively, and λ4is a regularization parameter; D ir (.) and D vi (.) and D represent the infrared light discriminator and the visible light discriminator, respectively; ||·||2represents the L2norm; and N represents the number of images.

Images