Infrared and visible light image fusion method based on cross-domain feature fusion and GAN

Abstract

The invention belongs to the field of image processing, and discloses an infrared and visible light image fusion method based on cross-domain feature fusion and GAN, and the method comprises the steps: effectively extracting the feature information of a shallow layer and a deep layer in a source image through designing a shallow layer-deep layer feature extraction module, and reducing model network parameters; by constructing a cross-domain feature fusion module, cross-modal complementary information is mined and integrated, information loss and noise interference during image fusion are reduced, and the representation intensity of a fused image is ensured; by designing a main and auxiliary feature loss function, structure information, texture information, gradient information and secondary information in a source image are fully extracted, and finally a model is trained to obtain a detection result and perform evaluation. According to the method, while the contrast and texture details of the source image are fully reserved, hidden auxiliary information in the source image can be deeply mined and effectively utilized, and the overall quality of the fused image is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared and visible light image fusion method based on cross-domain feature fusion and GAN. Background Art

[0002] The process of image fusion is to combine the image information collected by at least two sensors in the same environment to obtain more accurate environmental image information. The uniqueness of infrared and visible light image fusion lies in the complementarity of information: the reflected light captured by the visible light sensor contains rich detail information, while the infrared sensor can detect the thermal target information in the scene. By combining these two types of information, high-quality images containing more scene information can be obtained.

[0003] However, in multimodal image fusion, especially the fusion of infrared and visible light images, single-scale feature extraction often fails to capture all important and subtle information in the image. Information in large objects or complex scenes may be reflected at different scales, and single-scale feature extraction tends to ignore these details, resulting in the loss of some important features of the image.

[0004] In addition, due to the essential differences in the imaging mechanisms of infrared and visible light sensors, the information features they capture and highlight are different. Currently, many deep learning-based image fusion methods only extract features by stacking simple convolutional layers, which often cannot fully exploit the feature information in multimodal source images and cannot achieve effective information exchange between different modalities. In particular, these methods perform poorly in identifying key feature areas in multimodal images and cannot maintain the appropriate visual intensity and appearance characteristics of the image from a holistic perspective.

[0005] In addition, existing image fusion methods also have obvious deficiencies in utilizing key auxiliary information in the source image. Due to the lack of reference images, many fusion algorithms rely mainly on intensity loss functions to learn multimodal features, and the design of such intensity loss functions is usually based only on the contrast of infrared images and the gradient information of visible light images. This approach ignores the secondary information in the source image that also contains rich semantics. This information is valuable for optimizing the image fusion process and should be taken seriously. Summary of the invention

[0006] Purpose of the invention: The present invention provides an infrared and visible light image fusion method based on cross-domain feature fusion and GAN, named ACFAMGan, which can deeply mine and effectively utilize the auxiliary information hidden in the source image while fully preserving the contrast and texture details of the source image.

[0007] Technical solution: The present invention proposes a method for fusion of infrared and visible light images based on cross-domain feature fusion and GAN, comprising the following steps:

[0008] (1) Preprocess the data set to construct a training data set;

[0009] (2) Design a shallow-deep feature extraction module and a cross-domain feature fusion module to build a generator;

[0010] (3) Construct a dual discriminator to fit the infrared and visible light image data distributions respectively;

[0011] (4) Design the main and auxiliary feature loss functions and train the network to obtain a trained generative adversarial network;

[0012] (5) Test and evaluate network performance.

[0013] The specific operation of step (1) is as follows:

[0014] (1.1) Using 10 pairs of TNO images and 30 pairs of INO images, the fusion results of the traditional image fusion method GFF are used as pre-fusion images. In step 4 of the present invention, the pre-fusion images are used as labels to construct the loss function.

[0015] (1.2) The 30 pairs of infrared and visible light images and pre-fused images are cropped one by one from left to right and from top to bottom, with the cropping size of 120×120, resulting in 27,966 pairs of infrared and visible light images and pre-fused images, which are used as training data sets.

[0016] The specific operation of step (2) is as follows:

[0017] The generator adopts an encoder-decoder structure. The feature map first passes through a convolution layer (3×3 convolution layer, LeakyReLU activation function) to convert the image into feature information that is convenient for subsequent feature extraction, and then passes through a shallow-deep feature extraction module (SDFE) to extract the shallow and deep feature information of the image. The module includes 3 branches. The upper branch (shallow feature extraction) consists of 4 layers of densely connected blocks, which extracts the shallow feature information of the image through residual connections. The middle branch (deep feature extraction) is composed of an MDTA module and a lightweight attention module (ECA-Net). While extracting the deep information of the image, it reduces the number of model parameters. The lower branch transmits the input image and fuses it with the images extracted by the upper and middle branches, and then passes through a cross-domain feature fusion module (AHCFM) This module mainly includes two consecutive identical cross-domain feature fusions. The input image is projected to query (Q), key (K), and value (V) according to the learnable weight matrix, and then the K and V information of the two branches are cross-input into the two branches. The upper branch passes through the MCA module to capture the dependency and semantic information between the input information, and then undergoes a layer normalization (LN) process to increase the generalization ability of the model. The input image of the lower branch and the processed image are fused to extract the complementary information between the infrared and visible light images. After two consecutive fusions, the branch feature information is finally fused through a convolutional layer. The decoder consists of 4 consecutive convolutional modules. All convolutional modules are composed of convolutional layers and LeakyReLU activation functions. The convolution kernel size is 3×3 and the step size is 1.

[0018] In step (2), the specific operations of the dense connection block are as follows:

[0019] Y i =F i (cnt(Y 0 ,Y 1 ,…,Y i-1 ))

[0020] Among them, Y i is the feature of the output of the i-th layer, F i is a nonlinear transformation, and cnt(·) is a concatenation operation of the features output by the previous layers.

[0021] In step (2), the specific operation of (MDTA) is as follows:

[0022] Y∈Φ H×w×c

[0023]

[0024]

[0025] F s (Q,K,V)=V×Fsm (Q×K / ω)

[0026] Among them, Y, are the input feature map and output feature map respectively, H, W, C are the length, width, and height channels of the input image respectively, Q is the query vector, which indicates the current point or position information of interest, K is the key vector, which indicates the identification feature of each position, V is the value vector, which indicates the valuable information for subsequent processing, and W Q ,W K ,W V are the learned projection matrices, W 1 (·) and W 2 (·) are 1×1 and 3×3 convolutions, respectively, and F s (Q, K, V) is the self-attention calculation, F sm (·) is a normalization operation, and ω is a scaling parameter used to adjust the size of the product of Q and K.

[0027] In step (2), the specific operations of (ECA-Net) are as follows:

[0028] τ=ρ(B 1 D k (Y)

[0029]

[0030] Among them, B 1 D k is an adaptive one-dimensional convolution, ρ is a sigmoid activation function, y i is the i-th component of Y, τ i is the corresponding weight, Ω i For i The set of k adjacent channels, k is the coverage of intra-group interaction, α = 2, b = 1, |x| odd is the odd number closest to x.

[0031] In step (2), the specific operation of (AHCFM) is as follows:

[0032]

[0033] Y′ 1 =LN(MCA(Q 1 ,K 2 ,B 2 ))+Y 1

[0034] Y′ 2 =LN(MCA(Q 2 ,K 1 ,V 1))+Y 2

[0035] Y″ 1 =LN(MCA(Q 1 ,K 2 ,V 2 ))+Y′ 1

[0036] Y″ 2 =LN(MCA(Q 2 ,K 1 ,V 1 ))+Y′ 2

[0037] Among them, {Q 1 ,K 1 ,V 1},{Q 2 ,K 2 ,V 2} are {Q, K, V} of the image after infrared and visible light feature extraction, respectively, Y 1 ,Y 2 are the input images before cross-domain fusion of infrared and visible light, Y 1 ′,Y 2 ′ are the output images after the first cross-domain fusion, LN(·) is the layer normalization operation, MCA(·) is the multi-head cross attention operation, and Y′ 1 ,Y′ 2 It is the output image after secondary cross-domain fusion.

[0038] Wherein, in step (3), the discriminator includes the discriminator D ir and the discriminator D vi :

[0039] Discriminator D ir Used to identify whether the input image is a fusion image or an infrared image, the discriminator D vi Used to identify whether the input image is a fusion image or a visible light image, the discriminator D ir and the discriminator D vi With the same network structure and shared parameters, the discriminator D ir and the discriminator D vi They are composed of 5 consecutive convolutional modules and a fully connected layer. All convolutional modules are composed of convolutional layers and LeakyReLU activation functions. The convolution kernel size is 3×3, the step size is 2, and the number of output feature map channels of the 5 convolutional layers is set to 32, 64, 64, 128 and 256 respectively. The features extracted by the first 5 convolutional layers are first used to classify the input image, and then the fully connected layer is used to determine the input image type.

[0040] The specific operation of step (4) is as follows:

[0041] (4.1) Construct the main and auxiliary feature loss functions, and the total loss function is as follows:

[0042]

[0043] in, is the generator loss, is the generator adversarial loss, is the feature loss, λ,ξ 1 ,ξ 2 ,ξ 3 is a hyperparameter, is the structural similarity loss, is the texture loss, is the gradient loss, is the discriminator loss, is the infrared discriminator loss, Visible light discriminator loss,

[0044] Generator Adversarial Loss The definition is as follows:

[0045]

[0046] Among them, I f is the fused image, P G is the fusion image data distribution, D ir To use infrared light and fusion image as discriminator input, D vis To use visible light and fused images as discriminator input,

[0047] Structural Similarity Loss The definition is as follows:

[0048]

[0049] Among them, θ 1 ,θ 2 is a hyperparameter, and its value is 0.5. 1 ,I 2 ,I f are infrared image, visible light image and fused image respectively, ssim(·) is the structural similarity operation,

[0050] Texture Loss The definition is as follows:

[0051]

[0052] in, is the Sobel gradient operator, used to calculate texture information, |·| is used to calculate the absolute value, ‖·‖1 is the 1-norm of the matrix, H, W are the length and width of the image,

[0053] Gradient Loss The definition is as follows:

[0054]

[0055] in,, is the main gradient loss, is the secondary gradient loss, δ is the hyperparameter, I f ,I ir ,I vis They are fused image, infrared image and visible light image respectively. is the second-order gradient operator, is the F-norm of the matrix,

[0056] Infrared discriminator loss The definition is as follows:

[0057]

[0058] Among them, P ir and P vis is the distribution of infrared and visible light image data, P G To fuse image data distribution, is a uniform sampling of the middle area between the real sample and the generated sample, φ 1 and φ 2 is a hyperparameter.

[0059] (4.2) The Adam optimizer is selected, the initial learning rate is set to 0.0001, the batch size is set to 32, and the number of epochs is set to 100 to train the network.

[0060] The beneficial effects of the present invention are as follows:

[0061] (1) The present invention adopts an image fusion architecture based on a generative adversarial network (GAN). The end-to-end characteristics of the network model make the fusion process more concise and efficient, which is in line with the application requirements of infrared and visible light image fusion.

[0062] (2) The present invention designs a shallow-deep feature extraction module, which enables the generator network to better extract shallow and deep feature information in the source image, while effectively reducing the model network parameters and improving the model information transmission capability.

[0063] (3) The present invention constructs a cross-domain feature fusion module, mines and integrates complementary information across modalities, effectively prevents information loss and noise interference in the fusion process, and ensures the appropriate representation strength of the fused image from an overall perspective.

[0064] (4) The present invention designs a primary and secondary feature loss function to extract structural information, texture information, gradient information and hidden secondary information from the source image, so that the fused image can more effectively combine the contrast of the infrared image and the texture details of the visible light image, thereby generating a higher quality fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is the overall network structure diagram of the present invention;

[0066] Figure 2 This is a network structure diagram of the generator of the present invention;

[0067] Figure 3 This is a network structure diagram of the shallow-deep feature extraction module (SDFE) of the present invention;

[0068] Figure 4 This is a network structure diagram of the MDTA module of the present invention;

[0069] Figure 5 This is the network structure diagram of the lightweight attention module (ECA-Net) of the present invention;

[0070] Figure 6 This is a network structure diagram of the cross-domain feature fusion module (AHCFM) of the present invention;

[0071] Figure 7 This is the network structure diagram of the identifier of the present invention. DETAILED DESCRIPTION

[0072] In order to better understand the present invention, the present invention is further described below in conjunction with the accompanying drawings in the examples of the present invention, but it is not intended to limit the present invention.

[0073] The present invention is an infrared and visible light image fusion method based on cross-domain feature fusion and GAN, and its overall network structure is as follows: Figure 1 As shown in the figure, the infrared and visible light images are first subjected to shallow-deep feature extraction, and then cross-domain feature fusion is performed to generate a fused image. The fused image and the source image are passed to the discriminator to identify whether the input image is a fused image or a source image. Then, the main and auxiliary feature loss functions are constructed, the network is trained, and finally the network performance is tested. The specific steps include:

[0074] Step (1): Preprocess the data set to construct a training data set, including the following steps:

[0075] (1.1) Using 10 pairs of TNO images and 30 pairs of INO images, the fusion results of the traditional image fusion method GFF are used as pre-fusion images. In step 4 of the present invention, the pre-fusion images are used as labels to construct the loss function.

[0076] (1.2) The 30 pairs of infrared and visible light images and pre-fused images are cropped one by one from left to right and from top to bottom, with the cropping size of 120×120, resulting in 27,966 pairs of infrared and visible light images and pre-fused images, which are used as training data sets.

[0077] Step (2): Design a shallow-deep feature extraction module and a cross-domain feature fusion module to build a generator.

[0078] like Figure 2 As shown in the figure, the generator adopts an encoder-decoder structure. The feature map first passes through a convolution layer (3×3 convolution layer, LeakyReLU activation function) to convert the image into feature information that is convenient for subsequent feature extraction, and then passes through a shallow-deep feature extraction module (SDFE) to extract the shallow and deep feature information of the image, as shown in Figure 3 As shown in the figure, the module includes three branches. The upper branch consists of four layers of densely connected blocks, which extracts shallow feature information of the image through residual connection. The middle branch consists of MDTA module and lightweight attention module (ECA-Net), which extracts deep information of the image while reducing the number of model parameters. The lower branch transmits the input image and fuses it with the image extracted by the upper and middle branches, and then passes through a cross-domain feature fusion module (AHCFM). This module mainly includes two consecutive identical cross-domain feature fusions, which mainly projects the input image to query (Q), key (K), and ba according to the learnable weight matrix. lue(V), and then cross-input the K and V information of the two branches into the double branches. The upper branch passes through the MCA module to capture the dependency and semantic information between the input information, and then undergoes a layer normalization (LN) process to increase the generalization ability of the model. The input image of the lower branch and the processed image are fused to extract the complementary information between the infrared and visible light images. After two consecutive fusions, the branch feature information is finally fused through a convolutional layer. The decoder consists of 4 consecutive convolutional modules. All convolutional modules are composed of convolutional layers and LeakyReLU activation functions. The convolution kernel size is 3×3, and the step size is 1.

[0079] MDTA, ECA-Net and AHCFM Figure 4 , Figure 5 and Figure 6 The specific operations of dense connection block, MDTA, ECA-Net and AHCFM are as follows:

[0080] Y i =F i (cnt(Y 0 ,Y 1 ,…,Y i-1 ))

[0081] Among them, Y iis the feature of the output of the i-th layer, F i is a nonlinear transformation, and cnt(·) is a concatenation operation of the features output by the previous layers.

[0082] Y∈Φ H×w×c

[0083]

[0084] F s (Q,K,V)=V×F sm (Q×K / ω)

[0085] Among them, Y, are the input feature map and output feature map respectively, H, W, C are the length, width, and height channels of the input image respectively, Q is the query vector, which indicates the current point or position information of interest, K is the key vector, which indicates the identification feature of each position, V is the value vector, which indicates the valuable information for subsequent processing, and W Q ,W K ,W V are the learned projection matrices, W 1 (·) and W 2 (·) are 1×1 and 3×3 convolutions, respectively, and F s (Q, K, V) is the self-attention calculation, F sm (·) is a normalization operation, and ω is a scaling parameter used to adjust the size of the product of Q and K.

[0086] τ=ρ(B 1 D k (Y)

[0087]

[0088] Among them, B 1 D k is an adaptive one-dimensional convolution, ρ is a sigmoid activation function, y i is the i-th component of Y, τ i is the corresponding weight, Ω i For i The set of k adjacent channels, k is the coverage of intra-group interaction, α = 2, b = 1, |x| odd is the odd number closest to x.

[0089]

[0090] Y′ 1 =LN(MCA(Q 1 ,K 2 ,V 2 ))+Y 1

[0091] Y′ 2 =LN(MCA(Q 2 ,K 1 ,V 1 ))+Y 2

[0092] Y″ 1 =LN(MCA(Q 1 ,K 2 ,V 2 ))+Y′ 1

[0093] Y″ 2 =LN(MCA(Q 2 ,K 1 ,V 1 ))+Y′ 2

[0094] Among them, {Q 1 ,K 1 ,V 1},{Q 2 ,K 2 ,V 2} are {Q, K, V} of the image after infrared and visible light feature extraction, respectively, Y 1 ,Y 2 are the input images before cross-domain fusion of infrared and visible light, Y 1 ′,Y 2 ′ are the output images after the first cross-domain fusion, LN(·) is the layer normalization operation, MCA(·) is the multi-head cross attention operation, and Y″ 1 ,Y″ 2 It is the output image after secondary cross-domain fusion.

[0095] Step (3): Construct a dual discriminator to fit the infrared and visible light image data distributions respectively.

[0096] like Figure 7 As shown, the fused image and the source image, namely the infrared image and the visible light image, are passed to the discriminator, which includes the discriminator D ir and the discriminator D vi , to determine whether the input image is a fusion image or a source image, the discriminator D ir Used to identify whether the input image is a fusion image or an infrared image, the discriminator D vi Used to identify whether the input image is a fusion image or a visible light image, the discriminator D ir and the discriminator D vi With the same network structure and shared parameters, the discriminator D ir and the discriminator D viThey are composed of 5 consecutive convolutional modules and a fully connected layer. All convolutional modules are composed of convolutional layers and LeakyReLU activation functions. The convolution kernel size is 3×3, the step size is 2, and the number of output feature map channels of the 5 convolutional layers is set to 32, 64, 64, 128 and 256 respectively. The features extracted by the first 5 convolutional layers are first used to classify the input image, and then the fully connected layer is used to determine the input image type.

[0097] Step (4): Construct the main and auxiliary feature loss functions, select the Adam optimizer, set the initial learning rate to 0.0001, the batch size to 32, the number of epochs to 100, and start training the network.

[0098] Step (5): Test the performance of the network. In order to prove the effectiveness of the fusion method proposed in the present invention, five mainstream infrared and visible light image fusion methods are selected for comparison, namely FusionGAN method, GANMCC method, GTF method, CNN method and RP method. All comparison methods are tested using the default parameters provided in the literature. The four indicators of standard deviation (SD), average gradient (AG), mutual information (MI) and peak signal-to-noise ratio (PSNR) are used as objective quantitative indicators to analyze the fusion results of the TNO dataset. The following indicators are all positive evaluation indicators. The higher the score, the stronger the fusion performance. The results are shown in Table 1.

[0099]

[0100] As can be seen from Table 1, the RDPEBGan algorithm ranked second in the SD index, indicating that the algorithm can make the fused image have a higher contrast and a more obvious display effect. It ranked first and second in AG and MI respectively, indicating that the fused image has good texture details. However, it ranked fifth in the PSNR index, indicating that the fused image has certain distortion problems. However, given that this index is very similar to the results of other algorithms, in general, the RDPEBGan algorithm can achieve better fusion effects, which fully verifies its rationality and effectiveness.

[0101] The above description is only a preferred specific embodiment of the present invention and is not intended to limit the present invention. After reading the content of the present invention, those skilled in the art may make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for fusion of infrared and visible light images based on cross-domain feature fusion and GAN, characterized by: The following steps are involved: (1) Preprocess the data set to construct a training data set; (2) Construct a generator. The generator adopts an encoder-decoder structure and uses a dual-branch encoder data stream for feature extraction. Any branch of the encoder includes a convolutional layer, a shallow-deep feature extraction module (SDFE), and a cross-domain feature fusion module (AHCFM). The SDFE module includes three branches. The upper branch consists of four layers of densely connected blocks. It extracts shallow feature information of the image through residual connections. The middle branch consists of an MDTA module and a lightweight attention module (ECA-Net). While extracting deep information of the image, it reduces the number of model parameters. The lower branch transmits the input image and fuses it with the image extracted by the upper and middle branches. Cross-domain feature fusion The module (AHCFM) includes two consecutive identical cross-domain feature fusions. The input image is projected to query (Q), key (K), and value (V) according to a learnable weight matrix, and then the K and V information of the two branches are cross-input into the two branches. The upper branch passes through the MCA module to capture the dependency and semantic information between the input information, and then undergoes a layer normalization (LN) process to increase the generalization ability of the model. The input image of the lower branch and the processed image are fused to extract the complementary information between the infrared and visible light images. After two consecutive fusions, the branch feature information is finally fused through a convolutional layer. The decoder is 4 consecutive convolutional modules. (3) Construct a dual discriminator, which includes the discriminator D ir and the discriminator D vi , the discriminator D ir Used to identify whether the input image is a fusion image or an infrared image, the discriminator D vi Used to identify whether the input image is a fusion image or a visible light image, the discriminator D ir and the discriminator D vi Have the same network structure and share parameters; (4) Construct the main and auxiliary feature loss functions and train the network to obtain a trained generative adversarial network.

2. According to claim 1, the infrared and visible light image fusion method based on cross-domain feature fusion and GAN is characterized in that: Step (1) constructs a training data set and performs preprocessing operations on it, which specifically includes the following steps: (1.1) Using infrared and visible light images from the TNO dataset and the INO dataset, the fusion results of the traditional image fusion method GFF are used as pre-fusion images, and the pre-fusion images are used as labels to construct the loss function; (1.2) The infrared and visible light images and the pre-fused images are cropped into image blocks to obtain infrared visible light images and pre-fused images, which are used as training data sets.

3. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: The decoder consists of 4 consecutive convolution modules. All convolution modules are composed of convolution layers and LeakyReLU activation functions. The convolution kernel size is 3×3 and the step size is 1.

4. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: The specific operations of the dense connection block are as follows: AND i =F i (cnt(Y0,Y1,…,Y i-1 )) Among them, Y i is the feature of the output of the i-th layer, F i is a nonlinear transformation, and cnt(·) is a concatenation operation of the features output by the previous layers.

5. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: The specific operation of the MDTA module is as follows: Y∈Φ H×w×c F s (Q,K,V)=V×F sm (Q×K / ω) Among them, Y, are the input feature map and output feature map respectively, H, W, C are the length, width, and height channels of the input image respectively, Q is the query vector, which indicates the current point or position information of interest, K is the key vector, which indicates the identification feature of each position, V is the value vector, which indicates the valuable information for subsequent processing, and W Q ,W K ,W V are the learned projection matrices, W1(·) and W2(·) are 1×1 and 3×3 convolutions, respectively, and F s (Q, K, V) is the self-attention calculation, F sm (·) is a normalization operation, and ω is a scaling parameter used to adjust the size of the product of Q and K.

6. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: The specific operation of the lightweight attention module is as follows: τ=ρ(B1D k (Y)) Among them, B1D k is an adaptive one-dimensional convolution, ρ is a sigmoid activation function, y i is the i-th component of Y, τ i is the corresponding weight, Ω i For i The set of k adjacent channels, k is the coverage of intra-group interaction, α = 2, b = 1, |x| odd is the odd number closest to x.

7. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: The cross-domain feature fusion module specifically operates as follows: Y′1=LN(MCA(Q1,K2,V2))+Y1 Y′2=LN(MCA(Q2,K1,V1))+Y2 Y″1=LN(MCA(Q1,K2,V2))+Y′1 Y″2=LN(MCA(Q2,K1,V1))+Y′2 Among them, {Q1, K1, V1}, {Q2, K2, V2} are the {Q, K, V} of the images after infrared and visible light feature extraction, respectively, Y1, Y2 are the input images before cross-domain fusion of infrared and visible light, respectively, Y1′, Y2′ are the output images after the first cross-domain fusion, LN(·) is the layer normalization operation, MCA(·) is the multi-head cross attention operation, and Y″1, Y″2 are the output images after the second cross-domain fusion.

8. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: The discriminator D in step (3) ir and the discriminator D vi They are composed of 5 consecutive convolutional modules and a fully connected layer. All convolutional modules are composed of convolutional layers and LeakyReLU activation functions. The convolution kernel size is 3×3, the step size is 2, and the number of output feature map channels of the 5 convolutional layers is set to 32, 64, 64, 128 and 256 respectively. The features extracted by the first 5 convolutional layers are first used to classify the input image, and then the fully connected layer is used to determine the input image type.

9. The infrared and visible light image fusion method based on cross-domain feature fusion and GAN according to claim 1 is characterized in that: Step (4) constructs the main and auxiliary feature loss functions. The training network specifically includes the following steps: (4.1) Construct the main and auxiliary feature loss functions, and the total loss function is as follows: in, is the generator loss, is the generator adversarial loss, is the feature loss, λ,ξ1,ξ2,ξ3 are hyper parameters, is the structural similarity loss, is the texture loss, is the gradient loss, is the discriminator loss, is the infrared discriminator loss, Visible light discriminator loss, Generator Adversarial Loss The definition is as follows: Among them, I f is the fused image, P G is the fusion image data distribution, D ir To use infrared light and fusion image as discriminator input, D vis To use visible light and fused images as discriminator input, Structural Similarity Loss The definition is as follows: Among them, θ1, θ2 are hyperparameters, and their values ​​are all 0.5, I1, I2, I f are infrared image, visible light image and fused image respectively, ssim(·) is the structural similarity operation, Texture Loss The definition is as follows: in, is the Sobel gradient operator used to calculate texture information, |·| is used to calculate the absolute value, ‖·‖1 is the 1-norm of the matrix, H and W are the length and width of the image, Gradient Loss The definition is as follows: in,, is the main gradient loss, is the secondary gradient loss, δ is the hyperparameter, I f ,I ir ,I vis They are fused image, infrared image and visible light image respectively. is the second-order gradient operator, is the F-norm of the matrix, Infrared discriminator loss The definition is as follows: Among them, P ir and P vis is the distribution of infrared and visible light image data, P G To fuse image data distribution, is a uniform sampling of the middle area between the real sample and the generated sample, φ1 and φ2 are hyper parameters, (4.2) Adam optimizer is used for network training.

Images