Infrared and visible light image fusion method based on Swin Transformer and GAN

Abstract

The present invention proposes an infrared and visible light image fusion method based on Swin Transformer and GAN, which mainly solves the problem of how to simultaneously retain the salient target information in the infrared image and the texture detail information in the visible light image during the image fusion process. This method combines the traditional method with the deep learning method, and utilizes the global feature extraction capability of Swin Transformer and the adversarial training process of the generative adversarial network to achieve high-quality fusion effect. In addition, the source image is decomposed by implicit low-rank representation technology, and a dual-branch structure is adopted in the generator to fully integrate the complementary information in the source image. In order to improve the training efficiency, the method also introduces pre-fusion technology to provide prior knowledge for the generative adversarial network. The method aims to meet the efficient and accurate requirements for the fusion of infrared and visible light images in modern remote sensing, military and other fields, and brings new methods to the field of image processing.

Description

Technical field:

[0001] The present invention belongs to the technical field of image fusion, and in particular to an infrared and visible light image fusion method based on Swin Transformer and GAN. Background technology:

[0002] The task of image fusion aims to integrate multimodal source images with complementary information into a fused image with richer information and easier to use in subsequent applications, so as to make up for the shortcomings of single sensor imaging. Due to the complementarity and convenience of infrared and visible light image information, image fusion, as an important branch of multimodal image fusion tasks, has been widely used in many fields such as security, military, video surveillance, and target tracking. Infrared images are not easily affected by external conditions and can effectively highlight the thermal radiation subject from the background even in bad weather, but they usually have low spatial resolution and insufficient texture details. Visible light images have higher resolution and clear texture details, which are more in line with human visual perception preferences, but are easily affected by external factors such as bad weather, poor lighting conditions, and obstacles. Infrared and visible light image fusion technology can effectively combine the advantages of the two images to generate a fused image that contains both the thermal radiation information in the source infrared image and the rich texture details in the source visible light image.

[0003] The technical solution closest to the present invention is CN202110220561.2. This solution proposes a method for fusion of infrared and visible light images based on three-scale decomposition, which mainly decomposes the visible light image and the infrared image into three scales, namely the background brightness layer, the significant feature layer and the detail layer, and then fuses the different decomposition layers using different fusion methods. The core idea of ​​this method is to ensure the real-time fusion of infrared and visible light images, and to retain the background brightness information to improve the quality of the fused image. However, this solution does not involve how to use implicit low-rank representation technology to perform a more refined decomposition of the source image, nor does it consider how to use generative adversarial networks for feature extraction and fusion.

[0004] Another technical solution, CN202310434441.1, discloses a method for fusion of infrared and visible light images with target feature embedding. This method mainly improves the image fusion result through the target detection meta-feature embedding process. Although this method effectively solves the mismatch problem between the target detection task and the image fusion task, its design of the fusion method is limited to the embedding of target features, and does not involve the feature extraction and fusion strategy of multimodal images at the base layer and detail layer.

[0005] The existing infrared and visible light image fusion technologies have the following main shortcomings: Most existing technologies mainly rely on fixed scale decomposition methods, such as three-scale decomposition. Although this method can achieve preliminary image fusion, it may not be able to fully capture the global and local information in the image in complex scenes. When dealing with image fusion, existing technologies often only focus on specific features, such as salient features or background brightness, and ignore other important image details, resulting in the fused image may lose some key information. Although some technologies attempt to improve the fusion effect by embedding target features, this method is limited to target detection tasks and may add additional computational burden in practical applications. Existing methods lack the use of generative adversarial networks in the fusion process, which may make the fusion effect unsatisfactory, especially in high-quality image fusion scenarios.

[0006] In view of the above technical shortcomings, this patent proposes a new infrared and visible light image fusion method. The invention solves the following main technical problems: how to effectively extract global and local structural information in multimodal image fusion; how to combine implicit low-rank representation technology with generative adversarial networks to achieve high-quality image fusion; how to ensure the real-time performance and superior effect of the fusion process, especially in complex environments and changing scenes. Summary of the invention:

[0007] The purpose of the present invention is to address the defects of the prior art and provide an infrared and visible light image fusion method based on Swin Transformer and GAN to solve the problems raised by the above background technology.

[0008] The infrared and visible light image fusion method based on Swin Transformer and GAN includes the following steps:

[0009] S1: Use implicit low-rank representation to decompose infrared images and visible light images into base layer images and detail layer images, and perform normalization;

[0010] S2: pre-fusion of base layer and detail layer images respectively;

[0011] S3: Input the pre-fused base layer and detail layer images into a generative adversarial network, which consists of a generator and two discriminators;

[0012] S4: Generate the final fused image using the trained GAN.

[0013] As a technical preferred solution of the present invention: the implicit low-rank decomposition and image normalization in S1 specifically include the following steps:

[0014] S1-1: Decompose the infrared image using implicit low-rank representation to obtain infrared base layer sub-band image and infrared detail layer sub-band image;

[0015] S1-2: Decompose the visible light image using implicit low-rank representation to obtain a visible light base layer subband image and a visible light detail layer subband image;

[0016] Among them, the implicit low-rank representation LatLRR processes both the column and row directions of the image data matrix through optimization problems to simultaneously extract the global and local structural information in the image data.

[0017] As a preferred technical solution of the present invention: the specific steps of pre-fusion of the base layer and detail layer images in S2 are as follows:

[0018] S2-1: performing an average operation on the base layer subband parts of different modal images for pre-fusion to obtain a pre-fused base layer image;

[0019] S2-2: performing a maximum value operation on detail layer subband parts of different modal images for pre-fusion to obtain a pre-fused detail layer image.

[0020] As a technical preferred solution of the present invention: the generator network in S3 is composed of the following parts:

[0021] S3-1-1: Base layer branch, in order to extract global features in the base layer image, this branch includes: the first CNN convolution block, used to extract shallow feature information; multiple Swin Transformer layers, used to further extract deep features of the base layer subband image; the last CNN convolution block, used to reduce the dimension of the features output by the Swin Transformer layer;

[0022] S3-1-2: Detail layer branch, in order to extract local features in the detail layer image, this branch includes: a spatial branch, which uses a densely connected multi-layer convolution structure to extract spatial texture features; a structural branch, which extracts structural features through up- and down-sampling of multiple convolution operations;

[0023] S3-1-3: Image reconstruction part, in order to fuse and reconstruct the features extracted by the two branches, including: convolution layer for feature dimension reduction; Tanh activation layer for outputting the fused image;

[0024] Among them, the output features of the base layer branch are connected with the output features of the detail layer branch in the channel dimension, and then input into the image reconstruction part for further processing and reconstruction.

[0025] As a technical preferred solution of the present invention: the discriminator network in S3 is composed of the following parts:

[0026] S3-2-1: The first five convolution blocks, each of which includes: a convolution layer for extracting image features; a BatchNormalization layer for feature normalization; and a ReLU activation layer for enhancing the nonlinear expression capability of the model.

[0027] S3-2-2: The sixth layer is a fully connected layer, whose main functions are: compressing the features extracted by the convolution block and outputting the decision result of the discriminator; using the Sigmoid activation function, mapping the output value to between 0 and 1, indicating the probability that the input image is the real source image;

[0028] The output of each convolution block serves as the input of the next convolution block, and the output of the fully connected layer represents the decision result of the discriminator.

[0029] As a technical preferred solution of the present invention: the characteristics of the generator loss function required for training the generative adversarial network in S4 include:

[0030] S4-1: Generator total loss L G It consists of two parts: adversarial loss L adv and content loss L content , the total loss of the generator is L G The definition of L is: G =L adv +λL content , where λ is used to control L adv and L content The weight coefficient for the balance between the two losses;

[0031] S4-1-1: Countermeasures against loss L adv , the loss of the discriminator is fed back to the generator to guide the generator to generate a fused image that is closer to the real data distribution, defined as: L adv =E[-log(D ir (I f ))]+E[-log(D vis (I f ))], where D(·) is the probability that the discriminator judges whether the input image is from the real image or the fused image, and E[·] is the average value of the discriminator output for each training batch;

[0032] S4-1-2: Content loss L content , including strength loss L int , gradient loss L grad and variance-based structural similarity loss L Var-SSIM , defined as: L content =αL int +βLgrad +γL Var-SSIM , where α, β and γ are weight coefficients;

[0033] S4-1-2-1: Strength loss L int , forcing the fused image to have a similar intensity distribution as the source image, is defined as: Among them, H and W are the height and width of the input image, ||·|| F is the F norm, a is the weight coefficient;

[0034] S4-1-2-2: Gradient loss L grad , guides the generator to generate a fused image containing the texture detail information of the source image, defined as: in, is the gradient operator, b is the weight coefficient;

[0035] S4-1-2-3: Variance-based structural similarity loss L Var-SSIM , calculate the brightness, contrast and structural similarity between images, defined as:

[0036]

[0037]

[0038] Among them, N represents the number of blocks after the image is segmented, and SSIM represents structural similarity.

[0039] As a technical preferred solution of the present invention: the characteristics of the discriminator loss function required for training the generative adversarial network in S4 include:

[0040] S4-2-1: Infrared discriminator loss D ir , which is used to force the generator to retain the salient target intensity information in the infrared image when generating the fused image, and is defined as:

[0041] S4-2-2: Visible light discriminator loss D vis , which is used to force the generator to retain the rich detail texture information in the visible light image when generating the fused image, and is defined as:

[0042] Among them, the infrared discriminator D ir The input is the infrared image I ir Or fusion image I f , visible light discriminator D vis The input is a visible light image I vis Or fusion image I f .

[0043] As a technical preferred solution of the present invention: the S3-1-1 basic layer branch is composed of the following parts:

[0044] S3-1-1-1: The first layer is a CNN convolution block, which includes: a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1; a batch normalization layer; a LeakyReLU activation layer; wherein the number of output channels is 64, which is used to extract shallow feature information in the base layer subband image;

[0045] S3-1-1-2: The 2nd to 4th layers are Swin Transformer layers (STL), which are used to further extract the deep features of the base layer sub-band image; and a residual connection structure is used between multiple layers of STL;

[0046] S3-1-1-3: The fifth layer is a CNN convolution block, which includes: a convolution layer with a convolution kernel size of 3×3, a stride of 1, and a padding of 1; a batch normalization layer; a LeakyReLU activation layer; and the number of output channels is 64;

[0047] S3-1-1-4: Swin Transformer divides the image into multiple small windows for multi-head self-attention calculation, which has higher computational efficiency than Visual Transformer (ViT); residual connections are used between multiple layers of STL, and layer normalization is performed in each layer of STL;

[0048] S3-1-1-5: In the window-based multi-head self-attention calculation, the input is divided into multiple non-overlapping local windows by setting the window size, and the query, key, and value feature matrices are calculated. The calculation formula is:

[0049] Q=Φ W M Q ,K=Φ W M K ,V=Φ W M V

[0050]

[0051] Among them, the local window feature map Φ W , the corresponding query, key, and value feature matrices are Q, K, and V; M Q 、M K and M V is a trainable parameter matrix shared in different windows; d represents the dimension size of Q and K, and B represents the trainable relative position bias.

[0052] As a technical preferred solution of the present invention: the S3-1-2 detail layer branch is composed of the following parts:

[0053] S3-1-2-1: The detail layer branch consists of two small branches: the spatial branch and the structural branch; the spatial branch is used to extract spatial texture feature information from the detail layer image; the structural branch is used to extract the remaining global structural feature information in the detail layer image;

[0054] S3-1-2-2: At the initial stage of the detail layer branch, a convolution operation is performed on the input detail layer image to output a feature map with 16 channels, which is then input into the spatial branch and the structural branch respectively;

[0055] S3-1-2-3: The spatial branch adopts a densely connected four-layer convolution structure, that is, the input of each convolution layer is connected to the feature maps output by all previous convolution layers on the channel. The number of input feature map channels of the four convolution layers is 16, 8, 16, and 24 respectively, and the number of output feature map channels is 8. Each convolution layer includes: a convolution layer with a convolution kernel size of 3×3, a stride of 1, a padding of 1, and a padding method of reflection padding; a batch normalization layer; and a LeakyReLU activation layer.

[0056] S3-1-2-4: The structural branch reduces the feature map size and increases the number of channels through three convolution downsampling operations by two times, and then restores it to the original size through one convolution upsampling operation by eight times. The number of input feature map channels of the four layers of convolution are 16, 32, 64, and 32 respectively. The number of output feature map channels is first increased to 64 and then reduced to 32. The first three layers of convolution include: a convolution layer with a convolution kernel size of 3×3, a stride of 2, a padding of 1, and a padding method of reflection padding; a batch normalization layer; and a LeakyReLU activation layer.

[0057] S3-1-2-5: The number of feature map channels finally output by the structure branch is 32, which is consistent with the number of output channels of the spatial branch.

[0058] As a technical preferred solution of the present invention: the S3-1-3 image reconstruction consists of the following parts:

[0059] S3-1-3-1: After the base layer branch and the detail layer branch extract the features of the source images of different modalities, these features are connected based on the channel dimension as the input of the image reconstruction part;

[0060] S3-1-3-2: The image reconstruction part reduces the dimension and reconstructs the connected high-dimensional features to further generate a fused image;

[0061] S3-1-3-3: The image reconstruction part is mainly composed of the following structures: convolution layer, which uses convolution with a kernel size of 1×1, a step size of 1, and a padding of 0, in order to reduce the dimensionality of high-dimensional feature data; Tanh activation layer, which is used to normalize the calculation results of the convolution layer;

[0062] S3-1-3-4: Through the above structure, a fused image is finally generated, ensuring that the fused image fully integrates the features of the source image and has good visual effects and high quality.

[0063] Compared with the related prior art, this patent has the following main technical advantages, and the beneficial effects of the invention are:

[0064] 1) Deep feature extraction: By combining the respective advantages of CNN and Swin Transformer, this method can extract deeper feature information from infrared and visible light images, so that the fused image has better feature representation ability.

[0065] 2) Multi-level fusion: The decomposition method of the base layer and the detail layer is adopted so that the fused image can not only preserve the low-level texture information but also represent the high-level global structure information.

[0066] 3) Real-time and efficiency: This method not only ensures the fusion effect, but also utilizes the locality of convolution and the globality of SwinTransformer, and has high computational efficiency and real-time performance.

[0067] 4) Strong adaptability: This method is not only suitable for specific infrared and visible light image fusion tasks, but can also be widely used in other multimodal image fusion scenarios.

[0068] 5) Optimized objective function design: Combining adversarial loss and content loss, the fused image can both deceive the discriminator and be similar to the source image content, thus obtaining a more realistic and natural fusion result.

[0069] 6) Balance between structure and texture: Through specially designed spatial branches and structural branches, the structural information and texture details of the image can be balanced.

[0070] The present invention proposes an infrared and visible light image fusion method based on Swin Transformer and GAN. The effectiveness and generalization of the method proposed in the present invention are verified by conducting experiments on public data sets. The experimental results show that compared with other infrared and visible light image fusion methods, in terms of qualitative subjective visual effects, the fused image obtained by the method proposed in the present invention has higher contrast, better overall visual effects, and is more in line with the human visual perception system; furthermore, in terms of quantitative objective evaluation indicators, the method proposed in the present invention is superior to the average values ​​of the comparison methods in terms of entropy, spatial frequency, standard deviation and edge strength by about 4.36%, 23.45%, 16.68% and 16.94% respectively. In summary, the method proposed in the present invention can highlight the significant targets in the infrared image while retaining as much clear texture detail information in the visible light image as possible, that is, it highlights the significant infrared targets on the basis of the visible light image, thereby further improving the fusion performance and laying the foundation for its downstream advanced visual tasks. Description of the drawings:

[0071] Figure 1 is a flow chart of the method of the present invention;

[0072] Figure 2 It is the overall framework of the method proposed in the present invention;

[0073] Figure 3 It is a schematic diagram of implicit low-rank representation decomposition of an image provided by the present invention;

[0074] Figure 4 is a schematic diagram of pre-fusion of different sub-band images provided by the present invention;

[0075] Figure 5 It is the generator network structure provided by the present invention;

[0076] Figure 6 It is a schematic diagram of two consecutive layers of STLs provided by the present invention;

[0077] Figure 7 This is the discriminator network structure provided by the present invention. Specific implementation method:

[0078] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0079] The implicit low-rank representation of the traditional method can simultaneously extract the global structural information and local structural information in the source image and has strong robustness to noise. In the existing generative adversarial network model, although the structure of a single generator and two discriminators can retain the detailed texture information in the visible light image while retaining the salient targets in the infrared image, its network backbone structure is still mainly focused on the convolutional neural network (CNN) used to extract local features of the image. The Swin Transformer network not only has the global feature extraction capability of the standard Transformer network but also has higher computational efficiency. Based on the above analysis, in order to more specifically extract different feature information in the source image and further improve the fusion performance, the present invention proposes an infrared and visible light image fusion method based on Swin Transformer and GAN. The proposed network model consists of a generator and two discriminators. The fused image is continuously optimized through adversarial iterative training between the generator and the two discriminators, so that the complementary information in the infrared and visible light images is retained as much as possible. The proposed method uses an implicit low-rank representation to decompose the source image into a base layer subband image with global structural information and a detail layer subband image with local feature information, which are then input into the dual branches of the generator respectively, so that the network can extract different features more specifically. Specifically, in the base layer branch, Swin Transformer is used to extract the global subject information in the base layer subband image; in the detail layer branch, CNN is used to extract the local detail information in the detail layer subband image. In addition, in view of the instability and time-consuming problems in the GAN training process, prior conditions are introduced into the generator network to improve the training efficiency of GAN. Specifically, before the base layer and detail layer images are input into the generator, a pre-fused image is generated by taking the average of the base layer image and the maximum of the detail layer image, thereby providing prior knowledge for the generator network. The experimental results on the public dataset show that the method proposed in the present invention can not only fully retain the salient targets in the infrared image, but also effectively restore the texture detail information in the visible light image.

[0080] The infrared and visible light image fusion method based on Swin Transformer and GAN includes the following steps:

[0081] S1: Use implicit low-rank representation to decompose infrared images and visible light images into base layer images and detail layer images, and perform normalization;

[0082] The implicit low-rank decomposition and image normalization specifically include the following steps:

[0083] S1-1: Decompose the infrared image using implicit low-rank representation to obtain infrared base layer sub-band image and infrared detail layer sub-band image;

[0084] S1-2: Decompose the visible light image using implicit low-rank representation to obtain a visible light base layer subband image and a visible light detail layer subband image;

[0085] Among them, the implicit low-rank representation LatLRR processes both the column and row directions of the image data matrix through optimization problems to simultaneously extract the global and local structural information in the image data.

[0086] S2: pre-fusion of base layer and detail layer images respectively;

[0087] S2-1: performing an average operation on the base layer subband parts of different modal images for pre-fusion to obtain a pre-fused base layer image;

[0088] S2-2: performing a maximum value operation on detail layer subband parts of different modal images for pre-fusion to obtain a pre-fused detail layer image;

[0089] S3: Input the pre-fused base layer and detail layer images into a generative adversarial network, which consists of a generator and two discriminators;

[0090] The generator network consists of the following parts:

[0091] S3-1-1: Base layer branch, in order to extract global features in the base layer image, this branch includes: the first CNN convolution block, used to extract shallow feature information; multiple Swin Transformer layers, used to further extract deep features of the base layer subband image; the last CNN convolution block, used to reduce the dimension of the features output by the Swin Transformer layer;

[0092] The base layer branch consists of the following parts:

[0093] S3-1-1-1: The first layer is a CNN convolution block, which includes: a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1; a batch normalization layer; a LeakyReLU activation layer; wherein the number of output channels is 64, which is used to extract shallow feature information in the base layer subband image;

[0094] S3-1-1-2: The 2nd to 4th layers are Swin Transformer layers (STL), which are used to further extract the deep features of the base layer sub-band image; and a residual connection structure is used between multiple layers of STL;

[0095] S3-1-1-3: The fifth layer is a CNN convolution block, which includes: a convolution layer with a convolution kernel size of 3×3, a stride of 1, and a padding of 1; a batch normalization layer; a LeakyReLU activation layer; and the number of output channels is 64;

[0096] S3-1-1-4: Swin Transformer divides the image into multiple small windows for multi-head self-attention calculation, which has higher computational efficiency than Visual Transformer (ViT); residual connections are used between multiple layers of STL, and layer normalization is performed in each layer of STL;

[0097] S3-1-1-5: In the window-based multi-head self-attention calculation, the input is divided into multiple non-overlapping local windows by setting the window size, and the query, key, and value feature matrices are calculated. The calculation formula is:

[0098] Q=Φ W M Q ,K=Φ W M K ,V=Φ W M V

[0099]

[0100] Among them, the local window feature map Φ W , the corresponding query, key, and value feature matrices are Q, K, and V; M Q 、M K and M V is a trainable parameter matrix shared in different windows; d represents the dimension size of Q and K, and B represents the trainable relative position bias.

[0101] S3-1-2: Detail layer branch, in order to extract local features in the detail layer image, this branch includes: a spatial branch, which uses a densely connected multi-layer convolution structure to extract spatial texture features; a structural branch, which extracts structural features through up- and down-sampling of multiple convolution operations;

[0102] The detail layer branch consists of the following parts:

[0103] S3-1-2-1: The detail layer branch consists of two small branches: the spatial branch and the structural branch; the spatial branch is used to extract spatial texture feature information from the detail layer image; the structural branch is used to extract the remaining global structural feature information in the detail layer image;

[0104] S3-1-2-2: At the initial stage of the detail layer branch, a convolution operation is performed on the input detail layer image to output a feature map with 16 channels, which is then input into the spatial branch and the structural branch respectively;

[0105] S3-1-2-3: The spatial branch adopts a densely connected four-layer convolution structure, that is, the input of each convolution layer is connected to the feature maps output by all previous convolution layers on the channel. The number of input feature map channels of the four convolution layers is 16, 8, 16, and 24 respectively, and the number of output feature map channels is 8. Each convolution layer includes: a convolution layer with a convolution kernel size of 3×3, a stride of 1, a padding of 1, and a padding method of reflection padding; a batch normalization layer; and a LeakyReLU activation layer.

[0106] S3-1-2-4: The structural branch reduces the feature map size and increases the number of channels through three convolution downsampling operations by two times, and then restores it to the original size through one convolution upsampling operation by eight times. The number of input feature map channels of the four layers of convolution are 16, 32, 64, and 32 respectively. The number of output feature map channels is first increased to 64 and then reduced to 32. The first three layers of convolution include: a convolution layer with a convolution kernel size of 3×3, a stride of 2, a padding of 1, and a padding method of reflection padding; a batch normalization layer; and a LeakyReLU activation layer.

[0107] S3-1-2-5: The number of feature map channels finally output by the structure branch is 32, which is consistent with the number of output channels of the spatial branch.

[0108] S3-1-3: Image reconstruction part, in order to fuse and reconstruct the features extracted by the two branches, includes: convolution layer for feature dimension reduction; Tanh activation layer for outputting the fused image;

[0109] Among them, the output features of the base layer branch are connected with the output features of the detail layer branch in the channel dimension, and then input into the image reconstruction part for further processing and reconstruction.

[0110] Image reconstruction consists of the following components:

[0111] S3-1-3-1: After the base layer branch and the detail layer branch extract the features of the source images of different modalities, these features are connected based on the channel dimension as the input of the image reconstruction part;

[0112] S3-1-3-2: The image reconstruction part reduces the dimension and reconstructs the connected high-dimensional features to further generate a fused image;

[0113] S3-1-3-3: The image reconstruction part is mainly composed of the following structures: convolution layer, which uses convolution with a kernel size of 1×1, a step size of 1, and a padding of 0, in order to reduce the dimensionality of high-dimensional feature data; Tanh activation layer, which is used to normalize the calculation results of the convolution layer;

[0114] S3-1-3-4: Through the above structure, a fused image is finally generated, ensuring that the fused image fully integrates the features of the source image and has good visual effects and high quality.

[0115] The discriminator network consists of the following parts:

[0116] S3-2-1: The first five convolution blocks, each of which includes: a convolution layer for extracting image features; a BatchNormalization layer for feature normalization; and a ReLU activation layer for enhancing the nonlinear expression capability of the model.

[0117] S3-2-2: The sixth layer is a fully connected layer, whose main functions are: compressing the features extracted by the convolution block and outputting the decision result of the discriminator; using the Sigmoid activation function, mapping the output value to between 0 and 1, indicating the probability that the input image is the real source image;

[0118] The output of each convolution block serves as the input of the next convolution block, and the output of the fully connected layer represents the decision result of the discriminator.

[0119] S4: Generate the final fused image using the trained generative adversarial network;

[0120] The characteristics of the generator loss function required for training a generative adversarial network include:

[0121] S4-1: Generator total loss L G It consists of two parts: adversarial loss L adv and content loss L content , the total loss of the generator is L G The definition of L is: G =L adv +λL content , where λ is used to control L adv and L content The weight coefficient for the balance between the two losses;

[0122] S4-1-1: Countermeasures against loss L adv , the loss of the discriminator is fed back to the generator to guide the generator to generate a fused image that is closer to the real data distribution, defined as: L adv =E[-log(D ir (I f ))]+E[-log(D vis (I f ))], where D(·) is the probability that the discriminator judges whether the input image is from the real image or the fused image, and E[·] is the average value of the discriminator output for each training batch;

[0123] S4-1-2: Content loss L content , including strength loss L int , gradient loss L grad and variance-based structural similarity loss L Var-SSIM , defined as: L content＝ αL int +βL grad +γL Var-SSIM , where α, β and γ are weight coefficients;

[0124] S4-1-2-1: Strength loss L int , forcing the fused image to have a similar intensity distribution as the source image, is defined as: Among them, H and W are the height and width of the input image, ||·|| F is the F norm, a is the weight coefficient;

[0125] S4-1-2-2: Gradient loss L grad , guides the generator to generate a fused image containing the texture detail information of the source image, defined as: Among them, ▽ is the gradient operator, b is the weight coefficient;

[0126] S4-1-2-3: Variance-based structural similarity loss L Var-SSIM , calculate the brightness, contrast and structural similarity between images, defined as:

[0127]

[0128]

[0129] Among them, N represents the number of blocks after the image is segmented, and SSIM represents structural similarity.

[0130] The characteristics of the discriminator loss function required for training a generative adversarial network include:

[0131] S4-2-1: Infrared discriminator loss D ir , which is used to force the generator to retain the salient target intensity information in the infrared image when generating the fused image, and is defined as: L Dir =E[-log(D ir (I ir ))]+E[-log(1-D ir (I F ))];

[0132] S4-2-2: Visible light discriminator loss D vis , which is used to force the generator to retain the rich detail texture information in the visible light image when generating the fused image, and is defined as: L Dvis =E[-log(Dvis (I vis ))]+E[-log(1-D vis (I F ))];

[0133] Among them, the infrared discriminator D ir The input is the infrared image I ir Or fusion image I f , visible light discriminator D vis The input is a visible light image I vis Or fusion image I f .

[0134] Example 1: Figure 1 As shown, the method flow of the present invention mainly includes the following steps: acquiring infrared and visible light source images; performing implicit low-rank representation decomposition on the source image; performing pre-fusion processing on the decomposed sub-band images; performing adversarial training through the designed generator and discriminator to obtain a fused image; and outputting the final fused image.

[0135] like Figure 2 As shown in FIG. 1 , the overall framework of the method proposed in the present invention includes a generator and two discriminators (an infrared discriminator and a visible light discriminator). The generator is responsible for generating a fused image, while the discriminator is responsible for judging the similarity between the generated fused image and the original infrared or visible light image.

[0136] like Figure 3 As shown in Figure 1, the source image is first decomposed by an implicit low-rank representation to obtain images of different sub-bands. This decomposition can effectively extract the basic and detailed features in the image and provide richer information for subsequent fusion.

[0137] like Figure 4 As shown in Figure 1, according to the characteristics of different sub-band images, the corresponding pre-fusion strategy is used to process them. The purpose of this step is to ensure that the generator can better fuse the characteristics of each sub-band in the subsequent adversarial training.

[0138] like Figure 5 As shown in Figure 1, the generator consists of multiple convolutions, Swin Transformer, and convolution transpose layers. Its task is to generate high-quality fused images based on the input infrared and visible light image features.

[0139] like Figure 6 As shown in the figure, in the two-layer STLs structure, each layer is mainly composed of window-based multi-head self-attention calculation and multi-layer perceptron. This structure can effectively extract long-distance dependencies in the image and improve the fusion effect.

[0140] like Figure 7As shown in Figure 1, the discriminator consists of multiple convolutional layers and is used to determine whether the input image is a real infrared or visible light image. Through adversarial training with the generator, the discriminator can encourage the generator to generate fused images that are closer to the real distribution.

[0141] Embodiment 2: This embodiment provides a method for fusion of infrared and visible light images based on Swin Transformer and GAN, such as Figure 1 and Figure 2 As shown, the specific steps include: (1) Acquire an infrared image and the corresponding visible light image. (2) Use LatLRR technology to decompose the infrared image and the visible light image into a base layer and a detail layer respectively. (3) For the base layer image, first use CNN to extract shallow features, and then use Swin Transformer to extract deep global features. (4) For the detail layer image, it is divided into two branches for processing: Spatial branch: Use a four-layer convolution structure to extract spatial texture features. Structural branch: Extract global structural features through convolution downsampling and upsampling operations. (5) Connect the processing results of the base layer and the detail layer, then reduce the dimension through a 1x1 convolution kernel, and then use the Tanh activation function to normalize to obtain a fused image.

[0142] Through the above embodiments, the present invention proposes an infrared and visible light image fusion method based on Swin Transformer and GAN, and the overall framework of the proposed method is mainly composed of a generator and two discriminators. The proposed method combines the traditional method with the deep learning method, and uses the global feature extraction ability of Swin Transformer and the adversarial training process of GAN to generate a fused image that contains both salient targets and rich texture details, thereby further improving the fusion performance. In the method proposed in the present invention, an implicit low-rank representation is used to decompose the source image into a base layer image and a detail layer image, and a dual-branch structure is adopted in the generator, wherein the base layer branch is used to extract global feature information in the base layer image, and the detail layer branch is used to extract local feature information in the detail layer image, so as to fully integrate the complementary information in the source image. In addition, in order to improve the training efficiency and stability of GAN, the present invention pre-fuses the input image by taking the average value of the base layer and the maximum value of the detail layer, thereby providing prior knowledge for GAN. The two discriminators in the method proposed in the present invention have the same network structure. The discriminator judges whether the input image comes from the real source image or the fused image generated by the generator. Through iterative adversarial training with the generator, the data distribution of the fused image is made close to both the source infrared image and the source visible light image.

[0143] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. The infrared and visible light image fusion method based on Swin Transformer and GAN is characterized by: The method comprises the following steps: S1: Use implicit low-rank representation to decompose infrared images and visible light images into base layer images and detail layer images, and perform normalization; S2: Pre-fusion of base layer and detail layer images respectively; S3: Input the pre-fused base layer and detail layer images into a generative adversarial network, which consists of a generator and two discriminators; The generator network consists of the following parts: S3-1-1: Base layer branch, in order to extract global features in the base layer image, this branch includes: the first CNN convolution block, used to extract shallow feature information; multiple Swin Transformer layers, used to further extract deep features of the base layer subband image; the last CNN convolution block, used to reduce the dimension of the features output by the Swin Transformer layer; The base layer branch is composed of the following parts: S3-1-1-1: The first layer is a CNN convolution block, which includes: a convolution layer with a convolution kernel size of 3×3, a step size of 1, and a padding of 1; a batch normalization layer; a LeakyReLU activation layer; wherein the number of output channels is 64, which is used to extract shallow feature information in the base layer subband image; S3-1-1-2: The 2nd to 4th layers are Swin Transformer layers (STL), which are used to further extract the deep features of the base layer sub-band image; and a residual connection structure is used between multiple layers of STL; S3-1-1-3: The fifth layer is a CNN convolution block, which includes: a convolution layer with a convolution kernel size of 3×3, a stride of 1, and a padding of 1; a batch normalization layer; a LeakyReLU activation layer; and the number of output channels is 64; S3-1-1-4: Swin Transformer divides the image into multiple small windows for multi-head self-attention calculation, which has higher computational efficiency than Visual Transformer (ViT); residual connections are used between multiple layers of STL, and layer normalization is performed in each layer of STL; S3-1-1-5: In the window-based multi-head self-attention calculation, the input is divided into multiple non-overlapping local windows by setting the window size, and the query, key, and value feature matrices are calculated. The calculation formula is: Q=Φ W M Q ,K=Φ W M K ,V=Φ W M V Among them, the local window feature map Φ W , and its corresponding query, key, and value feature matrices are Q, K, and V; M Q 、M K and M V is a trainable parameter matrix shared in different windows; d represents the dimension size of Q and K, and B represents the trainable relative position bias; S3-1-2: Detail layer branch, in order to extract local features in the detail layer image, this branch includes: a spatial branch, which uses a densely connected multi-layer convolution structure to extract spatial texture features; a structural branch, which extracts structural features through up- and down-sampling of multiple convolution operations; The detail layer branch consists of the following parts: S3-1-2-1: The detail layer branch consists of two small branches: the spatial branch and the structural branch; the spatial branch is used to extract spatial texture feature information from the detail layer image; the structural branch is used to extract the remaining global structural feature information in the detail layer image; S3-1-2-2: At the initial stage of the detail layer branch, a convolution operation is performed on the input detail layer image to output a feature map with 16 channels, which is then input into the spatial branch and the structural branch respectively; S3-1-2-3: The spatial branch adopts a densely connected four-layer convolution structure, that is, the input of each convolution layer is connected to the feature maps output by all previous convolution layers on the channel. The number of input feature map channels of the four convolution layers is 16, 8, 16, and 24 respectively, and the number of output feature map channels is 8. Each convolution layer includes: a convolution layer with a convolution kernel size of 3×3, a stride of 1, a padding of 1, and a padding method of reflection padding; a batch normalization layer; and a LeakyReLU activation layer. S3-1-2-4: The structural branch reduces the feature map size and increases the number of channels through three convolution downsampling operations by two times, and then restores it to the original size through one convolution upsampling operation by eight times. The number of input feature map channels of the four layers of convolution are 16, 32, 64, and 32 respectively. The number of output feature map channels is first increased to 64 and then reduced to 32. The first three layers of convolution include: a convolution layer with a convolution kernel size of 3×3, a stride of 2, a padding of 1, and a padding method of reflection padding; a batch normalization layer; and a LeakyReLU activation layer. S3-1-2-5: The number of feature map channels finally output by the structure branch is 32, which is consistent with the number of output channels of the spatial branch; S3-1-3: Image reconstruction part, in order to fuse and reconstruct the features extracted by the two branches, including: convolution layer for feature dimension reduction; Tanh activation layer for outputting the fused image; Among them, the output features of the base layer branch are connected with the output features of the detail layer branch in the channel dimension, and then input into the image reconstruction part for further processing and reconstruction; Image reconstruction consists of the following components: S3-1-3-1: After the base layer branch and the detail layer branch extract the features of the source images of different modalities, these features are connected based on the channel dimension as the input of the image reconstruction part; S3-1-3-2: The image reconstruction part reduces the dimension and reconstructs the connected high-dimensional features to further generate a fused image; S3-1-3-3: The image reconstruction part is mainly composed of the following structures: convolution layer, which uses convolution with a kernel size of 1×1, a step size of 1, and a padding of 0, in order to reduce the dimensionality of high-dimensional feature data; Tanh activation layer, which is used to normalize the calculation results of the convolution layer; S3-1-3-4: Through the above structure, a fused image is finally generated to ensure that the fused image fully integrates the features of the source images and has good visual effects and high quality; The discriminator network consists of the following parts: S3-2-1: The first five convolution blocks, each of which includes: a convolution layer for extracting image features; a BatchNormalization layer for feature normalization; and a ReLU activation layer for enhancing the nonlinear expression capability of the model. S3-2-2: The sixth layer is a fully connected layer, whose main functions are: compressing the features extracted by the convolution block and outputting the decision result of the discriminator; using the Sigmoid activation function, mapping the output value to between 0 and 1, indicating the probability that the input image is the real source image; The output of each convolutional block is used as the input of the next convolutional block, and the output of the fully connected layer represents the decision result of the discriminator. S4: Generate the final fused image using the trained generative adversarial network; The characteristics of the generator loss function required for training a generative adversarial network include: S4-1: Generator total loss L G It consists of two parts: adversarial loss L adv and content loss L content , the total loss of the generator is L G The definition of L is: G =L adv +λL content , where λ is used to control L adv and L content The weight coefficient for the balance between the two losses; S4-1-1: Countermeasures against loss L adv , the loss of the discriminator is fed back to the generator to guide the generator to generate a fused image that is closer to the real data distribution, defined as: L adv =E[-log(D ir (I F ))]+E[-log(D vis (I F ))], where D(·) is the probability that the discriminator judges whether the input image is from the real image or the fused image, and E[·] is the average value of the discriminator output for each training batch; S4-1-2: Content loss L content , including strength loss L int , gradient loss L grad and variance-based structural similarity loss L Var-SSIM , defined as: L content =αL int +βL grad +γL Var-SSIM , where α, β and γ are weight coefficients; S4-1-2-1: Strength loss L int , forcing the fused image to have a similar intensity distribution as the source image, is defined as: Among them, H and W are the height and width of the input image, ||·|| F is the F norm, a is the weight coefficient; S4-1-2-2: Gradient loss L grad [, guides the generator to generate a fused image containing the texture detail information of the source image, defined as: in, is the gradient operator, b is the weight coefficient; S4-1-2-3: Variance-based structural similarity loss L Var-SSIM , calculate the brightness, contrast and structural similarity between images, defined as: Among them, N represents the number of blocks after the image is segmented, and SSIM represents structural similarity; The characteristics of the discriminator loss function required for training a generative adversarial network include: S4-2-1: Infrared discriminator loss D ir , which is used to force the generator to retain the salient target intensity information in the infrared image when generating the fused image, and is defined as: S4-2-2: Visible light discriminator loss D vis , which is used to force the generator to retain the rich detail texture information in the visible light image when generating the fused image, and is defined as: Among them, the infrared discriminator D ir The input is the infrared image I ir Or fused image I F , visible light discriminator D vis The input is a visible light image I vis Or fused image I F。 2. The infrared and visible light image fusion method based on Swin Transformer and GAN according to claim 1 is characterized in that: The implicit low-rank decomposition and image normalization in S1 specifically include the following steps: S1-1: Decompose the infrared image using implicit low-rank representation to obtain infrared base layer sub-band image and infrared detail layer sub-band image; S1-2: Decompose the visible light image using implicit low-rank representation to obtain a visible light base layer subband image and a visible light detail layer subband image; Among them, the implicit low-rank representation LatLRR processes both the column and row directions of the image data matrix through optimization problems to simultaneously extract the global and local structural information in the image data.

3. The infrared and visible light image fusion method based on Swin Transformer and GAN according to claim 1, characterized in that: The specific steps of pre-fusion of the base layer and detail layer images in S2 are as follows: S2-1: performing an average operation on the base layer subband parts of different modal images for pre-fusion to obtain a pre-fused base layer image; S2-2: performing a maximum value operation on detail layer subband parts of different modal images for pre-fusion to obtain a pre-fused detail layer image.

Images