Method for efficiently converting visible light images into infrared light images based on gan

Abstract

The application provides a method for efficiently converting a visible light image into an infrared light image based on a GAN, referred to as IMGAN. In the generation network, an InceptioNeXtV2 network which is lighter than ConvNeXt V2 and improves performance is proposed as an encoder, a selective state space model represented by Mamba is introduced into the decoder, and a designed LFEM module for extracting local information is combined to improve the fidelity of the image. A guided fusion strategy GFS is designed to ensure that the generated infrared image not only retains the structural information of the visible light image, but also embodies the physical characteristics of infrared imaging. Texture loss and color loss are designed as additional auxiliary functions to constrain the bilateral mapping to solve the problems of rough texture and color difference. The method is evaluated on four different data sets, and the results are compared with existing architectures (including Pix2Pix, ThermalGAN and InfraGAN). Compared with the InfraGAN architecture, the method improves the SSIM and PSNR on the VEDAI data set by 2.38% and 7.28% respectively.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and digital image processing, in particular to a method for efficiently converting visible light images into infrared light images based on GAN. BACKGROUND

[0002] In the field of computer vision, infrared imaging is increasingly important in object detection, medical diagnosis, satellite remote sensing, and other fields. However, the scarcity of infrared datasets and the high cost of acquisition limit the application of infrared images. Using deep learning technology to convert visible light images into infrared light images to meet the demand of infrared scene prediction. The existing visible-infrared image conversion method based on GAN still has the following shortcomings:(1) the confidence of the infrared image converted by the model is poor;(2) the infrared features of different semantic information lack details and have color differences. SUMMARY

[0003] The present application provides an improved method for converting visible light images into infrared images based on a generative adversarial network (GAN) architecture. By designing an efficient generator and discriminator architecture, a physically consistent loss function, and a multi-scale feature fusion mechanism, high-quality infrared images are generated. Through the adversarial training of the generator and the discriminator, the quality of the generated images is optimized, making the generated infrared images have high realism and physical consistency.

[0004] The present application adopts the following technical scheme: a method for efficiently converting visible light images into infrared light images based on GAN, comprising the following steps:

[0005] Input the visible light image and convert it into an infrared image through an adversarial network;

[0006] For the generator, use InceptioNeXt V2 network as the encoder to extract multi-scale features from the input visible light image;

[0007] In the decoder, use the collaborative extraction module SEM and the selective state space model Mamba module to extract global features, the LFEM module to supplement local information, and generate an infrared image;

[0008] Use GFS guided fusion strategy or feature channel direct splicing to fuse the multi-scale features from the encoder and the infrared features generated by the decoder;

[0009] For the discriminator, evaluate the authenticity of the generated infrared image, identify the true and false of the overall image and each pixel point; the discriminator uses U-Net structure, uses spectral normalization residual block SRB as encoder and decoder, and optimizes the generator through adversarial training;

[0010] The GFS guided fusion strategy is used to fuse the features from the encoder and the infrared features generated by the decoder;

[0011] A composite loss function is calculated and minimized, which includes structure consistency loss, mean absolute value loss, texture loss, color loss and adversarial loss, to facilitate the minimization of the difference between the generated infrared image and the real infrared image.

[0012] The visible light image is a preprocessed image, and the preprocessing step includes normalizing the image so that the image pixel value is in the range of [-1, 1].

[0013] The decoder of the generator performs the following steps:

[0014] The input features are sequentially enhanced by the synergistic extraction module SEM, and the spatial and channel features are processed by the selective state space model Mamba module, and then fused with the input features;

[0015] The fused features are then spliced with the fused features after the LFEM module, as the output of the decoder.

[0016] The synergistic extraction module SEM performs the following steps:

[0017] The input features are sequentially enhanced by the synergistic extraction module SEM, and the spatial and channel features are processed by the selective state space model Mamba module, and then fused with the input features;

[0018] The selective state space model Mamba module performs the following steps:

[0019] After the input features pass through the normalization layer, they are divided into two branches:

[0020] In the first branch, the input is sequentially processed by a linear layer and an activation function;

[0021] In the second branch, the input is sequentially processed by a linear layer, a depth separable convolution and an activation function, and then input into a 2d selective scanning module for further feature extraction; subsequently, another normalization layer is used to normalize the features;

[0022] The output of the first branch is used to perform element-wise production to combine the two paths of the first branch and the second branch; finally, a linear layer is used to mix the features, and this result is combined with the residual connection to form the output of the VSS block;

[0023] The 2d selective scanning module is composed of three parts: scan expansion, S6 block and scan merging;

[0024] The scan operation expands the input into a sequence along four different directions;

[0025] Then, S6 block extracts features from these sequences to ensure that the scanning process captures different features from each direction;

[0026] Scan merge fuses the features from the four directions together by summation, restoring the output to the same shape as the input.

[0027] The LFEM module performs the following steps:

[0028] The input is expanded to four times the original number of channels by a linear layer, and then split into two parts;

[0029] The first part is transformed multiple times at different scales to capture coarse-scale features at different scales; these outputs are connected and activated using a SiLU function, and then refined using a scSE block;

[0030] The second part is activated by a SiLU function and the elements are multiplied with the output of the first part through a concatenated output to realize the interaction between multi-scale features;

[0031] Finally, the combined features are processed by a linear layer to return the output to the original input dimension.

[0032] The discriminator adopts a U-Net structure, encodes and extracts mixed features using an SRB module, and after the spectral normalization residual block SRB in the middle layer, introduces a GCM module to enhance the global context relationship.

[0033] The GFS guided fusion strategy includes the following steps:

[0034] First, in the generator or discriminator, the decoder features F e and the encoder features F d from the previous layer are concatenated along the channel dimension, and the spatial position is averaged using global average pooling to generate global statistics representing the overall features of the channel;

[0035] Then, the weight W c for each channel is generated through two convolution layers:

[0036] W c =Sigmoid(Conv(ReLU(Conv((AvgPool([F e ,F d ]))))))

[0037] Where Sigmoid represents the Sigmoid function, Conv represents convolution, ReLU represents the ReLU function, and AvgPool represents average pooling.

[0038] The weight coefficient Wc After multiplying the broadcast with the original splicing feature, the channel is reweighted;

[0039]

[0040] Wherein, F c is the weighted feature;

[0041] The decoder feature F e and the encoder feature F d are added and fused, and the spatial attention coefficient W s is obtained through the Sigmoid function:

[0042]

[0043] Finally, the spatial attention coefficient W s and the feature map F c after channel attention are broadcast multiplied to correct the features again.

[0044] The texture loss is calculated based on the texture difference between the converted infrared image and the real infrared image to maintain the texture consistency of the generated image, and is expressed as follows:

[0045]

[0046] Wherein, L texture represents the texture loss, N i represents the normalization factor of the i-th layer, the product of the channel number, height and width of the layer feature map, F i represents the feature map obtained after the image passes through the i-th layer of VGG-19, X represents the visible light image, Y represents the real infrared image, G represents the generator, and G(X) represents the converted infrared image.

[0047] The color loss is calculated based on the color difference between the color distribution of the converted infrared image and the real infrared image to maintain the color consistency of the generated image, and is expressed as follows:

[0048]

[0049] Wherein, L color represents the color loss, and B represents the Gaussian blur convolution operation.

[0050] The system for efficiently converting a visible light image into an infrared light image based on GAN adopts an adversarial network, which comprises:

[0051] The generator adopts InceptioNeXt V2 network as the encoder to extract multi-scale features from the input visible light image.

[0052] In the decoder, global features are extracted using a synergistic extraction module (SEM) and a selective state space model (Mamba) module, a LFEM module supplements local information, and an infrared image is generated;

[0053] The multi-scale features from the encoder and the infrared features generated by the decoder are fused using a GFS guided fusion strategy or a feature channel direct splicing strategy.

[0054] A discriminator is used to evaluate the authenticity of the generated infrared image, and to identify the authenticity of the overall image and each pixel point; the discriminator adopts a U-Net structure, uses a spectral normalization residual block (SRB) as an encoder and a decoder, adds a global context module (GCM), and optimizes the generator through adversarial training.

[0055] The multi-scale features from the encoder and the infrared features generated by the decoder are fused using a GFS guided fusion strategy.

[0056] A composite loss function is used, which includes a structure consistency loss, a mean absolute value loss, a texture loss, a color loss, and an adversarial loss, to minimize the difference between the generated infrared image and the real infrared image.

[0057] The present application has the following advantages and benefits:

[0058] 1. The present application uses InceptioNeXt V2 network as an encoder, and combines SEM, Mamba module and LFEM module as a decoder, to effectively extract and restore multi-scale features in the image and enhance image details.

[0059] 2. The improved U-Net structure is used as the discriminator, and the spectral normalization residual block (SRB) is added, so that the discriminator can evaluate the authenticity of the generated image, and the generator can generate more realistic infrared images.

[0060] 3. The present application introduces a GFS guided fusion strategy to effectively fuse multi-level feature information.

[0061] 4. The present application designs a composite loss function including texture loss, color loss and adversarial loss to improve the texture consistency and physical consistency of the generated image. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 Loss structure diagram of visible light to infrared image conversion system.

[0063] Figure 2 Structure diagram of the generation network of IMGAN, and composition and connection diagram of IN module, SEM module, Mamba module and LFEM module.

[0064] Figure 3 The discriminant network structure diagram is shown, and the GFS module, the SRB module, and the GCM module schematic diagram are shown. DETAILED DESCRIPTION

[0065] The application will be further described below in conjunction with the accompanying drawings and embodiments.

[0066] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0067] The application designs a method for efficiently converting a visible light image into an infrared light image based on GAN, referred to as IMGAN. In the generation network, an InceptioNeXt V2 network that is lighter than ConvNeXt V2 and improves performance is proposed as an encoder, a selective state space model represented by Mamba is introduced into the decoder, and a designed LFEM module for extracting local information is combined to improve the fidelity of the image. A guided fusion strategy GFS is designed to ensure that the generated infrared image not only retains the structural information of the visible light image, but also embodies the physical characteristics of infrared imaging. Texture loss and color loss are designed as additional auxiliary functions to constrain the bilateral mapping to solve the problems of rough texture and color difference. The method is evaluated on four different datasets, and the results are compared with existing architectures including Pix2Pix, ThermalGAN and InfraGAN. Compared with the InfraGAN architecture, the SSIM and PSNR of the method on the VEDAI dataset are increased by 2.38% and 7.28%, respectively.

[0068] An input pair of visible light images is input, with a size of (BxCxHxW), wherein B is a batch size, C is a channel number, and H and W are the height and width of the image, respectively;

[0069] The generator extracts multi-scale deep features from the input visible light image by using the InceptioNeXt V2 network as an encoder;

[0070] The global features are extracted by using a collaborative extraction module SEM and a selective state space model Mamba module in the decoder of the generator, and the LFEM module supplements local information, gradually restores the details of the input image, and generates an infrared image;

[0071] The GFS guided fusion strategy or feature direct splicing by channel is used to fuse the features from the encoder and the infrared features generated by the decoder to improve the network performance.

[0072] The discriminator evaluates the authenticity of the generated image, and discriminates the true and false of the overall image and each pixel point. The discriminator adopts the U-Net structure, uses the spectral normalization residual block (SRB) as the encoder and the decoder, adds the global context module (GCM) to improve the performance of the discriminator, and optimizes the generator through the adversarial training to make the generated image more real.

[0073] The composite loss function is calculated and minimized, the composite loss function includes structure consistency loss, mean absolute value loss, texture loss, color loss and adversarial loss, and the difference between the generated infrared image and the real infrared image is minimized.

[0074] As shown in Figure 1 The specific technical solutions are as follows:

[0075] The visible light to infrared image conversion method based on the generative adversarial network of the application comprises the following steps:

[0076] 1. Input data: input a pair of visible light images, the size is BxCxHxW, wherein B is the batch size, C is the number of channels (RGB is 3), H and W are the height and width of the image (for example, 256x256 pixels). All input images are normalized so that the pixel value of the image is in the range of [-1, 1].

[0077] 2. Generator architecture:

[0078] Encoder: InceptioNeXt V2 network is used as the encoder. The network uses multi-scale convolution kernels to extract global features and local details of the image through convolution operations of different sizes. The output of the encoder is a multi-dimensional deep feature, which contains high-level information of the input image.

[0079] Decoder: The decoder is composed of SEM module, Mamba module and LFEM module. The Mamba module gradually restores the low-resolution features of the image and maps them to higher-resolution image details. The LFEM module is used to enhance the local details of the image and improve the spatial resolution of the generated image.

[0080] Guided fusion strategy: this module is used to fuse the global features from the encoder and the local features from the decoder. Through an adaptive weighting mechanism, the GFS module fuses the features at multiple scales to optimize the quality of the generated image.

[0081] 3. Discriminator architecture:

[0082] The discriminator adopts an improved U-Net structure, which designs an SRB module, i.e., a residual block with spectral normalization technology, to improve the stability and discrimination accuracy of the network. Spectral normalization can effectively suppress mode collapse in the training process of the discriminator and improve the authenticity evaluation of the generated image.

[0083] The task of the discriminator is to determine whether the input image is a real image or a generated image. By comparing the infrared image generated by the generator with the real infrared image, the discriminator provides feedback signals to the generator, prompting the generator to generate more realistic infrared images.

[0084] 4. Loss function:

[0085] Adversarial loss: Through the adversarial training of the generative adversarial network, the generator generates images to "deceive" the discriminator, and the discriminator guides the optimization of the generator by distinguishing true and false images. Adversarial loss can push the generator to generate more realistic infrared images.

[0086] Texture loss: Calculate the texture difference between the generated image and the real image to ensure that the texture structure of the generated image is similar to that of the real image.

[0087] Color loss: Calculate the difference between the color distribution of the generated image and the real infrared image to ensure that the color features of the generated image conform to the physical properties of the infrared image.

[0088] 5. Training process:

[0089] The generator and discriminator are optimized in an alternating training manner. First, the generator generates an infrared image based on the input visible light image, and then the discriminator evaluates the generated image and provides error feedback to the generator. The generator adjusts the weights using the feedback signal to gradually improve the image generation quality.

[0090] During training, the generator and discriminator are optimized simultaneously through a composite loss function to ensure that the generated image achieves a high level in terms of texture, color, and physical consistency.

[0091] 6. Output:

[0092] After training is complete, the generator can generate corresponding infrared images based on input visible light images. The generated infrared images are highly consistent with real infrared images in terms of details, texture, and color, and are suitable for target detection, image enhancement, and other applications in low-visibility environments.

[0093] The present invention includes the following steps:

[0094] (1) Generator network design, as shown in Figure 2

[0095] ​Encoder: ConvNeXt is a modern convolutional neural network (CNN) architecture inspired by the design principles of vision transformers (ViTs). It aims to bridge the gap between traditional CNNs and ViTs by incorporating several key design elements of transformers while preserving the efficiency and simplicity of convolutional operations. ConvNeXt V2 is an evolution of the ConvNeXt architecture, which drops Layer Scale and uses GRN. On multiple benchmarks, ConvNeXt V2 outperforms ConvNeXt, demonstrating its stronger adaptability, better performance, and effective mitigation of feature collapse issues in different tasks.

[0096] In the ConvNeXt V2 block, the feature is first processed by a 7x7 deep convolution to achieve spatial information interaction. The network introduces GRN (Global Response Normalization) to improve the traditional Batch Normalization. It normalizes the response of each channel, improves the stability during training, and enhances the convergence of the model. Compared with Batch Normalization, GRN can better adapt to large-scale image datasets, especially for deep network structures, with significant effects. Then, a multi-layer perceptron (MLP) is used to help the network integrate features efficiently, further improving the accuracy. A mechanism called Shifted Window Attention is also introduced to calculate the attention of the local window of the image. This method can effectively capture spatial relationships while avoiding the computational overhead of global attention. Finally, DropPath is used for regularization techniques similar to Dropout. It prevents overfitting by randomly dropping some paths during training, thereby improving the generalization ability of the model. Through these designs, ConvNeXt V2 retains the classic convolutional neural network architecture while further integrating modern design ideas, improving performance and training efficiency.

[0097] InceptionNeXt, however, introduces an innovative approach that decomposes the large-kernel depth convolutions in ConvNeXt into four parallel branches: one small square kernel, two orthogonal band kernels, and one identity mapping. This decomposition not only reduces the memory access cost but also significantly improves the throughput. Inspired by this design, a new variant of the ConvNeXt V2 architecture is proposed, called InceptioNext V2, which replaces the separable convolutions in ConvNeXt v2 with Inception blocks, similarly decomposing the depth convolutions in ConvNeXt V2 by splitting the input features into four channel partitions. The first three partitions are each composed of one-eighth of the input feature channels, while the fourth partition occupies five-eighths. Depth convolutions with different kernel sizes are applied to the first three features, using 3x3, 1x11, and 11x1 different-sized convolutions as kernels, while the fourth feature remains unchanged to preserve the original information and reduce memory overhead, defined as IDConv. InceptionNeXt V2 achieves superior performance while reducing computational overhead.

[0098] The encoder part of the generator includes an InceptioNeXt V2 network that uses multiple convolution kernels of different sizes to extract multi-scale features of the image. The network takes a three-channel RGB image I ∈ R W×H×3 as input and outputs a global feature map where d is the down-sampling coefficient, W, H are the width and height of the image, respectively, C is the number of channels, and R is a real number.

[0099] Decoder: The global feature extractor based on SSM uses mamba as the key component. After Layer Normalization, the input is divided into two branches. In the first branch, the input goes through a linear layer followed by an activation function. In the second branch, the input is processed through a linear layer, a depthwise separable convolution, and an activation function, then input into a 2D selective scanning (SS2D) module for further feature extraction. Subsequently, the features are normalized using Layer Normalization, then element-wise production is performed using the output of the first branch to merge the two paths. Finally, a linear layer is used to mix the features, and this result is combined with the residual connection to form the output of the VSS block. The SS2D module consists of three main parts: scan expansion, S6 block, and scan merging. The scan operation expands the input into sequences along four different directions (top-left to bottom-right, bottom-right to top-left, top-right to bottom-left, and bottom-left to top-right). Then, the S6 block extracts features from these sequences, ensuring that the scanning process captures different features from each direction. The scan merging operation fuses the features from the four directions by summation, restoring the output to the same shape as the input.

[0100] Local Feature Extractor LFEM: This model is designed for efficient and effective local feature extraction, utilizing multi-scale processing through the combination of depthwise separable convolution, pooling, and attention mechanisms. Initially, the input is expanded in channels by a linear layer to four times the original, then split into two parts. The first part undergoes multiple transformations at different scales: "dwconv5x5" and "conv1x1" capture medium-scale features, "dwconv3x3" and "conv1x1" capture fine-scale features, and "max-pooling" and "conv1x1" capture coarse-scale features. These outputs are concatenated and activated using the SiLU function, then refined using the scSE block, which emphasizes important features through spatial and channel recalibration. The second part of the split input is activated by SiLU and multiplied by the elements of the fine concatenation output, enabling interaction between multi-scale features. Finally, the combined features are processed by a linear layer to return the output to the original input dimension. This structure effectively captures multi-scale features, reduces computational cost through depthwise separable convolution, and enhances the model's focus on important features using attention mechanisms, resulting in efficient and robust feature extraction.

[0101] Convolution operations in the decoder SEM enhance feature extraction by capturing spatial hierarchies before passing data to Mamba, which is good at modeling long-term dependencies and sequential relationships with linear complexity compared to Transformers. This synergy enables the model to effectively process spatial and channel features, improving performance on image generation tasks. The LFEM module is used to enhance the details of local regions in the image, improving the performance of the generated image in local regions. The decoder is used to gradually restore the low-resolution features of the image, generating higher-resolution infrared image details.

[0102] See Figure 2 Fuse in (a) Generator, using GFS guided fusion strategy or feature concatenation directly by channel.

[0103] Guided Fusion Strategy GFS:

[0104] The present invention proposes a guided fusion strategy (GFS) as the core feature fusion module, seamlessly integrated into the generator and discriminator. The main function of GFS is to effectively fuse the rich features extracted by the encoder with the upsampled features, while adaptively adjusting the weights to highlight the key features of the infrared image. In the generator, GFS ensures that the generated infrared image retains the structural details of the visible light image while capturing the physical properties of infrared imaging. In the discriminator, GFS promotes guided feature fusion, enhancing the discriminator's ability to evaluate the authenticity and physical consistency of the infrared image. This dual-role strategy strengthens the constraints on the generated image, improving the synergy between the generation and recognition processes.

[0105] The GFS feature fusion module adopts an adaptive weighting strategy to effectively fuse the features extracted by the encoder and the infrared features generated by the decoder. The experimental results show that, in the generator, the GFS guided fusion strategy or the feature channel direct splicing is used for different data sets, the features from the encoder and the infrared features generated by the decoder are fused; and in the discriminator, the GFS fusion is better.

[0106] First, the decoder features F e from the previous layer are spliced together with the encoder features F d according to the channel dimension, the spatial position is averaged using global average pooling to generate global statistics representing the overall features of the channel, reflecting the importance of the channel between the entire feature map, and then the weight W c of each channel is generated through two convolution layers.

[0107] W c = Sigmoid(Conv(ReLU(Conv((AvgPool([F e , F d ]))))))

[0108] After obtaining the weight coefficient W c , the weights reflect the importance of the channel, and are multiplied with the original spliced features through broadcast, reweighting the channel, highlighting important features and suppressing unimportant features, thereby enhancing the ability of feature representation. After one convolution operation, the channel number is reduced to one half to adapt to the input from the spatial attention. According to the global context information, the weight of each channel is dynamically adjusted, so that the model can adaptively select the most relevant features under different inputs.

[0109]

[0110] The two features are added and fused, only the intermediate feature is convolved to simplify the calculation, and the channel number is compressed to 1 to obtain the spatial attention map, focusing on important spatial positions, and the spatial attention coefficient W s is obtained through the Sigmoid function.

[0111]

[0112] Finally, the spatial attention coefficient is multiplied with the feature map F c through channel attention by broadcast, which is used to correct the features again to focus on spatial features. In this way, the network can focus on more important areas in the channel and space at the same time, and complete adaptive weight adjustment to help the model better fuse features.

[0113]

[0114] (2) Discriminator network design, such as Figure 3 As shown:

[0115] The discriminator adopts a U-Net structure, where SRB includes residual blocks and spectral normalization to suppress mode collapse during training and enhance the ability to discriminate generated images.

[0116] The discriminator is based on the U-Net discriminator and has been modified according to training requirements. The discriminator takes a 512×512×4 tensor as input. Downsampling is performed using average pooling, while spectral residual blocks (SRBs) encode and extract mixed features. A Global Context Module (GCM) is introduced in an intermediate layer with 256 channels to enhance global contextual relationships; the GCM is executed after the SRB is completed within an intermediate SRB. At the encoder, the model outputs a 4×4×2048 tensor, which is simplified to a scalar statistic by element-wise summation. This statistic is passed through a fully connected layer and then through a sigmoid activation function to produce a 1×1 scalar output, evaluating the overall realism of the image. The decoder is symmetric to the encoder. Upsampling is performed using linear interpolation, generating a 512×512×1 matrix to evaluate the realism of each pixel. To enhance the discriminator's capabilities, GFS is used between layers to effectively guide feature integration.

[0117] The detailed process is as follows: inspired by the denoising diffusion probability model, a self-attention mechanism is added in the middle layer of the encoder and the decoder, and the attention mechanism can enhance the ability of the discriminator to capture long-term dependencies in the input data, thereby improving the discriminant ability by considering the global context rather than relying only on local features. In addition, self-attention enables the discriminator to dynamically adjust the importance of various features according to the relevance of the identification task, thereby more effectively processing complex and structured inputs. This adaptability enhances the robustness and generalization ability of the model across different datasets. And a guided fusion strategy is added between layers. The discriminator produces two outputs: one based on low-frequency information to evaluate the authenticity of the overall image, and the other to evaluate the authenticity of each pixel, focusing on high-frequency details. This method has been proven to be very effective in image generation tasks. In the architecture of the discriminator, SRB is composed of convolutional layers and activation functions, and batch normalization is omitted. This design enables the network to learn the residual function (difference) rather than a direct mapping, facilitating the training of deeper networks. Spectral normalization is applied to the residual blocks within the discriminator. Spectral normalization is a technique that controls the Lipschitz constant of the network by normalizing the spectral norm (largest singular value) of each layer's weight matrix. By applying spectral normalization to the discriminator, it can be prevented from becoming disproportionately stronger than the generator. Maintaining this balance is crucial for the training dynamics of GANs, as it ensures that neither network dominates the other. If this balance cannot be maintained, problems such as mode collapse or gradient vanishing can occur.

[0118] (3) Loss function design:

[0119] The composite loss function includes the following five parts:

[0120] Structural consistency loss, using the SSIM loss function to maintain the structural consistency of the visible light image and the infrared image. Mean absolute value loss L1, mainly acting on the low-frequency information of the entire image, constraining the infrared image from the pixel level.

[0121] Texture loss, calculated based on the texture difference between the converted infrared image and the real infrared image to maintain the texture consistency of the generated image;

[0122] Color loss, calculated based on the color difference between the color distribution of the converted infrared image and the real infrared image to maintain the color consistency of the generated image;

[0123] Adversarial loss, through the adversarial training of the generative adversarial network, to push the generator to generate higher quality infrared images.

[0124] 1. Texture loss:

[0125] In image generation tasks, L1 loss mainly focuses on the accurate matching of pixel values, and calculates the absolute difference between the generated image and the real image in pixel values, which is accurate in low-frequency components, but weak in capturing high-frequency details. SSIM loss mainly measures the similarity of local structures. It measures the similarity of brightness, contrast and structure between the generated image and the real image. It focuses on the perceptual quality and local structure of the image, and is more sensitive to medium and high frequency details. Although SSIM has better capturing ability in medium and high frequencies than L1 loss, it is still insufficient to capture complex texture details. Therefore, introducing texture loss can further enhance the detail quality of the generated image and improve the realism of the generated image, especially when dealing with complex textures and rich details.

[0126] VGG-19 model has a wide range of applications in computer vision, such as image classification, object detection, image style transfer, compared with other convolutional neural network models, VGG-19 has a deeper network structure, which makes it better to learn the complex features of the image, and enhances the expression ability of the model. The deep features of VGG-19 can capture higher-level semantic information, and the shallow features can capture lower-level detail information. However, choosing too deep a layer may cause the gradient vanishing problem of the loss function, and choosing too shallow a layer may cause overfitting or loss of some important semantic information. Therefore, when constructing texture loss, the middle layer is selected to extract features for loss, which can capture the required texture semantic information and will not cause gradient vanishing. According to experience, the conv4_2 layer of VGG-19, i.e. the first 22 layers, is selected, the first layer of VGG-19 network is modified to adapt to single-channel input, then the generated image and the real label image are input into the VGG-19 network to obtain their feature representations up to that layer. As shown in the figure, the visualization image of VGG-19 extracting features. Calculate the squared error loss between the feature representations of the generated image and the target image at each layer, and accumulate them to make the final total loss reflect the differences at multiple levels. In this way, the generated model will consider the feature differences at multiple levels during the optimization process, ensuring that the generated image is similar to the target image at different scales and detail levels. After the above process, the texture loss is obtained, which is shown in the following formula.

[0127]

[0128] where N i is the normalization factor of the i-th layer, the product of the channel number, height and width of the layer feature map, F i is the feature map obtained after the image passes through the i-th layer of VGG-19.

[0129] 2. Color loss:

[0130] To ensure the color style between the generated image and the real image is similar, color loss is used to constrain the same color distribution between the generated image and the real image. To reduce the color difference between the generated image and the real image, Gaussian blur is applied and the Euclidean distance between the obtained feature representations is calculated. The implementation is as follows: first, a Gaussian blur kernel is constructed, and then the Gaussian blur kernel is used as a convolution kernel to perform convolution operation on the image to obtain the blurred image; then the mean square error function of the input image and the target image is calculated as the loss function. As shown in the formula.

[0131]

[0132] Wherein, B represents the Gaussian blur convolution operation.

[0133] 3. Other loss: structure consistency loss, using SSIM loss function to keep the structure consistent between the visible light image and the infrared image. SSIM is an index for evaluating the local contrast, brightness and structural similarity between two images. The average absolute value loss L1 mainly acts on the low frequency information of the whole image, and constrains the infrared image from the pixel level.

Claims

1. A method for efficiently converting visible light images into infrared light images based on GAN, characterized in that, Includes the following steps: Input a visible light image, and convert it into an infrared image using an adversarial network; For the generator, the InceptioNeXt V2 network is used as the encoder to extract multi-scale features from the input visible light image; In the decoder, the SEM co-extraction module and the Mamba selective state-space model module are used to extract global features, and the LFEM module is used to supplement local information to generate infrared images. Using GFS-guided fusion strategies or features directly stitched together by channel, multi-scale features from the encoder and infrared features generated by the decoder are fused together. For the discriminator, the authenticity of the generated infrared image is evaluated to distinguish between the authenticity of the overall image and the authenticity of each pixel. The discriminator adopts a U-Net structure, using spectral normalized residual blocks (SRBs) as the encoder and decoder, and optimizes the generator through adversarial training. A GFS-guided fusion strategy is used to fuse features from the encoder and infrared features generated by the decoder. Calculate and minimize a composite loss function, which includes structural consistency loss, mean absolute value loss, texture loss, color loss, and adversarial loss, to facilitate minimizing the difference between the generated infrared image and the real infrared image; The decoder of the generator performs the following steps: The input features are sequentially processed by the collaborative extraction module SEM-enhanced feature extraction and the selective state-space model Mamba module to process spatial and channel features before being fused with the input features. The fused features are then processed by the LFEM module and concatenated with the fused features to serve as the output of the decoder. The LFEM module performs the following steps: The input is expanded to four times its original size through a linear layer and then split into two parts; The first part involves multiple transformations at different scales to capture coarse-scale features at different scales. These outputs are concatenated and activated using SiLU functions, and then refined using scSE blocks; The second part is activated by the SiLU function, and the elements are multiplied by the output of the first part through concatenation to realize the interaction between multi-scale features; Finally, the combined features are processed through a linear layer, and the output is returned to the original input dimension. The GFS boot fusion strategy includes the following steps: First, in the generator or discriminator, the features from the decoder in the previous layer are... and encoder features The data is concatenated according to the channel dimension, and global average pooling is used to average the spatial locations to generate a global statistic that represents the overall characteristics of the channel. Then, weights for each channel are generated through two convolutional layers. : ； in, This represents the Sigmoid function. Represents convolution. Represents the ReLU function Indicates average pooling; Obtain the weighting coefficients Then, the original splicing features are broadcast and multiplied to reweight the channels; ； in, The features are weighted; decoder features and encoder features The two components are added together and then fused, and the spatial attention coefficients are obtained by using the Sigmoid function. : ； Finally, the spatial attention coefficient Feature map with channel attention Broadcast multiplication is performed to further correct the features.

2. The method for efficiently converting visible light images into infrared light images based on GAN as described in claim 1, characterized in that, The visible light image is a preprocessed image, and the preprocessing step includes normalizing the image so that the image pixel values ​​are in the range of [-1, 1].

3. The method for efficiently converting visible light images into infrared light images based on GAN as described in claim 1, characterized in that, The SEM co-extraction module performs the following steps: The input features are sequentially passed through depthwise separable convolution, normalization layer, and linear layer, connected and activated by SiLU function, and then output through another linear layer.

4. The method for efficiently converting visible light images into infrared light images based on GAN as described in claim 1, characterized in that, The selective state-space model Mamba module performs the following steps: After the input features pass through the normalization layer, they are divided into two branches: In the first branch, the input passes through a linear layer and an activation function in sequence; In the second branch, the input is processed sequentially through a linear layer, a depthwise separable convolution, and an activation function, and then fed into a 2D selective scanning module for further feature extraction; subsequently, another normalization layer is used to normalize the features. The output of the first branch is used to perform element-wise production to merge the two paths of the first and second branches; finally, features are blended using a linear layer and this result is combined with residual connections to form the output of the VSS block. The 2D selective scanning module consists of three parts: scan extension, S6 block, and scan merging. The scanning operation unfolds the input into a sequence along four different directions; Then, the S6 block extracts features from these sequences to ensure that the scanning process captures different features from each direction; Scan merging combines features from four directions by summing them, restoring the output to the same shape as the input.

5. The method for efficiently converting visible light images into infrared light images based on GAN as described in claim 1, characterized in that, The discriminator adopts a U-Net structure, uses a spectral normalized residual block (SRB) module to encode and extract mixed features, and introduces a GCM module after the intermediate spectral normalized residual block (SRB) module to enhance the global context relationship.

6. The method for efficiently converting visible light images into infrared light images based on GAN as described in claim 1, characterized in that, The texture loss, calculated based on the texture difference between the converted infrared image and the real infrared image to maintain the texture consistency of the generated image, is expressed as follows: ； in, Indicates texture loss. N i Represents the normalization factor for the i-th layer, which is the product of the number of channels, height, and width of the feature map of that layer. F i Let X represent the feature map obtained after the image passes through the i-th layer of VGG-19, where X represents the visible light image, Y represents the true infrared image, G represents the generator, and G(X) represents the converted infrared image. The color loss is calculated based on the color difference between the color distribution of the converted infrared image and the real infrared image, in order to maintain the color consistency of the generated image, as expressed below: ； in, Indicates color loss. B This represents the Gaussian blur convolution operation.

7. A system for efficiently converting visible light images into infrared light images based on GAN, employing an adversarial network, characterized in that: include: The generator uses the InceptioNeXt V2 network as the encoder to extract multi-scale features from the input visible light image; In the decoder, global features are extracted using the collaborative extraction module SEM and the selective state-space model Mamba module, local information is supplemented by the LFEM module, and infrared images are generated. Using GFS-guided fusion strategies or features directly stitched together by channel, multi-scale features from the encoder and infrared features generated by the decoder are fused together. The decoder of the generator is configured to execute: The input features are sequentially processed by the collaborative extraction module SEM-enhanced feature extraction and the selective state-space model Mamba module to process spatial and channel features before being fused with the input features. The fused features are then processed by the LFEM module and concatenated with the fused features to serve as the output of the decoder. The LFEM module is configured to execute: The input is expanded to four times its original size through a linear layer and then split into two parts; The first part involves multiple transformations at different scales to capture coarse-scale features at different scales. These outputs are concatenated and activated using SiLU functions, and then refined using scSE blocks; The second part is activated by the SiLU function, and the elements are multiplied by the output of the first part through concatenation to realize the interaction between multi-scale features; Finally, the combined features are processed through a linear layer, and the output is returned to the original input dimension. The discriminator is used to evaluate the authenticity of the generated infrared image, distinguishing between the authenticity of the overall image and the authenticity of each pixel. The discriminator adopts a U-Net structure, using spectral normalized residual blocks (SRBs) as the encoder and decoder, and optimizes the generator through adversarial training. A GFS-guided fusion strategy is used to fuse features from the encoder and infrared features generated by the decoder. The GFS boot fusion strategy is configured to execute: First, in the generator or discriminator, the features from the decoder in the previous layer are... and encoder features The data is concatenated according to the channel dimension, and global average pooling is used to average the spatial locations to generate a global statistic that represents the overall characteristics of the channel. Then, weights for each channel are generated through two convolutional layers. : ； in, This represents the Sigmoid function. Represents convolution. Represents the ReLU function Indicates average pooling; Obtain the weighting coefficients Then, the original splicing features are broadcast and multiplied to reweight the channels; ； in, The features are weighted; decoder features and encoder features The two components are added together and then fused, and the spatial attention coefficients are obtained by using the Sigmoid function. : ； Finally, the spatial attention coefficient Feature map with channel attention Perform broadcast multiplication to further correct the features; A composite loss function, comprising structural consistency loss, mean absolute value loss, texture loss, color loss, and adversarial loss, is used to minimize the difference between the generated infrared image and the real infrared image.

Images