An infrared image generation method combining space-frequency domain

Abstract

The application discloses an infrared image generation method combining space and frequency domains, comprising the following steps: 1, obtaining training set images; 2, constructing a GAN network combining space and frequency domains; 3, inputting the training set into the GAN network combining space and frequency domains to perform training, and obtaining a trained GAN network combining space and frequency domains; and 4, generating infrared images based on the trained GAN network combining space and frequency domains. The method is simple in steps and reasonable in design, the GAN network combining space and frequency domains realizes generation of visible light images into infrared images, retains frequency domain features and strengthens edge constraints, and utilizes a multi-scale discriminator and a composite loss function to optimize adversarial training, improve the generation quality of infrared images and solve the problem that it is difficult to balance local details and overall consistency.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for generating infrared images in the combined spatial and frequency domains. Background Technology

[0002] Infrared imagery, with its unique advantages of being insensitive to changes in lighting and penetrating complex weather conditions such as smoke, rain, and snow, occupies an irreplaceable position in fields such as autonomous driving environmental perception, drone remote sensing monitoring, and nighttime security early warning. Real infrared data comes from two sources: direct imaging and acquisition from publicly available datasets. In direct imaging, infrared thermal imagers cost approximately 5-8 times more than visible light cameras, and are limited by physical shooting conditions, making it difficult to meet the data requirements of "all scenes, all times." Existing publicly available visible light-infrared data suffers from limited scene coverage and alignment accuracy, failing to meet the diverse data needs of downstream models. Therefore, it is necessary to convert easily obtainable visible light images into infrared images that conform to thermal radiation characteristics, thereby achieving low-cost expansion of infrared image datasets.

[0003] However, existing methods still have limitations: GAN-type methods mostly focus on improving the generator and discriminator structures, and have limited ability to acquire multi-scale features. Edge-guided methods enhance contour consistency by introducing edge loss, but due to the lack of frequency domain constraints, the generated images have coarse textures and low quality.

[0004] Furthermore, the conversion of visible light to infrared images requires not only realistic infrared visual details and textures but also adherence to the physical laws of thermal radiation. Current optimization methods based on the spatial domain generally focus on extracting detailed features while lacking global and structural constraints, making it difficult to balance local details with overall consistency, resulting in low quality. Secondly, although edge guidance and frequency domain constraints have been introduced to address the issue of consistency between texture and thermal radiation laws, deep fusion of the spatial and frequency domains has not been achieved, limiting the improvement effect.

[0005] Therefore, a well-designed method for generating infrared images using a joint spatial-frequency domain is needed. This method uses a joint spatial-frequency domain GAN network to generate infrared images from visible light images, preserving frequency domain features and strengthening edge constraints. Furthermore, it utilizes a multi-scale discriminator and a composite loss function to optimize adversarial training, improve the quality of generated infrared images, and solve the problem of balancing local details with overall consistency. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a method for generating infrared images by combining spatial and frequency domains, which addresses the shortcomings of the prior art. The method is simple in steps and reasonable in design. It uses a combined spatial and frequency domain GAN network to generate infrared images from visible light images, preserves frequency domain features and strengthens edge constraints. Furthermore, it utilizes a multi-scale discriminator and a composite loss function to optimize adversarial training, improve the quality of generated infrared images, and solve the problem of balancing local details with overall consistency.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for generating infrared images in the joint spatial frequency domain, characterized in that the method includes the following steps: Step 1: Acquiring paired visible light and infrared images: Select pairs of visible light and infrared images from the public dataset and use these pairs as the training set; wherein, one visible light image and one infrared image are used as a set of training data. Step 2: Construct a joint spatial-frequency domain GAN network; wherein the joint spatial-frequency domain GAN network includes a generator and a discriminator, the generator includes an encoder and a decoder, the encoder includes an SPPF_PSA module, and the decoder includes an EDFFN_SEE module; the discriminator includes a first average pooling layer, a second average pooling layer, a D1 discriminator, a D2 discriminator, and a D3 discriminator; Step 3: Input the training set into the joint spatial frequency domain GAN network for training to obtain the trained joint spatial frequency domain GAN network. Step 4: Generate infrared images based on the trained joint spatial frequency domain GAN network.

[0008] The aforementioned method for generating infrared images in the joint spatial frequency domain further includes an encoder comprising a Conv+LeakyReLU layer, a first downsampling layer, a second downsampling layer, a third downsampling layer, and a Conv+ReLU layer; a decoder comprising a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a Tconv+Tanh layer; a first SPPF_PSA module is added between the second downsampling layer and the third downsampling layer; and the output of the Conv+ReLU layer is added to a second SPPF_PSA module. A first EDFFN_SEE module is added between the third upsampling layer and the fourth upsampling layer, and a second EDFFN_SEE module is added between the fourth upsampling layer and the Tconv+Tanh layer. The output of the Conv+LeakyReLU layer and the output of the second EDFFN_SEE module are concatenated in the channel dimension and then input into the Tconv+Tanh layer. The output of the first downsampling layer and the output of the first EDFFN_SEE module are concatenated in the channel dimension and then input into the fourth upsampling layer. The output of the first SPPF_PSA module and the output of the second upsampling layer are concatenated in the channel dimension and then input into the third upsampling layer. The output of the third downsampling layer and the output of the first upsampling layer are concatenated in the channel dimension and then input into the second upsampling layer.

[0009] The above-mentioned method for generating infrared images in the joint spatial frequency domain, further, step three, is as follows: Step 301: The visible light image is input into the encoder for processing using a computer to obtain the first to sixth feature maps; Step 302: Input the first feature map to the sixth feature map into the decoder for processing to obtain the output feature map, and record it as the infrared generated image; Step 303: Stitch the visible light image and the corresponding infrared generated image along the channel dimension to obtain the stitched feature map; Step 304: Process the spliced ​​feature map through the first average pooling layer to obtain the first dimensionality-reduced feature map; process the first dimensionality-reduced feature map through the second average pooling layer to obtain the second dimensionality-reduced feature map. Step 305: Process the spliced ​​feature map, the first dimensionality reduction feature map, and the second dimensionality reduction feature map through the D1 discriminator, the D2 discriminator, and the D3 discriminator, respectively, to obtain the first discriminant feature map, the second discriminant feature map, and the third discriminant feature map; Step 306, according to The adversarial loss of the discriminator is obtained. ;in, Representing visible light images and true infrared images The value of the i-th pixel in the first discriminative feature map output by the D1 discriminator. Representing visible light images and infrared generated images The i-th pixel value in the first discriminant feature map output by the D1 discriminator, where i and m1 are positive integers; 1≤i≤m1, and m1 is the total number of pixels in the first discriminant feature map; Representing visible light images and true infrared images The j-th pixel value in the second discriminative feature map output by the D2 discriminator. Representing visible light images and infrared generated images The j-th pixel value in the second discriminant feature map output by the D2 discriminator, where j and m2 are positive integers; 1≤j≤m2, and m2 is the total number of pixels in the second discriminant feature map; Representing visible light images and true infrared images The k-th pixel value in the third discriminative feature map output by the D3 discriminator Representing visible light images and infrared generated images The value of the kth pixel in the third discriminant feature map output by the D3 discriminator, where k and m3 are positive integers; 1≤k≤m3, and m3 is the total number of pixels in the third discriminant feature map; Step 307, according to The perceptual loss function is obtained. ;in, It is a positive integer. ,and Represents true infrared images ReLU in the VGG19 network model Feature map output from layer 1 Indicates infrared generated image ReLU in the VGG19 network model Feature map output from layer 1 ; , and Relu The number of channels, length, and width of the feature map output from layer 1; ReLU in the VGG19 network model Manhattan distance between the two feature maps output by layer 1; according to To obtain frequency loss Where Fq(·) represents the q-th pixel value in the feature map output by performing a two-dimensional fast Fourier transform on the spatial dimension of the image, Re[·] and Im[·] represent taking the real and imaginary parts of the complex number of the pixel value, respectively; q and Q are positive integers, and Q is the total number of pixels in the feature map output by the two-dimensional fast Fourier transform. Step 308, according to The total loss function is obtained. ;in, and These represent the first weight, second weight, third weight, and fourth weight, respectively. This represents the L1 loss between the real infrared image and the generated infrared image; Step 309: The computer uses the Adam optimization method. It inputs a set of training data and first utilizes generative adversarial loss. The D1, D2, and D3 discriminators are trained until the adversarial loss is obtained. Minimum-level D1 discriminator, D2 discriminator, and D3 discriminator; Step 30A: The computer uses the Adam optimization method, inputs the training data, and, under the D1 discriminator, D2 discriminator, and D3 discriminator determined in step 309, uses the total loss function. Train the network's generator until the total loss function is reached. Minimum, complete training on this set of training data; Step 30B: Follow the methods from Step 309 to Step 30A until the entire training set is trained, completing one iteration of training; Step 30C, repeat steps 309 to 30B, iterate training until the preset number of rounds is met, and obtain the trained joint spatial frequency domain GAN network.

[0010] The aforementioned method for generating infrared images in the joint spatial frequency domain further includes step 301, in which a visible light image is input into an encoder for processing using a computer to obtain the first to sixth feature maps. The specific process is as follows: The visible light image is input into a Conv+LeakyReLU layer for processing using a computer to obtain the first feature map; The first feature map is processed through the first downsampling layer to obtain the second feature map; The second feature map is processed through the second downsampling layer to obtain the third feature map; The third feature map is processed by the first SPPF_PSA module to obtain the fourth feature map; The fourth feature map is processed through the third downsampling layer to obtain the fifth feature map; The fifth feature map is processed by the Conv+ReLU layer and then by the second SPPF_PSA module to obtain the sixth feature map.

[0011] The aforementioned method for generating infrared images in the joint spatial frequency domain further includes step 302, where the first to sixth feature maps are input into the decoder for processing to obtain the output feature map. The specific process is as follows: The sixth feature map is processed through the first upsampling layer to obtain the first upsampling feature map; The first upsampled feature map and the fifth feature map are concatenated and then input into the second upsampled layer for processing to obtain the second upsampled feature map. The second upsampled feature map and the fourth feature map are concatenated and then input into the third upsampled layer for processing to obtain the third upsampled feature map. The third upsampled feature map is processed by the first EDFFN_SEE module to obtain the fourth upsampled feature map; The fourth upsampled feature map and the second feature map are concatenated and then input into the fourth upsampled layer for processing to obtain the fifth upsampled feature map. The fifth upsampled feature map is processed by the second EDFFN_SEE module to obtain the sixth upsampled feature map; The sixth upsampled feature map and the first feature map are concatenated and then fed into the Tconv+Tanh layer for processing to obtain the output feature map.

[0012] The aforementioned method for generating infrared images in the joint spatial frequency domain further includes, in which the first SPPF_PSA module and the second SPPF_PSA module both include an SPPF module and a C2PSA module, wherein the C2PSA module has three PSABlock branches; the output of the SPPF module is input to the C2PSA module, and the output of the C2PSA module is the output of the first SPPF_PSA module and the second SPPF_PSA module; Both the first EDFFN_SEE module and the second EDFFN_SEE module include an EDFFN module, a Scharr edge extraction operator, batch normalization, a Gelu activation function layer, and an element-wise multiplication module.

[0013] The aforementioned method for generating infrared images in the joint spatial frequency domain further employs the same method for obtaining a fourth upsampled feature map from a third upsampled feature map via the first EDFFN_SEE module, and for obtaining a sixth upsampled feature map from a fifth upsampled feature map via the second EDFFN_SEE module. The specific process is as follows: Step A: Denote the third and fifth upsampled feature maps as input feature maps; denote the fourth and sixth upsampled feature maps as enhanced feature maps; Step B: Process the input feature map through the EDFFN module to obtain the first intermediate feature map; Step C: Fuse the input feature map and the first intermediate feature map by element-wise addition to obtain the fused feature map; Step D: Use the Scharr edge extraction operator to extract edges from the fused feature map to obtain the edge feature map; Step E: Batch normalize the edge feature map to obtain the normalized edge feature map; Step F: Pass the normalized edge feature map through a Gelu activation function layer to obtain a sparse feature map; Step G: The sparse feature map is used as an edge attention mask and combined with the input feature map through an element-wise multiplication module to perform a Hadamard product operation, resulting in an enhanced feature map.

[0014] Compared with the prior art, the present invention has the following advantages: 1. The method of the present invention has simple steps and reasonable design, and solves the problem of balancing local details and overall consistency when generating infrared images based on visible light images.

[0015] 2. This invention incorporates an SPPF_PSA module into the joint spatial-frequency domain GAN network encoder. This module characterizes targets of different scales in visible light images and extracts key information from complex scenes, while suppressing background and redundant information. Furthermore, the serial operation of SPPF and the branching design of C2PSA ensure that the introduction of this module does not significantly increase the model's computational load. The EDFFN_SEE module is added to the decoder to address the problem that feature extraction and reconstruction can lead to blurred edges, a similar overall appearance, insufficient texture detail, and difficulty in accurately distinguishing various targets in the resulting infrared images, thus yielding more refined infrared images.

[0016] 3. This invention incorporates the EDFFN_SEE module into the joint spatial-frequency domain GAN network encoder. This retains information about the frequencies of interest in the feature map while, through edge attention masking, generates infrared images with finer textures, more precise edges, and easier subsequent use. Simultaneously, the EDFFN module performs Fourier operations on the dimensionality-reduced features to reduce computational load. This efficient design ensures that the introduction of this module has minimal impact on the model's computational complexity.

[0017] 4. The discriminator of this invention includes a D1 discriminator, a D2 discriminator, and a D3 discriminator. It introduces a multi-scale discriminator structure, integrates low, medium, and high spatial resolution discriminator information, improves discriminator capability, optimizes the game between the discriminator and the generator, guides the generation of high-quality infrared images, and improves generation quality and training stability.

[0018] 5. The total loss in this invention not only considers adversarial loss, but also L1 loss, perceptual loss and frequency loss, comprehensively improving the visual and semantic quality of the generated image.

[0019] In summary, the method of this invention is simple in steps and reasonable in design. It uses a spatial frequency domain GAN network to generate infrared images from visible light images, preserves frequency domain features and strengthens edge constraints. Furthermore, it utilizes a multi-scale discriminator and a composite loss function to optimize adversarial training, improve the quality of infrared image generation, and solve the problem of balancing local details with overall consistency.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention.

[0022] Figure 2 This is a schematic diagram of the generator of the present invention.

[0023] Figure 3 This is a schematic diagram of the structure of the EDFFN_SEE module of the present invention.

[0024] Figure 4 This is a schematic diagram of the discriminator of the present invention.

[0025] Figure 5 This is the visualized detection result of the present invention. Detailed Implementation

[0026] like Figures 1 to 4 As shown, the infrared image generation method of the present invention, which combines spatial and frequency domains, includes the following steps: Step 1: Acquiring paired visible light and infrared images: Select pairs of visible light and infrared images from the public dataset and use these pairs as the training set; wherein, one visible light image and one infrared image are used as a set of training data. Step 2: Construct a joint spatial-frequency domain GAN network; wherein the joint spatial-frequency domain GAN network includes a generator and a discriminator, the generator includes an encoder and a decoder, the encoder includes an SPPF_PSA module, and the decoder includes an EDFFN_SEE module; the discriminator includes a first average pooling layer, a second average pooling layer, a D1 discriminator, a D2 discriminator, and a D3 discriminator; Step 3: Input the training set into the joint spatial frequency domain GAN network for training to obtain the trained joint spatial frequency domain GAN network. Step 4: Generate infrared images based on the trained joint spatial frequency domain GAN network.

[0027] In this embodiment, the encoder includes a Conv+LeakyReLU layer, a first downsampling layer, a second downsampling layer, a third downsampling layer, and a Conv+ReLU layer; the decoder includes a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a Tconv+Tanh layer; a first SPPF_PSA module is added between the second downsampling layer and the third downsampling layer; and the output of the Conv+ReLU layer is added to a second SPPF_PSA module. A first EDFFN_SEE module is added between the third upsampling layer and the fourth upsampling layer, and a second EDFFN_SEE module is added between the fourth upsampling layer and the Tconv+Tanh layer. The output of the Conv+LeakyReLU layer and the output of the second EDFFN_SEE module are concatenated in the channel dimension and then input into the Tconv+Tanh layer. The output of the first downsampling layer and the output of the first EDFFN_SEE module are concatenated in the channel dimension and then input into the fourth upsampling layer. The output of the first SPPF_PSA module and the output of the second upsampling layer are concatenated in the channel dimension and then input into the third upsampling layer. The output of the third downsampling layer and the output of the first upsampling layer are concatenated in the channel dimension and then input into the second upsampling layer.

[0028] In this embodiment, step three is as follows: Step 301: The visible light image is input into the encoder for processing using a computer to obtain the first to sixth feature maps; Step 302: Input the first feature map to the sixth feature map into the decoder for processing to obtain the output feature map, and record it as the infrared generated image; Step 303: Stitch the visible light image and the corresponding infrared generated image along the channel dimension to obtain the stitched feature map; Step 304: Process the spliced ​​feature map through the first average pooling layer to obtain the first dimensionality-reduced feature map; process the first dimensionality-reduced feature map through the second average pooling layer to obtain the second dimensionality-reduced feature map. Step 305: Process the spliced ​​feature map, the first dimensionality reduction feature map, and the second dimensionality reduction feature map through the D1 discriminator, the D2 discriminator, and the D3 discriminator, respectively, to obtain the first discriminant feature map, the second discriminant feature map, and the third discriminant feature map; Step 306, according to The adversarial loss of the discriminator is obtained. ;in, Representing visible light images and true infrared images The value of the i-th pixel in the first discriminative feature map output by the D1 discriminator. Representing visible light images and infrared generated images The i-th pixel value in the first discriminant feature map output by the D1 discriminator, where i and m1 are positive integers; 1≤i≤m1, and m1 is the total number of pixels in the first discriminant feature map; Representing visible light images and true infrared images The j-th pixel value in the second discriminative feature map output by the D2 discriminator. Representing visible light images and infrared generated images The j-th pixel value in the second discriminant feature map output by the D2 discriminator, where j and m2 are positive integers; 1≤j≤m2, and m2 is the total number of pixels in the second discriminant feature map; Representing visible light images and true infrared images The k-th pixel value in the third discriminative feature map output by the D3 discriminator Representing visible light images and infrared generated images The value of the kth pixel in the third discriminant feature map output by the D3 discriminator, where k and m3 are positive integers; 1≤k≤m3, and m3 is the total number of pixels in the third discriminant feature map; Step 307, according to The perceptual loss function is obtained. ;in, It is a positive integer. ,and Represents true infrared images ReLU in the VGG19 network model Feature map output from layer 1 Indicates infrared generated image ReLU in the VGG19 network model Feature map output from layer 1 ; , and Relu The number of channels, length, and width of the feature map output from layer 1; ReLU in the VGG19 network model Manhattan distance between the two feature maps output by layer 1; according to To obtain frequency loss Where Fq(·) represents the q-th pixel value in the feature map output by performing a two-dimensional fast Fourier transform on the spatial dimension of the image, Re[·] and Im[·] represent taking the real and imaginary parts of the complex number of the pixel value, respectively; q and Q are positive integers, and Q is the total number of pixels in the feature map output by the two-dimensional fast Fourier transform. Step 308, according to The total loss function is obtained. ;in, and These represent the first weight, second weight, third weight, and fourth weight, respectively. This represents the L1 loss between the real infrared image and the generated infrared image; Step 309: The computer uses the Adam optimization method. It inputs a set of training data and first utilizes generative adversarial loss. The D1, D2, and D3 discriminators are trained until the adversarial loss is obtained. Minimum-level D1 discriminator, D2 discriminator, and D3 discriminator; Step 30A: The computer uses the Adam optimization method, inputs the training data, and, under the D1 discriminator, D2 discriminator, and D3 discriminator determined in step 309, uses the total loss function. Train the network's generator until the total loss function is reached. Minimum, complete training on this set of training data; Step 30B: Follow the methods from Step 309 to Step 30A until the entire training set is trained, completing one iteration of training; Step 30C, repeat steps 309 to 30B, iterate training until the preset number of rounds is met, and obtain the trained joint spatial frequency domain GAN network.

[0029] In this embodiment, a computer is used to input the visible light image into an encoder for processing, obtaining the first to sixth feature maps. The specific process is as follows: The visible light image is input into a Conv+LeakyReLU layer for processing using a computer to obtain the first feature map; The first feature map is processed through the first downsampling layer to obtain the second feature map; The second feature map is processed through the second downsampling layer to obtain the third feature map; The third feature map is processed by the first SPPF_PSA module to obtain the fourth feature map; The fourth feature map is processed through the third downsampling layer to obtain the fifth feature map; The fifth feature map is processed by the Conv+ReLU layer and then by the second SPPF_PSA module to obtain the sixth feature map.

[0030] In this embodiment, step 302 involves inputting the first to sixth feature maps into the decoder for processing to obtain the output feature map. The specific process is as follows: The sixth feature map is processed through the first upsampling layer to obtain the first upsampling feature map; The first upsampled feature map and the fifth feature map are concatenated and then input into the second upsampled layer for processing to obtain the second upsampled feature map. The second upsampled feature map and the fourth feature map are concatenated and then input into the third upsampled layer for processing to obtain the third upsampled feature map. The third upsampled feature map is processed by the first EDFFN_SEE module to obtain the fourth upsampled feature map; The fourth upsampled feature map and the second feature map are concatenated and then input into the fourth upsampled layer for processing to obtain the fifth upsampled feature map. The fifth upsampled feature map is processed by the second EDFFN_SEE module to obtain the sixth upsampled feature map; The sixth upsampled feature map and the first feature map are concatenated and then fed into the Tconv+Tanh layer for processing to obtain the output feature map.

[0031] In this embodiment, both the first SPPF_PSA module and the second SPPF_PSA module include an SPPF module and a C2PSA module. The C2PSA module has three PSABlock branches. The output of the SPPF module is input to the C2PSA module, and the output of the C2PSA module is the output of the first SPPF_PSA module and the second SPPF_PSA module. Both the first EDFFN_SEE module and the second EDFFN_SEE module include an EDFFN module, a Scharr edge extraction operator, batch normalization, a Gelu activation function layer, and an element-wise multiplication module.

[0032] In this embodiment, the methods for obtaining the fourth upsampled feature map by processing the third upsampled feature map through the first EDFFN_SEE module and obtaining the sixth upsampled feature map by processing the fifth upsampled feature map through the second EDFFN_SEE module are the same, and the specific process is as follows: Step A: Denote the third and fifth upsampled feature maps as input feature maps; denote the fourth and sixth upsampled feature maps as enhanced feature maps; Step B: Process the input feature map through the EDFFN module to obtain the first intermediate feature map; Step C: Fuse the input feature map and the first intermediate feature map by element-wise addition to obtain the fused feature map; Step D: Use the Scharr edge extraction operator to extract edges from the fused feature map to obtain the edge feature map; Step E: Batch normalize the edge feature map to obtain the normalized edge feature map; Step F: Pass the normalized edge feature map through a Gelu activation function layer to obtain a sparse feature map; Step G: The sparse feature map is used as an edge attention mask and combined with the input feature map through an element-wise multiplication module to perform a Hadamard product operation, resulting in an enhanced feature map.

[0033] In this embodiment, the publicly available datasets used for the experiment are specifically the following four datasets: The KAIST dataset contains 95,328 aligned visible light and infrared image pairs for daytime and nighttime scenarios in three scenes: campus, road, and downtown. In this application, daytime data was selected for experiments. To avoid information redundancy, one frame was selected every 6 frames in the dataset, totaling 5,566 image pairs, as the training set; and one frame was selected every 20 frames, totaling 1,458 image pairs, as the test set. The FLIR dataset contains misaligned visible light and infrared image pairs from different scenes on urban roads and highways. This application selects aligned image pairs from the paper "Multispectral Fusion for Object Detection with CyclicFuse-and-Refine Blocks," removing data with black borders, and uses 3220 image pairs as the training set and 806 image pairs as the test set. The DayDrone dataset contains 7660 pairs of visible and infrared images of different scenes at various heights and angles. 5745 image pairs were selected as the training set, and 1915 image pairs were selected as the test set. The AVIID-1 dataset consists of 993 pairs of images taken in the morning at a drone height of 15 meters. Of these, 744 pairs were used as the training set and 249 pairs were used as the test set.

[0034] In this embodiment, SPPF stands for Spatial Pyramid Pooling Fast, so the SPPF module is a fast spatial pyramid pooling module.

[0035] C2PSA stands for Cross-Channel Parallel Spatial Attention, and the C2PSA module is a channel-parallel spatial attention mechanism module.

[0036] In this embodiment, it should be noted that the encoder mainly uses multi-layer convolution operations to extract spatial features from the rich details in the visible light image, obtaining a high-level feature vector rich in semantic information. The visible light image to be converted typically contains both large scenes such as trees and buildings, and small targets such as pedestrians and vehicles; conventional methods struggle to balance targets of different sizes. To better capture these features, SPPF and C2PSA modules are introduced into the encoder.

[0037] In this embodiment, the SPPF module and the C2PSA module are combined into the SPPF_PSA module. This module represents targets of different scales in visible light images and extracts key information from complex scenes while suppressing background and redundant information. Furthermore, the serial operation of SPPF and the branching design of C2PSA ensure that the introduction of this module does not significantly increase the model's computational load. Adding the SPPF_PSA module after the third and fifth layers of the encoder allows for better representation of the original visible light image in scenes with complex backgrounds (such as complex street scenes) and coexistence of targets of varying sizes (such as buses, cars, pedestrians, and buildings).

[0038] In this embodiment, it should be noted that the decoder mainly uses multi-layer transposed convolution operations to progressively upsample and feature map the high-level feature vectors extracted by the encoder, ultimately converting them into an image that conforms to infrared radiation characteristics. The numerous convolution operations during the encoding and decoding process, while performing feature extraction and reconstruction, result in blurred edges in the converted infrared image. While the overall image may appear similar, the texture detail is insufficient, making it difficult to accurately distinguish various targets. To obtain a more refined infrared image, we introduce the EDFFN_SEE module into the decoder.

[0039] In this embodiment, it should be noted that EDFFN stands for Efficient Discriminative Frequency Domain-based Feedforward Network, and the EDFFN module is an efficient discriminative frequency domain feedforward network.

[0040] In this embodiment, the x-direction gradient kernel of the Scharr edge extraction operator is used. and y-direction gradient kernel They are respectively Furthermore, the gradient magnitude is calculated using Euclidean distance.

[0041] In this embodiment, the EDFFN_SEE module is added. On the one hand, it retains the information of the frequencies of interest in the feature map; on the other hand, it generates infrared images with finer textures, more accurate edges, and easier subsequent use after being constrained by edge attention masks. At the same time, EDFFN performs Fourier operations on the dimensionality-reduced features to reduce the amount of computation. This efficient design ensures that the introduction of this module has little impact on the computational cost of the model.

[0042] In this embodiment, the input feature map and the first intermediate feature map are fused by element-wise addition. While preserving the original features, the frequency domain features of interest are also preserved, such as low-frequency information like the sky with small thermal radiation differences in infrared images, and high-frequency information like boundary textures with large temperature abrupt changes.

[0043] In this embodiment, the D1 discriminator, D2 discriminator, and D3 discriminator have the same structure and all use the PatchGAN discriminator, i.e., the Markov discriminator.

[0044] In this embodiment, it should be noted that, Figure 2 Down indicates downsampling, and Up indicates upsampling. Figure 4 In this context, AvgPool2d represents the average pooling layer.

[0045] In this embodiment, the size of both the visible light image and the infrared image in the training set is 3×256×256. The size of the first feature map is 64×128×128, the size of the second feature map is 128×64×64, the size of the third feature map is 256×32×32, the size of the fourth feature map is 256×32×32, the size of the fifth feature map is 512×16×16, and the size of the sixth feature map is 512×8×8. The size of the first upsampled feature map is 512×16×16. The size of the concatenated first and fifth upsampled feature maps is 1024×16×16. The size of the second upsampled feature map is 256×32×32. The size of the concatenated second and fourth upsampled feature maps is 512×32×32. The size of the third upsampled feature map is 128×64×64. The size of the fourth upsampled feature map is 128×64×64. The size of the fourth upsampled feature map, concatenated with the second feature map, is 256×64×64. The size of the fifth upsampled feature map is 64×128×128, and the size of the sixth upsampled feature map is also 64×128×128. The size of the concatenated sixth upsampled feature map and the first feature map is 128×128×128, and the size of the output feature map is 3×256×256.

[0046] In this embodiment, the size of the convolution kernel in the Conv+LeakyReLU layer is 4×4, the number of convolution kernels is 64, the stride is 2, and the padding is 1. The first, second, and third downsampling layers are all CBL modules, with internal convolutional kernels of size 4×4, stride 2, padding 1, and different numbers of convolutional kernels: 128 in the first downsampling layer, 256 in the second downsampling layer, and 512 in the third downsampling layer. The Conv+ReLU layer has a kernel size of 4×4, a kernel count of 512, a stride of 2, and a padding of 1. The first, second, third, and fourth upsampling layers are all composed of three activation functions: TConv, BN, and ReLU. TConv is a transposed convolution that performs upsampling to enlarge the image. The number of convolution kernels in the transposed convolutions of the three upsampling layers is different. The kernel size is 4×4, the stride is 2, and the padding is 1. The kernel size in the first upsampling layer is 512, the kernel size in the second upsampling layer is 256, the kernel size in the third upsampling layer is 128, and the kernel size in the fourth upsampling layer is 64. In the Tconv+Tanh layer, the kernel size is 4×4, the number of kernels is 3, the stride is 2, and the padding is 1. The pooling window size of the first average pooling layer and the second average pooling layer is 3×3, the stride is 2, and the padding is 1. like Figure 4 As shown in this embodiment, in specific implementation, the D1 discriminator, D2 discriminator, and D3 discriminator all include an input convolutional layer Conv1, a LeakyReLU activation function, a first CBL module, a second CBL module, a third CBL module, and an output convolutional layer Conv2; wherein, the CBL module is composed of Conv+BN+LeakyReLU activation function; in the first CBL module, the size of the convolutional kernel is 4×4, the number of convolutional kernels is 128, the stride is 2, and the padding is 1; in the second CBL module, the size of the convolutional kernel is 4×4, the number of convolutional kernels is 256, the stride is 2, and the padding is 1; in the third CBL module, the size of the convolutional kernel is 4×4, the number of convolutional kernels is 512, the stride is 1, and the padding is 1; The input convolutional layer has a kernel size of 4×4, a kernel count of 64, a stride of 2, and padding of 1; the output convolutional layer has a kernel size of 4×4, a kernel count of 1, a stride of 1, and padding of 1.

[0047] In this embodiment, when the 6×256×256 feature map passes through the D1 discriminator, the output feature map after the input convolutional layer is 64×128×128. After passing through the first CBL module, the output feature map is 128×64×64. The 128×64×64 feature map passes through the second CBL module and the output feature map is 256×32×32. The 256×32×32 feature map from the second CBL module passes through the third CBL module and the output feature map is 512×31×31. After passing through the third CBL module, the output feature map is 512×31×31. Finally, the discriminative feature map is output after passing through the output convolutional layer. Figure 1 ×30×30; Similarly, when the first dimensionality-reduced feature map (6×128×128) passes through the D2 discriminator, the output discriminative features are... Figure 1×14×14; Similarly, when the second dimensionality-reduced feature map (6×64×64) passes through the D3 discriminator, the output discriminative features are... Figure 1 ×6×6.

[0048] In this embodiment, the preset number of rounds is 200.

[0049] In this embodiment, during specific implementation, the first weight Second weight Third weight and the fourth weight The values ​​are 1, 100, 1 and 1.

[0050] In this embodiment, to comprehensively evaluate the model performance of the cross-modal conversion task from visible light images to infrared images, we selected five complementary metrics. Specifically, we used the pixel error metrics MSE (Mean Squared Error) and PSNR (Peak Signal-to-Noise Ratio) to quantify the pixel-level numerical differences between the generated infrared images and the real infrared images; we used the structural similarity index SSIM (Structural Similarity Index) to quantify the structural differences between the two; we used LPIPS (Learned Perceptual Image Patch Similarity) to evaluate the similarity at the subjective visual perception level; and we used FID (Fréchet Inception Distance) to verify the distribution consistency between the set of generated infrared images and the set of real infrared images.

[0051] In this embodiment, experiments were conducted on the four datasets mentioned above, and four advanced methods were selected for comparative analysis. These included three GAN-based methods—Pix2Pix, CycleGAN, and DR-AVIT—and the PID method, a diffusion model-based method. The PID method was tested on the KAIST and FLIR datasets using the original parameters from the paper, while the other methods were retrained using the original settings from the paper (and the PID method was also retrained on the other two datasets using the original settings from the paper). During training, images were first scaled to 286×286, then randomly cropped to 256×256, and trained for 200 epochs to ensure model convergence.

[0052] In this embodiment, the method significantly outperforms the other four methods in terms of PSNR and MSE. The PSNR results for the four datasets are 22.12, 22.83, 15.99, and 22.83, respectively, and the MSE results are 490.94, 395.22, 1946.5, and 545.16, respectively. This indicates that the method of this invention performs best in terms of pixel-level numerical difference and pixel error. The other methods, including CycleGAN, DR-AVIT, and PID, only achieve optimal performance in a few specific metrics.

[0053] To verify the applicability of the infrared images generated by the method of this invention in downstream tasks, the YOLOv11 object detection method was selected, and experiments were conducted on the AVIID-1 and DayDrone datasets. Object detection models were trained using the original infrared images from both datasets and infrared images generated by five conversion methods: Pix2Pix (Image-to-image translation with conditional adversarial networks), CycleGAN (Unpaired image-to-image translation using cycle-consistent adversarial networks), DR-AVIT (Toward diverse and realistic aerial visible-to-infrared image translation), PID (Physics-informed diffusion model for infrared image generation), and the method of this invention (SF-GAN). After 200 training rounds, the model with the best evaluation results was used to conduct object detection experiments on infrared images. During the detection process, AVIID-1 only tested the motor vehicle category, and DayDrone only tested the car category. The target detection results for each method were compared with the original infrared images labeled with the ground truth. Precision, AP_50, and F1-Score were used for specific evaluation, resulting in the detailed parameter parameters shown in Table 1 below. Figure 5 The visualization of the test results.

[0054] Table 1 Specific Indicator Parameters

[0055] As shown in Table 1, the test results on both datasets consistently demonstrate that the model trained using the method of this invention achieves the highest detection accuracy. The same conclusion is also reached through the visualization results in Figure 6. This indicates that using the method of this invention to convert visible light images into infrared images to expand the infrared dataset and then train the infrared target detection model is feasible and effective.

[0056] In summary, the method of this invention is simple in steps and reasonable in design. It uses a spatial frequency domain GAN network to generate infrared images from visible light images, preserves frequency domain features and strengthens edge constraints. Furthermore, it utilizes a multi-scale discriminator and a composite loss function to optimize adversarial training, improve the quality of infrared image generation, and solve the problem of balancing local details with overall consistency.

[0057] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for generating infrared images in the joint spatial frequency domain, characterized in that, The method includes the following steps: Step 1: Acquiring paired visible light and infrared images: Select pairs of visible light and infrared images from the public dataset and use these pairs as the training set; wherein, one visible light image and one infrared image are used as a set of training data. Step 2: Construct a joint spatial-frequency domain GAN network; wherein the joint spatial-frequency domain GAN network includes a generator and a discriminator, the generator includes an encoder and a decoder, the encoder includes an SPPF_PSA module, and the decoder includes an EDFFN_SEE module; the discriminator includes a first average pooling layer, a second average pooling layer, a D1 discriminator, a D2 discriminator, and a D3 discriminator; Step 3: Input the training set into the joint spatial frequency domain GAN network for training to obtain the trained joint spatial frequency domain GAN network. Step 4: Generate infrared images based on the trained joint spatial frequency domain GAN network.

2. The infrared image generation method in the joint spatial frequency domain according to claim 1, characterized in that: The encoder includes a Conv+LeakyReLU layer, a first downsampling layer, a second downsampling layer, a third downsampling layer, and a Conv+ReLU layer. The decoder includes a first upsampling layer, a second upsampling layer, a third upsampling layer, a fourth upsampling layer, and a Tconv+Tanh layer. A first SPPF_PSA module is added between the second downsampling layer and the third downsampling layer. The output of the Conv+ReLU layer is added to a second SPPF_PSA module. A first EDFFN_SEE module is added between the third upsampling layer and the fourth upsampling layer, and a second EDFFN_SEE module is added between the fourth upsampling layer and the Tconv+Tanh layer. The output of the Conv+LeakyReLU layer and the output of the second EDFFN_SEE module are concatenated in the channel dimension and then input into the Tconv+Tanh layer. The output of the first downsampling layer and the output of the first EDFFN_SEE module are concatenated in the channel dimension and then input into the fourth upsampling layer. The output of the first SPPF_PSA module and the output of the second upsampling layer are concatenated in the channel dimension and then input into the third upsampling layer. The output of the third downsampling layer and the output of the first upsampling layer are concatenated in the channel dimension and then input into the second upsampling layer.

3. The infrared image generation method in the joint spatial frequency domain according to claim 2, characterized in that: Step three, the specific process is as follows: Step 301: The visible light image is input into the encoder for processing using a computer to obtain the first to sixth feature maps; Step 302: Input the first feature map to the sixth feature map into the decoder for processing to obtain the output feature map, and record it as the infrared generated image; Step 303: Stitch the visible light image and the corresponding infrared generated image along the channel dimension to obtain the stitched feature map; Step 304: Process the spliced ​​feature map through the first average pooling layer to obtain the first dimensionality-reduced feature map; process the first dimensionality-reduced feature map through the second average pooling layer to obtain the second dimensionality-reduced feature map. Step 305: Process the spliced ​​feature map, the first dimensionality reduction feature map, and the second dimensionality reduction feature map through the D1 discriminator, the D2 discriminator, and the D3 discriminator, respectively, to obtain the first discriminant feature map, the second discriminant feature map, and the third discriminant feature map; Step 306, according to The adversarial loss of the discriminator is obtained. ;in, Representing visible light images and true infrared images The value of the i-th pixel in the first discriminative feature map output by the D1 discriminator. Representing visible light images and infrared generated images The i-th pixel value in the first discriminant feature map output by the D1 discriminator, where i and m1 are positive integers; 1≤i≤m1, and m1 is the total number of pixels in the first discriminant feature map; Representing visible light images and true infrared images The j-th pixel value in the second discriminative feature map output by the D2 discriminator. Representing visible light images and infrared generated images The j-th pixel value in the second discriminant feature map output by the D2 discriminator, where j and m2 are positive integers; 1≤j≤m2, and m2 is the total number of pixels in the second discriminant feature map; Representing visible light images and true infrared images The k-th pixel value in the third discriminative feature map output by the D3 discriminator Representing visible light images and infrared generated images The value of the kth pixel in the third discriminant feature map output by the D3 discriminator, where k and m3 are positive integers; 1≤k≤m3, and m3 is the total number of pixels in the third discriminant feature map; Step 307, according to The perceptual loss function is obtained. ;in, It is a positive integer. ,and Represents true infrared images ReLU in the VGG19 network model Feature map output from layer 1 Indicates infrared generated image ReLU in the VGG19 network model Feature map output from layer 1 ; , and Relu The number of channels, length, and width of the feature map output from layer 1; ReLU in the VGG19 network model Manhattan distance between the two feature maps output by layer 1; according to To obtain frequency loss Where Fq(·) represents the q-th pixel value in the feature map output by performing a two-dimensional fast Fourier transform on the spatial dimension of the image, Re[·] and Im[·] represent taking the real and imaginary parts of the complex number of the pixel value, respectively; q and Q are positive integers, and Q is the total number of pixels in the feature map output by the two-dimensional fast Fourier transform. Step 308, according to The total loss function is obtained. ;in, and These represent the first weight, second weight, third weight, and fourth weight, respectively. This represents the L1 loss between the real infrared image and the generated infrared image; Step 309: The computer uses the Adam optimization method. It inputs a set of training data and first utilizes generative adversarial loss. The D1, D2, and D3 discriminators are trained until the adversarial loss is obtained. Minimum-level D1 discriminator, D2 discriminator, and D3 discriminator; Step 30A: The computer uses the Adam optimization method, inputs the training data, and, under the D1 discriminator, D2 discriminator, and D3 discriminator determined in step 309, uses the total loss function. Train the network's generator until the total loss function is reached. Minimum, complete training on this set of training data; Step 30B: Follow the methods from Step 309 to Step 30A until the entire training set is trained, completing one iteration of training; Step 30C, repeat steps 309 to 30B, iterate training until the preset number of rounds is met, and obtain the trained joint spatial frequency domain GAN network.

4. The infrared image generation method in the joint spatial frequency domain according to claim 3, characterized in that: Step 301: The visible light image is input into the encoder for processing using a computer to obtain the first to sixth feature maps. The specific process is as follows: The visible light image is input into a Conv+LeakyReLU layer for processing using a computer to obtain the first feature map; The first feature map is processed by the first downsampling layer to obtain the second feature map; The second feature map is processed through the second downsampling layer to obtain the third feature map; The third feature map is processed by the first SPPF_PSA module to obtain the fourth feature map; The fourth feature map is processed through the third downsampling layer to obtain the fifth feature map; The fifth feature map is processed through the Conv+ReLU layer and then through the second SPPF_PSA module to obtain the sixth feature map.

5. The infrared image generation method in the joint spatial frequency domain according to claim 4, characterized in that: Step 302: Input the first to sixth feature maps into the decoder for processing to obtain the output feature map. The specific process is as follows: The sixth feature map is processed through the first upsampling layer to obtain the first upsampling feature map; The first upsampled feature map and the fifth feature map are concatenated and then input into the second upsampled layer for processing to obtain the second upsampled feature map. The second upsampled feature map and the fourth feature map are concatenated and then input into the third upsampled layer for processing to obtain the third upsampled feature map. The third upsampled feature map is processed by the first EDFFN_SEE module to obtain the fourth upsampled feature map; The fourth upsampled feature map and the second feature map are concatenated and then input into the fourth upsampled layer for processing to obtain the fifth upsampled feature map. The fifth upsampled feature map is processed by the second EDFFN_SEE module to obtain the sixth upsampled feature map; The sixth upsampled feature map and the first feature map are concatenated and then fed into the Tconv+Tanh layer for processing to obtain the output feature map.

6. The infrared image generation method in the joint spatial frequency domain according to claim 5, characterized in that: Both the first SPPF_PSA module and the second SPPF_PSA module include an SPPF module and a C2PSA module. The C2PSA module has three PSABlock branches. The output of the SPPF module is input to the C2PSA module, and the output of the C2PSA module is the output of the first SPPF_PSA module and the second SPPF_PSA module. Both the first EDFFN_SEE module and the second EDFFN_SEE module include an EDFFN module, a Scharr edge extraction operator, batch normalization, a Gelu activation function layer, and an element-wise multiplication module.

7. The infrared image generation method in the joint spatial frequency domain according to claim 6, characterized in that: The method for obtaining the fourth upsampled feature map by processing the third upsampled feature map through the first EDFFN_SEE module, and the method for obtaining the sixth upsampled feature map by processing the fifth upsampled feature map through the second EDFFN_SEE module, are all the same, and the specific process is as follows: Step A: Denote the third and fifth upsampled feature maps as input feature maps; denote the fourth and sixth upsampled feature maps as enhanced feature maps; Step B: Process the input feature map through the EDFFN module to obtain the first intermediate feature map; Step C: Fuse the input feature map and the first intermediate feature map by element-wise addition to obtain the fused feature map; Step D: Use the Scharr edge extraction operator to extract edges from the fused feature map to obtain the edge feature map; Step E: Batch normalize the edge feature map to obtain the normalized edge feature map; Step F: Pass the normalized edge feature map through a Gelu activation function layer to obtain a sparse feature map; Step G: The sparse feature map is used as an edge attention mask and combined with the input feature map through an element-wise multiplication module to perform a Hadamard product operation, resulting in an enhanced feature map.

Images