Visible light-to-infrared image migration method based on contrast learning
By using a generative adversarial network model based on contrastive learning, the generator produces high-quality infrared images, solving the problem of the lack of paired visible light and infrared image data, expanding the infrared dataset, and improving target detection and recognition performance.
Patent Information
- Application Number
- CN202511066162.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-12-12
AI Technical Summary
Existing technologies struggle to acquire clear and matching visible light and infrared images simultaneously under limited environmental conditions, resulting in a scarcity of infrared datasets and impacting target detection and recognition performance.
A generative adversarial network model based on contrastive learning is adopted, which includes a generator, a discriminator, and a feature patch extractor. Infrared images are generated by the contrastive learning generator. The quality of the generated images is measured by the PatchNCE loss function, SSIM, and PSNR. The loss functions of the generator and discriminator are optimized to achieve the transfer of visible light to infrared images.
It has achieved the augmentation and expansion of infrared datasets, providing data support for target detection and recognition. The generated infrared images are of high quality, which can improve the performance of target detection and recognition.
Smart Images
Figure CN121120405A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image generation, specifically to a method for transferring visible light images to infrared images based on contrastive learning. Background Technology
[0002] Visible light images are rich in texture and color information. Infrared images are sensitive to thermal radiation and play a crucial role in low-light, inclement weather, and target detection scenarios. Effectively pairing these two technologies and combining their advantages can support tasks such as target detection, recognition, and tracking.
[0003] Acquiring clear and matching visible light and infrared images simultaneously under limited environmental and hardware conditions is extremely challenging. Most current mainstream visible light to infrared image transfer methods rely on paired visible light and infrared datasets for training, but such paired datasets are scarce. Therefore, generating more high-quality infrared images based on existing visible light and infrared images to augment the corresponding infrared datasets is crucial for improving target detection and recognition performance. Summary of the Invention
[0004] To address the aforementioned technical problems, the present invention aims to propose a visible light to infrared image transfer method based on contrastive learning, which solves the problem of insufficient paired data between visible light and infrared images, and provides a guarantee for ultimately improving the performance of target detection and recognition.
[0005] To achieve the above objectives, the present invention provides a visible light to infrared image transfer method based on contrastive learning, the transfer method comprising the following steps:
[0006] Step 1: Define a generative adversarial network model based on contrastive learning. The network model includes: a generator structure based on ResNet (residual network), a discriminator structure of a three-layer convolutional neural network, and a feature patch extractor.
[0007] Step 2: Randomly initialize the network model parameters in the generator, discriminator, and feature patch extractor;
[0008] Step 3: Define the loss functions for the generator and discriminator;
[0009] Step four: Train the network model. During the training process, the parameters of the network model are updated using the Adam optimization algorithm.
[0010] Step 5: Input the real visible light image into the generator to generate an infrared image, compare it with the corresponding real infrared image, and calculate SSIM (structural similarity) and PSNR (peak signal-to-noise ratio) to measure the quality of the generated image.
[0011] Furthermore, in the generative adversarial network model based on contrastive learning, the generator is used to convert the input real visible light image into an infrared image output; the discriminator's input is either the real visible light image or the infrared image generated by the generator, and its output is a probability value indicating that the input image is real, used to determine the authenticity of the input image; the feature patch extractor includes: a feature reshaping module, a patch sampling module, an MLP and L2 normalization module, and the input of the feature patch extractor is the real image and the generated infrared image, and the output is the corresponding feature patch and its feature representation, used for contrastive learning.
[0012] Furthermore, the main mechanism of contrastive learning is as follows: the feature patch extractor divides the input image into multiple patches, each containing local texture and structural information. Patches with matching positions are defined as positive samples, and patches at other positions are defined as negative samples. By calculating the similarity between each sample and its corresponding positive and negative samples, and optimizing the model based on the PatchNCE loss function, the similarity between the input patch and the positive sample is increased, while the association with the negative sample is reduced.
[0013] Furthermore, the generator's loss function expression is as follows:
[0014]
[0015] Where: L G L represents the total loss of the generator. G_GAN L represents the adversarial loss component of the generator. PatchNCE Let G(x) represent the PatchNCE loss of the generator; X represents the real visible light dataset, x represents a real visible light image randomly sampled from X, and E represents the expected value; G(x) represents the infrared image generated by the generator G, and D() represents the output of the discriminator; [G(x)] i This represents the i-th patch in the generated image; This represents the i-th patch of the real image, with the superscript + indicating its relationship to [G(x)]. i For positive sample patch pairs; Let j represent the j-th patch of the real image, with the superscript - indicating its relationship to [G(x)]. i For negative sample patch pairs; N represents the number of patches in an image; f() is the feature representation of the input patch; sim(,) is the similarity function; and τ is a temperature parameter used to adjust the smoothness of the similarity.
[0016] Furthermore, the loss function expression for the discriminator is as follows:
[0017]
[0018] Where: L D L represents the total loss of the discriminator. D_realL represents the loss function part that determines whether a sample is a true sample. D_fake This represents the loss function part that determines whether a generated sample is fake; Y represents the real infrared dataset, and y represents a real infrared image randomly sampled from Y.
[0019] Furthermore, the expression for SSIM is as follows:
[0020]
[0021] Where: a and b represent the two images used for evaluation, μ a μ represents the average value of a. b σ represents the average value of b. a σ is the variance of a. b σ is the variance of b. ab c1 and c2 are the covariances of a and b; c1 and c2 are constants used to maintain stability.
[0022] Furthermore, the expression for PSNR is as follows:
[0023]
[0024] Where: n=8 represents the number of binary bits for each pixel in the image, and MSE is the mean square error between the real image and the generated image.
[0025] Beneficial effects:
[0026] By generating simulated infrared images using the aforementioned generative adversarial network based on contrastive learning, the infrared dataset is augmented and expanded, providing data support for target detection and recognition. Attached Figure Description
[0027] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0028] Figure 1 This is a general framework diagram for achieving visible light to infrared domain transfer using generative adversarial networks based on contrastive learning.
[0029] Figure 2 This is a schematic diagram of contrastive learning using PatchNCE loss;
[0030] Figure 3 The figure shows the experimental results of Example 1;
[0031] Figure 4 The figure shows the experimental results of Example 2. Detailed Implementation
[0032] This invention provides a generative adversarial network model based on contrastive learning, which can achieve image transfer in both paired and unpaired scenarios. The invention is illustrated in more detail below with examples.
[0033] This application provides a visible light to infrared image transfer method based on contrastive learning. The method is based on a generative adversarial network (GAN) model based on contrastive learning, which includes a generator, a discriminator, and a feature patch extractor. This network is applicable to paired and unpaired visible light and infrared datasets.
[0034] The implementation of the above model, as described in this application, specifically includes the following steps:
[0035] Step 1: Define the network structure
[0036] A ResNet-based generator architecture is employed, with nine residual blocks connected after the encoder. A three-layer convolutional neural network serves as the discriminator. The feature patch extractor includes a feature reshaping module, a patch sampling module, an MLP (Multilayer Perceptron), and an L2 normalization module.
[0037] Step 2: Initialize network parameters
[0038] The network weights and other parameters in the generator, discriminator, and feature patch extractor are randomly initialized.
[0039] Step 3: Define the loss function
[0040] 1. Generator Loss: This includes adversarial loss and PatchNCE loss (local contrastive loss). Minimizing the former makes the generator produce fake data that is difficult for the discriminator to distinguish from real data. Minimizing the latter allows the local feature distribution of the generated samples to approximate that of the real samples through contrastive learning. The generator loss function is expressed as follows:
[0041]
[0042] Where: L G L represents the total loss of the generator. G_GAN L represents the adversarial loss component of the generator. NCE Let G(x) represent the PatchNCE loss of the generator; X represents the real visible light dataset, x represents a real visible light image randomly sampled from X, and E represents the expected value; G(x) represents the infrared image generated by the generator G, and D() represents the output of the discriminator; [G(x)] i This represents the i-th patch in the generated image; This represents the i-th patch of the real image, with the superscript + indicating its relationship to [G(x)]. i For positive sample patch pairs; Let j represent the j-th patch of the real image, with the superscript - indicating its relationship to [G(x)]. iFor negative sample patch pairs; N represents the number of patches in an image; f() is the feature representation of the input patch; sim(,) is the similarity function; τ is a temperature parameter used to adjust the smoothness of the similarity.
[0043] 2. Discriminator Loss: This consists of two parts: classifying samples as true and classifying samples as false. Minimizing this loss improves the discriminator's ability to distinguish between true and false samples. The discriminator's loss function is expressed as follows:
[0044]
[0045] Where: L D L represents the total loss of the discriminator. D_real L represents the loss function part that determines whether a sample is a true sample. D_fake This represents the loss function part that determines whether a generated sample is fake; Y represents the real infrared dataset, and y represents a real infrared image randomly sampled from Y.
[0046] Step 4: Define the optimization algorithm
[0047] The Adam algorithm is used to update network parameters during training.
[0048] Step 5: Training Process
[0049] With a fixed generator, sample real and generated infrared images and input them into the discriminator. Calculate the discriminator loss and update the discriminator parameters using Adam. With a fixed discriminator, sample real visible light images, generate infrared images using the generator, input them into the discriminator and feature patch extractor, calculate the generator loss, and update the generator parameters using Adam. Alternate between these steps until the desired effect is achieved.
[0050] Step Six: Testing and Evaluation Process
[0051] A batch of real visible light images are input into the generator to generate infrared images. The generated images are then compared with the corresponding real infrared images, and the SSIM (structural similarity) and PSNR (peak signal-to-noise ratio) are calculated to measure the quality of the generated images.
[0052] SSIM (Simplified Simulation) measures the overall similarity of images by evaluating three features: brightness, contrast, and structure. The closer the value is to 1, the higher the similarity between the two images. With all three features having a weight of 1, the expression for SSIM is as follows:
[0053]
[0054] Where: a and b represent the two images used for evaluation, μ a μ represents the average value of a. b σ represents the average value of b. a σ is the variance of a. bσ is the variance of b. ab c1 and c2 are the covariances of a and b; c1 and c2 are constants used to maintain stability.
[0055] PSNR: Calculated based on the pixel difference between the real and generated images, measured in dB. PSNR reflects the similarity between the two images at the pixel level; a higher value indicates a higher similarity. The expression for PSNR is as follows:
[0056]
[0057] Where: n=8 represents the number of binary bits for each pixel in the image, and MSE is the mean square error between the real image and the generated image.
[0058] Example 1:
[0059] This invention provides a generative adversarial network model based on contrastive learning, the overall framework of which is as follows: Figure 1 As shown, it includes a generator G, a discriminator D, and a feature patch extractor G. enc A real visible light image is used to generate a fake infrared image. This fake image is then processed by a discriminator, which outputs the probability of it being real or fake. This probabilities are then compared and learned by a feature patch extractor.
[0060] Figure 2 The diagram illustrates contrastive learning. Taking the transformation from a horse to a zebra as an example, the zebra's head should be closely associated with the head of the input horse image, and less associated with other patches such as the horse's legs, back, and grass background. In this case, the head patch of the horse image is a positive sample, while the patches of other parts are negative samples. PatchNCE loss optimizes the model by calculating the similarity between the sample and the positive and negative samples, ensuring that the zebra's head features are closely associated with the head features of the input horse image, while reducing the association with other parts.
[0061] When visible light and infrared paired datasets are available, samples are read using a paired reading method. This example uses the publicly available street view paired dataset LLVIP.
[0062] Based on the steps outlined above, the network structure is defined, with Adam parameters set to β1 = 0.5 and β2 = 0.999, and an initial learning rate of 0.0001. For contrastive learning, real infrared images and generator-generated infrared images are used to calculate the PatchNCE loss, with both the adversarial loss and the PatchNCE loss having a weight of 1. After training, testing is performed, and the results are as follows... Figure 3 As shown, the corresponding SSIM and PSNR metrics are detailed in Table 1.
[0063] Table 1
[0064]
[0065] Example 2:
[0066] This invention provides a generative adversarial network model based on contrastive learning. When only unpaired visible light and infrared datasets are available, samples are read using an unpaired random method.
[0067] This example uses the publicly available VEDAI aerial imagery dataset, which is primarily designed for vehicle detection tasks. All images are collected from civilian scenes and do not involve military airports or equipment. It includes various types of civilian vehicles and a small number of civilian aircraft targets. Although this dataset is a paired dataset of visible light and infrared images, this study did not directly utilize its paired characteristics during model training. Instead, it used a non-paired approach to read the visible light and infrared images separately. Paired readings were only performed during the testing and validation phases to calculate the model's evaluation metrics.
[0068] The network structure and parameter settings are the same as in Example 1, but this example uses real visible light images and infrared images generated by the generator to calculate the PatchNCE loss. The adversarial loss weight is set to 1, and the PatchNCE loss weight is set to 0.15. After training, testing is performed, and the results are as follows. Figure 4 As shown, the corresponding SSIM and PSNR metrics are detailed in Table 2.
[0069] Table 2
[0070]
[0071] Comparison Table 1 Figure 3 The paired image migration effect shown in Table 2 is as follows. Figure 4 The unpaired image transfer results show that the model performs better with paired images. The generated images not only more closely resemble real infrared images overall, but also exhibit more refined and accurate features in fine structures such as edge contours and local textures. Unpaired transfer, lacking direct supervision from paired data, has limitations in detail preservation, with some fine structures appearing blurred and edges unclear. However, it should be noted that this model still possesses a certain degree of effectiveness in unpaired image transfer. It can utilize a large amount of unpaired image data to grasp the characteristics of both modalities at a macroscopic level for transfer, achieving an initial expansion of the infrared dataset.
Claims
1. A visible light to infrared image transfer method based on contrastive learning, characterized in that, The migration method includes the following steps: Step 1: Define a generative adversarial network model based on contrastive learning. The network model includes: a generator structure based on ResNet, a discriminator structure of a three-layer convolutional neural network, and a feature patch extractor. Step 2: Randomly initialize the network model parameters in the generator, discriminator, and feature patch extractor; Step 3: Define the loss functions for the generator and discriminator; Step 4: Train the network model. During the training process, the parameters of the network model are updated using the Adam algorithm. Step 5: Input the real visible light image into the generator to generate an infrared image, compare it with the corresponding real infrared image, and calculate SSIM and PSNR to measure the quality of the generated image.
2. The visible light to infrared image transfer method based on contrastive learning according to claim 1, characterized in that, In the generative adversarial network model based on contrastive learning, the generator is used to convert the input real visible light image into an infrared image output; the discriminator takes the input of the real visible light image or the infrared image generated by the generator, and outputs a probability value indicating that the input image is real, used to determine the authenticity of the input image; the feature patch extractor includes: a feature reshaping module, a patch sampling module, an MLP and L2 normalization module, the input of the feature patch extractor is the real image and the infrared image generated by the generator, and the output is the corresponding feature patch and its feature representation, used for contrastive learning.
3. The comparative learning according to claim 2, characterized in that, The feature patch extractor divides the input image into multiple patches, each containing local texture and structural information. Patches with matching positions are defined as positive samples, while patches at other positions are defined as negative samples. By calculating the similarity between each sample and its corresponding positive and negative samples, and optimizing the model based on the PatchNCE loss function, the similarity between the input patch and the positive sample is increased, while the association with the negative sample is reduced.
4. The visible light to infrared image transfer method based on contrastive learning according to claim 2, characterized in that, The generator's loss function expression is as follows: Where: L G L represents the total loss of the generator. G_GAN L represents the adversarial loss of the generator. NCE Let represent the PatchNCE loss of the generator; X represents the real visible light dataset, x represents a real visible light image randomly sampled from X, and E represents the expected value; G(x) represents the infrared image generated by the generator G, and D() represents the output of the discriminator; [G(x)] i This represents the i-th patch in the generated image; This represents the i-th patch of the real image, with the superscript + indicating its relationship to [G(x)]. i For positive sample patch pairs; Let j represent the j-th patch of the real image, with the superscript - indicating its relationship to [G(x)]. i For negative sample patch pairs; N represents the number of patches in an image; f() is the feature representation of the input patch; sim(,) is the similarity function; τ is a temperature parameter used to adjust the smoothness of the similarity.
5. The visible light to infrared image transfer method based on contrastive learning according to claim 2, characterized in that, The loss function expression for the discriminator is as follows: Where: L D L represents the total loss of the discriminator. D_real L represents the loss function part that determines whether a sample is a true sample. D_fake This represents the loss function part that determines whether a generated sample is fake; Y represents the real infrared dataset, and y represents a real infrared image randomly sampled from Y.
6. The visible light to infrared image transfer method based on contrastive learning according to claim 1, characterized in that, The expression for SSIM is as follows: Where: a and b represent the two images used for evaluation, μ a μ represents the average value of a. b σ represents the average value of b. a σ is the variance of a. b σ is the variance of b. ab c1 and c2 are the covariances of a and b; c1 and c2 are constants used to maintain stability.
7. The visible light to infrared image transfer method based on contrastive learning according to claim 1, characterized in that, The expression for PSNR is as follows: Where: n represents the number of binary bits for each pixel in the image, and MSE is the mean square error between the real image and the generated image.