Unsupervised infrared-to-visible image conversion method, apparatus and device

Abstract

The application relates to the field of artificial intelligence, and provides a kind of unsupervised infrared image to visible light image conversion method, device and equipment. The method comprises: obtaining an infrared image to be converted;Obtain a visible light image;The infrared image is input into an image generation model to obtain a visible light image output by the image generation model;Wherein, the image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model and a second image discrimination model, the image generation model is trained using a preset loss function, and the preset loss function is calculated based on a gradient loss function of the visible sample image, a total pixel loss function, a standard generative adversarial network loss function and a cycle consistency loss function. The embodiment of the application improves the accuracy of converting infrared images into visible light images.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an unsupervised method, apparatus, and device for converting infrared images to visible light images. Background Technology

[0002] Currently, infrared imaging technology utilizes the principle that all objects above absolute zero emit infrared light. Since the human body and heat-generating objects emit strong infrared light, while other non-heat-generating objects emit very weak red light, specialized infrared cameras can achieve infrared nighttime imaging. Infrared images are less affected by weather and lighting conditions and are primarily used for acquiring, detecting, and locating hot targets and low-light targets. However, their ability to reflect details and scene information is relatively poor. Visible light images contain rich scene information, but they are easily affected by weather and light intensity, especially at night. Images acquired using visible light cameras depend on the amount of illumination around the observed object; insufficient light at night leads to insufficient scene detail when using visible light cameras. Currently, infrared cameras are deployed in many surveillance systems to monitor and record nighttime activities, but the spatial resolution limitations of infrared images make objects difficult to identify. Therefore, converting them to visible RGB images is crucial. This not only makes infrared surveillance systems more powerful at night or in adverse weather conditions but also facilitates biometric identification of the acquired infrared images.

[0003] To improve the visual fidelity of night vision images, existing technologies mainly fall into the following three categories:

[0004] Infrared and Visible Image Fusion Techniques: Over the past few decades, image fusion has played a crucial role in enhancing the details of low-quality images. Numerous fusion techniques have been proposed in the literature to fuse images acquired from different imaging devices, including multi-scale decomposition-based fusion, sparse representation, fuzzy theory, and deep learning. While traditional fusion methods achieve high accuracy, their algorithmic performance is poor; convolutional neural network (CNN) architectures, although showing better performance, increase network complexity and are not accurate in preserving the details of the fused image.

[0005] Infrared image colorization techniques aim to convert the colors of visible RGB images into corresponding grayscale infrared images. These techniques can generally be categorized into parametric and non-parametric methods. Parametric methods estimate pixel chromaticity values ​​using trained models based on deep CNNs or GANs; while non-parametric methods utilize visible RGB images as reference images, which have a similar structure to the source infrared image to transfer colors into them. However, these methods require a training dataset or a color reference image with pixel registration to achieve better accuracy in color infrared images. Therefore, it is often difficult to meet this requirement when acquiring visible images in low-light or nighttime conditions.

[0006] Infrared to visible light image conversion techniques; the conversion of infrared to visible light images is essentially a process of learning a joint data distribution mapping function from an input domain with a given data distribution. Early deep learning methods produced very poor results for visible light images. Recent generative adversarial networks (GANs) have addressed this issue. GANs require training with pairs of images and use a loss function to normalize the generator model, thereby converting infrared images to visible light images. However, this supervised approach is only suitable for paired images, especially when the visible light images in night vision datasets are of poor quality, and this training method often fails to achieve accurate image conversion results.

[0007] In other words, the accuracy of converting infrared images into visible light images in existing technical solutions needs to be improved. Summary of the Invention

[0008] This application provides an unsupervised method, apparatus, and device for converting infrared images to visible light images, thereby improving the accuracy of converting infrared images to visible light images.

[0009] In a first aspect, embodiments of this application provide an unsupervised method for converting infrared images to visible light images, comprising:

[0010] Acquire the infrared image to be converted;

[0011] Acquire a visible light image; input the infrared image into an image generation model to obtain a visible light image output by the image generation model;

[0012] The image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model. The first image discrimination model is used to distinguish between the infrared sample image and the infrared generated image generated based on the infrared sample image, and the second image discrimination model is used to distinguish between the visible light sample image and the visible light generated image generated based on the visible light sample image.

[0013] The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0014] In one embodiment, the preset loss function is calculated based on the product of the gradient loss function of the visible sample image and the first weighting factor, the product of the total pixel loss function and the second weighting factor, the product of the standard generative adversarial network loss function and the third weighting factor, and the product of the cycle consistency loss function and the fourth weighting factor.

[0015] In one embodiment, the preset loss function is expressed by the following formula:

[0016] Preset loss function = λ0l grad +λ1l pix +λ2l GAN +λ3l cyc ;

[0017] Wherein, λ0, λ1, λ2, and λ3 represent the first weighting factor, the second weighting factor, the third weighting factor, and the fourth weighting factor, respectively; grad The gradient loss function represents the visible sample image; pix Represents the total inter-pixel loss function; l GAN This represents the standard generative adversarial network loss function; cyc This represents the cycle consistency loss function.

[0018] In one embodiment, before the step of acquiring the infrared image to be converted, the method further includes: training the image generation model using an image training model;

[0019] The image training model includes:

[0020] The encoding layer is used to extract features from the infrared sample image and the visible light sample image respectively to obtain a first feature map corresponding to the infrared sample image and a second feature map corresponding to the visible light sample image;

[0021] An image conversion layer is used to train the first feature map through a first generative adversarial network to obtain an infrared generated image, and to train the second feature map through a second generative adversarial network to obtain a visible light generated image.

[0022] The loss function calculation layer is used to calculate the preset loss function based on the infrared sample image, the visible light sample image, the infrared generated image, and the visible light generated image, and to train the image generation model using the preset loss function.

[0023] In one embodiment, the image training model further includes:

[0024] The image preprocessing layer is used to preprocess the input infrared sample image using an adaptive histogram equalization method.

[0025] In one embodiment, the coding layer and the image conversion layer each include a residual network, which is formed by parallel connections of multiple convolutional kernels of different scales.

[0026] Secondly, embodiments of this application provide an unsupervised infrared image to visible light image conversion apparatus, comprising:

[0027] The first acquisition module is used to acquire the infrared image to be converted;

[0028] The second acquisition module is used to acquire a visible light image; input the infrared image into an image generation model to obtain a visible light image output by the image generation model;

[0029] The image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model. The first image discrimination model is used to distinguish between the infrared sample image and the infrared generated image generated based on the infrared sample image, and the second image discrimination model is used to distinguish between the visible light sample image and the visible light generated image generated based on the visible light sample image.

[0030] The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0031] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the unsupervised infrared image to visible light image conversion method described in the first aspect.

[0032] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the unsupervised infrared image to visible light image conversion method described in the first aspect.

[0033] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the unsupervised infrared image to visible light image conversion method described in the first aspect.

[0034] The unsupervised infrared-to-visible-light image conversion method, apparatus, and device provided in this application embodiment converts infrared images into visible-light images by inputting the infrared image into an image generation model, thereby obtaining a visible-light image output by the image generation model. The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible-light generated image and the visible-light sample image. Since this application embodiment represents the preset loss function through a weighted combination of the gradient loss function, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function, the gradient loss function eliminates noise effects in the image, thereby ensuring the spatial continuity and smoothness of the generated visible-light image; the total inter-pixel loss function maintains the content consistency between the output visible-light image of the image generation model and the corresponding input infrared image; and the cycle consistency loss function restricts the space of the generated visible-light image while preserving the content information of the original input infrared image. Thus, the embodiments of this application effectively reduce noise in the generated visible light image and improve the overall visual effect of the input infrared image, thereby improving the accuracy of converting the infrared image into a visible light image. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is one of the flowcharts illustrating the unsupervised infrared image to visible light image conversion method provided in this application embodiment;

[0037] Figure 2This is the second schematic flowchart of the unsupervised infrared image to visible light image conversion method provided in the embodiments of this application;

[0038] Figure 3 This is a schematic diagram of the structure of the image training model provided in the embodiments of this application;

[0039] Figure 4 This is a schematic diagram of the structure of discriminator D1 or discriminator D2 provided in the embodiments of this application;

[0040] Figure 5 This is a schematic diagram of the structure of the residual network layer provided in the embodiments of this application;

[0041] Figure 6 This is a schematic diagram of the structure of the unsupervised infrared image to visible light image conversion device provided in the embodiments of this application;

[0042] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0044] Please refer to Figure 1 , Figure 1 This application provides an unsupervised method for converting infrared images to visible light images. The method may include:

[0045] Step 200: Obtain the infrared image to be converted;

[0046] Electronic devices acquire infrared images captured by infrared cameras, based on the principle of infrared nighttime imaging. These infrared images can be various infrared images from different application scenarios within publicly available datasets.

[0047] Step 300: Obtain a visible light image; input the infrared image into the image generation model to obtain the visible light image output by the image generation model;

[0048] The electronic device inputs the infrared image into the image generation model to obtain a visible light image output by the image generation model.

[0049] The image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model. The first image discrimination model is used to distinguish between the infrared sample image and the infrared generated image generated based on the infrared sample image, and the second image discrimination model is used to distinguish between the visible light sample image and the visible light generated image generated based on the visible light sample image.

[0050] Image generation models are generator networks based on generative adversarial networks (GANs). The structure of an image generation model can be found in [reference needed]. Figure 3 Generator G1 or Generator G2. Infrared sample images are infrared images captured by an infrared camera based on the principle of infrared nighttime imaging. Similar to infrared images, infrared sample images can be various infrared images from different application scenarios in a publicly available dataset. Infrared sample images are used for sample training. Visible light sample images are ordinary images that are perceptible to the human eye and captured by a camera. Visible light sample images can be ordinary images perceptible to the human eye and applied in various fields.

[0051] It should be noted that the image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0052] By inputting the infrared image into an image generation model, a visible light image is obtained from the output of the image generation model. The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image. In this embodiment, the preset loss function is represented by a weighted combination of the gradient loss function, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The gradient loss function eliminates noise effects in the image, thereby ensuring the spatial continuity and smoothness of the generated visible light image. The total inter-pixel loss function maintains the content consistency between the output visible light image of the image generation model and the corresponding input infrared image. The cycle consistency loss function limits the space of the generated visible light image and retains the content information of the original input infrared image. Therefore, this embodiment effectively reduces noise in the generated visible light image and improves the overall visual effect of the input infrared image, thereby increasing the accuracy of converting the infrared image into a visible light image.

[0053] For other aspects of this application, please refer to Figure 2 Before step 200, which is to acquire the infrared image to be converted, the method further includes step 100, which is to train the image generation model using an image training model.

[0054] Specifically, please refer to Figure 3 The image training model includes an encoding layer 10, an image transformation layer 20, and a loss function calculation layer. The encoding layer 10 includes two autoencoder modules, namely autoencoder module AE1 and autoencoder module AE2. The image transformation layer 20 includes two generator adversarial networks, namely a first generator adversarial network and a second generator adversarial network. Each generator adversarial network includes a generator and a discriminator module. The autoencoder effectively enhances features in infrared or visible light sample images while reducing noise. The generator adversarial network's generator reconstructs a given input image (i.e., an infrared or visible light sample image) into its original and transformed forms. The discriminator operates similarly to the decoder module, comparing the similarity between the transformed image and the original image (i.e., the infrared or visible light sample image) and attempting to determine whether it was generated by the generator.

[0055] Encoding layer 10 is used to extract features from infrared sample images and visible light sample images respectively to obtain a first feature map corresponding to the infrared sample image and a second feature map corresponding to the visible light sample image. Specifically, the structure of encoding layer 10 is as follows: Figure 1 As shown, the autoencoder module AE1 receives input from the input layer and completes the encoding process through convolutional layers to obtain a first feature map or a second feature map containing image features. Both autoencoder modules AE1 and AE2 consist of two convolutional layers (i.e., ... Figure 1 The convolutional network consists of a Conv layer and three residual network layers. In each convolutional layer, m×n×f represents the number of input channels, the number of output channels, and the size of the convolutional kernel, respectively. The stride of each layer is as follows: Figure 1 As shown, the first layer uses the LReLU function as the activation function, and the second layer uses the PReLU function. The design of these two ReLU functions can not only reduce the computational cost, but also reduce the risk of network overfitting during training.

[0056] Image transformation layer 20 is used to train the first feature map using a first generative adversarial network to obtain an infrared generated image, and to train the second feature map using a second generative adversarial network to obtain a visible light generated image. The first generative adversarial network of image transformation layer 20 includes a generator G1 and a discriminator D1, and the second generative adversarial network includes a generator G2 and a discriminator D2.

[0057] The structure of the generator is as follows Figure 3 The first few layers of generator G1 or generator G2 are responsible for decoding high-level features to reconstruct the input infrared or visible light sample image. Therefore, the weights of the first few layers of the generator are shared in both the infrared and visible light image domains. It should be noted that... Figure 3 In this context, Conv layer represents a convolutional layer, and PreLU represents an activation function layer.

[0058] Compared to sigmoid functions, the hyperbolic tangent function Tanh has the advantage of finding local or global minima more quickly due to its larger derivative range. The hyperbolic tangent function Tanh also reduces the risk of gradient vanishing to some extent. Therefore, generator G1 or generator G2 chooses the Tanh function as the activation function for the network output.

[0059] Please refer to the network structure of discriminator D1 and discriminator D2. Figure 4 Discriminators D1 and D2 both consist of two convolutional layers and three intermediate deep convolutional layers, as shown below. Figure 4 As shown, this network model structure achieves the best performance with the fewest parameters.

[0060] It should be noted that, Figure 4 In this context, Conv represents a convolutional layer, PreLU represents an activation function layer, and DepthwiseConv represents a depthwise convolutional layer.

[0061] The loss function calculation layer is used to calculate the preset loss function based on the infrared sample image, the visible light sample image, the infrared generated image, and the visible light generated image, and to train the image generation model using the preset loss function.

[0062] Furthermore, the preset loss function is calculated based on the product of the gradient loss function of the visible sample image and the first weighting factor, the product of the total pixel loss function and the second weighting factor, the product of the standard generative adversarial network loss function and the third weighting factor, and the product of the cycle consistency loss function and the fourth weighting factor.

[0063] This application employs various loss functions to train the image generation model. By multiplying each loss function by a weighting factor, the final preset loss function yields a superior result, leading to better visible light images generated by the image generation model.

[0064] Specifically, the preset loss function is expressed by the following formula:

[0065] Preset loss function = λ0l grad +λ1l pix +λ2l GAN +λ3l cyc ;

[0066] Wherein, λ0, λ1, λ2, and λ3 represent the first weighting factor, the second weighting factor, the third weighting factor, and the fourth weighting factor, respectively; grad The gradient loss function represents the visible sample image; pix Represents the total inter-pixel loss function; l GAN This represents the standard generative adversarial network loss function; cyc This represents the cycle consistency loss function.

[0067] Gradient loss function λ0l grad : Represents the Euclidean distance between adjacent pixels in an infrared or visible light sample image. This application uses a gradient loss function to eliminate image noise, thereby ensuring the spatial continuity and smoothness of the generated visible light image. The specific formula is as follows:

[0068]

[0069] in, This represents the image generated from the converted visible light.

[0070] Total inter-pixel loss function λ1l pixTo maintain consistency between the output image of generator G1 or generator G2 and the corresponding input image in unsupervised methods, this application embodiment uses the inter-pixel loss function L1 of infrared sample images and visible light sample images to represent the content.

[0071]

[0072] Where T and V represent the input infrared image and visible light image, respectively. and These represent the infrared and visible light generated images, respectively, generated by generators G1 and G2.

[0073] Standard Generative Adversarial Network Loss Function l GAN To improve the visual quality of the generated image and match it with the target image, this method employs the standard generative adversarial network loss function l. GAN In the training of a generative adversarial network (GAN) for converting infrared images to visible light images, the generator G1 or G2 is trained to learn the mapping relationship between the input image and the converted image. For example, in this method, the discriminator network D2 is trained to maximize the objective function to distinguish between the generated visible light image and the visible light image input into the autoencoder AE1. In contrast, the generator network G2 is trained to minimize the loss function so that the generated visible light image looks closer to the real input image (i.e., the generated visible light image). Therefore, the total generator loss function l GAN It can be represented by the loss functions of two generators:

[0074]

[0075] in and Let G1 and G2 represent the standard generative adversarial network loss functions, respectively.

[0076]

[0077]

[0078] Where q1 and q2 represent the output feature maps generated by the infrared image autoencoder AE1 and the visible light image autoencoder AE2, respectively.

[0079] Cyclic consistency loss function l cycSince the converted visible light image may not retain the content information of the original input infrared image, this embodiment, inspired by the CycleGAN algorithm, utilizes a cycle consistency loss function to reconstruct the input infrared image (process autoencoder AE1 to generator G1), and then convert it back to a visible light image (process autoencoder AE2 to generator G2), and vice versa. The cycle consistency loss function restricts the space for generating the visible light image while preserving the content information of the original input infrared image. To simulate the joint loss function of the autoencoder and generative adversarial network, the cycle consistency loss function of this method is represented by the forward cycle consistency function and the backward cycle consistency function as follows:

[0080]

[0081] in,

[0082]

[0083] Therefore, the preset loss function is represented by a weighted combination of the gradient loss function, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The gradient loss function eliminates noise in the image, ensuring the spatial continuity and smoothness of the generated visible light image. The total inter-pixel loss function maintains the content consistency between the output visible light image of the image generation model and the corresponding input infrared image. The cycle consistency loss function restricts the space of the generated visible light image while preserving the content information of the original input infrared image. Thus, this embodiment effectively reduces noise in the generated visible light image and improves the overall visual effect of the input infrared image, enhancing the accuracy of converting infrared images into visible light images.

[0084] For other aspects of the embodiments of this application, please refer to Figure 3 The image training model further includes:

[0085] Image preprocessing layer 30 is used to preprocess the input infrared sample image using an adaptive histogram equalization method.

[0086] The image preprocessing layer 30 is located before the coding layer 10. Since the infrared sample images in this embodiment are obtained by acquiring infrared images from a publicly available dataset, and given that infrared images from publicly available datasets have characteristics such as blurred image formation and poor layering, this embodiment employs an adaptive histogram equalization (AHE) algorithm to preprocess the input infrared sample images in order to improve the local contrast of the infrared sample images and obtain more image details.

[0087] Please refer to Figure 3In other aspects of the embodiments of this application, the coding layer 10 and the image conversion layer 20 respectively include a residual network layer 50 (i.e., Figure 3 The residual network layer 50 consists of ResBlock_1-ResBlock_3 and ResBlock_4-ResBlock_6. The residual network layer 50 is a residual network layer module based on the Inception model, and its network structure design is as follows: Figure 5 As shown, the residual network layer 50 is formed by parallel connections of multiple convolutional kernels of different scales.

[0088] It should be noted that, Figure 5 In this context, Conv represents a convolutional layer, PreLU represents an activation function layer, and DepthwiseConv represents a depthwise convolutional layer.

[0089] Specifically, in one embodiment, this application uses parallel branches of three convolutional kernels of different scales in the convolutional layer. It is well known that global information in an image can be effectively extracted using large convolutional kernels, while local information can be effectively extracted using small convolutional kernels. Based on this principle, it is difficult for a typical convolutional network to extract all useful information using a single fixed convolutional kernel. Therefore, please refer to... Figure 5 In this embodiment, multiple scales are selected on the same convolutional layer, namely 1×1, 3×3, and 5×5 convolutional kernels. Simultaneously, to reduce the computational cost of the network, the residual network layer 50 provided in this embodiment employs a depthwise convolution structure in the intermediate stage. The depthwise convolution kernels are applied to each individual channel, reducing the computational cost by a factor of C, where C is the number of channels. Finally, the three parallel convolution results are concatenated into a single output.

[0090] Please refer to Figure 3 In other aspects of the embodiments of this application, it should be noted that the embodiments of this application also provide a latent space layer 40 (i.e., latent space Z) to store the output features q1 generated by the autoencoder AE1 and the output features q2 generated by the autoencoder AE2. Then, using an auxiliary random variable Φ, the non-differential sampling process is parameterized into a differential sampling process through parameterization, allowing the autoencoder to be trained using the backpropagation algorithm. The encoding operations q1~f1(q1|T) and q2~f2(q2|V) of the autoencoder are implemented using the following formulas:

[0091] q1 = AE1(T) + Φ;

[0092] q2=AE2(V)+Φ;

[0093] Wherein, Φ represents a random array with values ​​in the range of 0-1. The size of this array is the same as the output size of the autoencoder. The network proposed in this method uses 128 random sample values.

[0094] Therefore, in this embodiment, the infrared and visible light images are mapped to the common latent space generated by the ends of autoencoders AE1 and AE2, respectively, and then the adversarial training of generators G1 and G2 is used to map them to the corresponding images in the infrared and visible light image domains.

[0095] This application embodiment connects two similar autoencoder modules (autoencoder AE1 and autoencoder AE2) by using weight sharing. Since the last few layers of the autoencoder are responsible for extracting high-level features in the image, the latent space Z shares the weights of the last few layers of the autoencoder in the infrared image domain and the visible light image domain, thereby achieving the purpose of associating the features of the two images.

[0096] The following describes the unsupervised infrared image to visible light image conversion apparatus provided in the embodiments of this application. The unsupervised infrared image to visible light image conversion apparatus described below and the unsupervised infrared image to visible light image conversion method described above can be referred to in correspondence with each other.

[0097] Please refer to Figure 6 An unsupervised infrared image to visible light image conversion device, comprising:

[0098] The first acquisition module 201 is used to acquire the infrared image to be converted;

[0099] The second acquisition module 202 is used to acquire a visible light image; input the infrared image into an image generation model to obtain a visible light image output by the image generation model;

[0100] The image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model. The first image discrimination model is used to distinguish between the infrared sample image and the infrared generated image generated based on the infrared sample image, and the second image discrimination model is used to distinguish between the visible light sample image and the visible light generated image generated based on the visible light sample image.

[0101] The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0102] This embodiment of the unsupervised infrared-to-visible-light image conversion device obtains a visible-light image output by inputting the infrared image into an image generation model. The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, and the inter-pixel loss function between the visible-light generated image and the visible-light sample image. Since this embodiment represents the preset loss function through a weighted combination of the gradient loss function, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function, the gradient loss function eliminates image noise, thereby ensuring the spatial continuity and smoothness of the generated visible-light image; the total inter-pixel loss function maintains the content consistency between the output visible-light image of the image generation model and the corresponding input infrared image; and the cycle consistency loss function restricts the space of the generated visible-light image while retaining the content information of the original input infrared image. Thus, the embodiments of this application effectively reduce noise in the generated visible light image and improve the overall visual effect of the input infrared image, thereby improving the accuracy of converting the infrared image into a visible light image.

[0103] In one embodiment, the preset loss function is calculated based on the product of the gradient loss function of the visible sample image and the first weighting factor, the product of the total pixel loss function and the second weighting factor, the product of the standard generative adversarial network loss function and the third weighting factor, and the product of the cycle consistency loss function and the fourth weighting factor.

[0104] In one embodiment, the preset loss function is expressed by the following formula:

[0105] Preset loss function = λ0l grad +λ1l pix +λ2l GAN +λ3l cyc ;

[0106] Wherein, λ0, λ1, λ2, and λ3 represent the first weighting factor, the second weighting factor, the third weighting factor, and the fourth weighting factor, respectively; grad The gradient loss function represents the visible sample image; pix Represents the total inter-pixel loss function; l GAN This represents the standard generative adversarial network loss function; cyc This represents the cycle consistency loss function.

[0107] In one embodiment, the unsupervised infrared image to visible light image conversion device further includes: a model training module for training the image generation model using an image training model;

[0108] The image training model includes:

[0109] The encoding layer is used to extract features from the infrared sample image and the visible light sample image respectively to obtain a first feature map corresponding to the infrared sample image and a second feature map corresponding to the visible light sample image;

[0110] An image conversion layer is used to train the first feature map through a first generative adversarial network to obtain an infrared generated image, and to train the second feature map through a second generative adversarial network to obtain a visible light generated image.

[0111] The loss function calculation layer is used to calculate the preset loss function based on the infrared sample image, the visible light sample image, the infrared generated image, and the visible light generated image, and to train the image generation model using the preset loss function.

[0112] In one embodiment, the image training model further includes:

[0113] The image preprocessing layer is used to preprocess the input infrared sample image using an adaptive histogram equalization method.

[0114] In one embodiment, the coding layer and the image conversion layer each include a residual network, which is formed by parallel connections of multiple convolutional kernels of different scales.

[0115] It should be noted that the terminal and network device provided in this application embodiment can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0116] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other through the communication bus 740. The processor 710 can call a computer program in the memory 730 to execute the steps of an unsupervised infrared image to visible light image conversion method, including, for example: acquiring an infrared image to be converted; acquiring a visible light image; inputting the infrared image into an image generation model to obtain a visible light image output by the image generation model; wherein, the image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model, the first image discrimination model being used to distinguish between the infrared sample image and an infrared generated image based on the infrared sample image, and the second image discrimination model being used to distinguish between the visible light sample image and a visible light generated image based on the visible light sample image; the image generation model is trained using a preset loss function, the preset loss function being calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function; the total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, and the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0117] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0118] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the unsupervised infrared image to visible light image conversion method provided in the above embodiments, such as: acquiring an infrared image to be converted; acquiring a visible light image; inputting the infrared image into an image generation model to obtain a visible light image output by the image generation model; wherein the image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model, the first... An image discrimination model is used to distinguish between an infrared sample image and an infrared generated image based on the infrared sample image. A second image discrimination model is used to distinguish between a visible light sample image and a visible light generated image based on the visible light sample image. The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0119] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to execute the steps of the unsupervised infrared image to visible light image conversion method provided in the above embodiments, such as: acquiring an infrared image to be converted; acquiring a visible light image; inputting the infrared image into an image generation model to obtain a visible light image output by the image generation model; wherein, the image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model, the first image discrimination model being used to distinguish between the infrared sample image and an infrared generated image generated based on the infrared sample image, and the second image discrimination model being used to distinguish between the visible light sample image and a visible light generated image generated based on the visible light sample image; the image generation model is trained using a preset loss function, the preset loss function being calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function; the total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, and the inter-pixel loss function between the visible light generated image and the visible light sample image.

[0120] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An unsupervised method for converting infrared images to visible light images, characterized in that, include: Acquire the infrared image to be converted; Acquire visible light images; The infrared image is input into the image generation model to obtain the visible light image output by the image generation model; The image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model. The first image discrimination model is used to distinguish between the infrared sample image and the infrared generated image generated based on the infrared sample image, and the second image discrimination model is used to distinguish between the visible light sample image and the visible light generated image generated based on the visible light sample image. The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image. Before the step of acquiring the infrared image to be converted, the method further includes the step of training the image generation model through an image training model; The image training model includes an encoding layer, an image transformation layer, a loss function calculation layer, and a latent space layer. The encoding layer includes two autoencoder modules, namely autoencoder module AE1 and autoencoder module AE2. The latent space layer is used to store the output features q1 generated by autoencoder module AE1 and the output features q2 generated by autoencoder module AE2. Using an auxiliary random variable Φ, the non-differential sampling process is parameterized into a differential sampling process in a parameterized manner, and the autoencoder module can be trained by the backpropagation algorithm.

2. The unsupervised infrared image to visible light image conversion method according to claim 1, characterized in that, The preset loss function is calculated based on the product of the gradient loss function of the visible sample image and the first weighting factor, the product of the total pixel loss function and the second weighting factor, the product of the standard generative adversarial network loss function and the third weighting factor, and the product of the cycle consistency loss function and the fourth weighting factor. The preset loss function is expressed by the following formula: Preset loss function = λ0l grad +λ1l pix +λ2l GAN +λ3l cyc ; Wherein, λ0, λ1, λ2, and λ3 represent the first weighting factor, the second weighting factor, the third weighting factor, and the fourth weighting factor, respectively; grad The gradient loss function represents the visible sample image; pix Represents the total inter-pixel loss function; l GAN This represents the standard generative adversarial network loss function; cyc This represents the cycle consistency loss function.

3. The unsupervised infrared image to visible light image conversion method according to claim 1, characterized in that, The image training model includes: The encoding layer is used to extract features from the infrared sample image and the visible light sample image respectively to obtain a first feature map corresponding to the infrared sample image and a second feature map corresponding to the visible light sample image; An image conversion layer is used to train the first feature map through a first generative adversarial network to obtain an infrared generated image, and to train the second feature map through a second generative adversarial network to obtain a visible light generated image. The loss function calculation layer is used to calculate the preset loss function based on the infrared sample image, the visible light sample image, the infrared generated image, and the visible light generated image, and to train the image generation model using the preset loss function.

4. The unsupervised infrared image to visible light image conversion method according to claim 3, characterized in that, The image training model also includes: The image preprocessing layer is used to preprocess the input infrared sample image using an adaptive histogram equalization method.

5. The unsupervised infrared image to visible light image conversion method according to claim 3, characterized in that, The coding layer and the image conversion layer each include a residual network, which is formed by parallel connections of multiple convolutional kernels of different scales.

6. An unsupervised infrared image to visible light image conversion device, characterized in that, include: The first acquisition module is used to acquire the infrared image to be converted; The second acquisition module is used to acquire visible light images; The infrared image is input into the image generation model to obtain the visible light image output by the image generation model; The image generation model is generated based on an infrared sample image, a visible light sample image, a first image discrimination model, and a second image discrimination model. The first image discrimination model is used to distinguish between the infrared sample image and the infrared generated image generated based on the infrared sample image, and the second image discrimination model is used to distinguish between the visible light sample image and the visible light generated image generated based on the visible light sample image. The image generation model is trained using a preset loss function, which is calculated based on the gradient loss function of the visible sample image, the total inter-pixel loss function, the standard generative adversarial network loss function, and the cycle consistency loss function. The total inter-pixel loss function includes the inter-pixel loss function between the infrared generated image and the infrared sample image, as well as the inter-pixel loss function between the visible light generated image and the visible light sample image. Before acquiring the infrared image to be converted, the process also includes: training the image generation model using an image training model; The image training model includes an encoding layer, an image transformation layer, a loss function calculation layer, and a latent space layer. The encoding layer includes two autoencoder modules, namely autoencoder module AE1 and autoencoder module AE2. The latent space layer is used to store the output features q1 generated by autoencoder module AE1 and the output features q2 generated by autoencoder module AE2. Using an auxiliary random variable Φ, the non-differential sampling process is parameterized into a differential sampling process in a parameterized manner, and the autoencoder module can be trained by the backpropagation algorithm.

7. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the unsupervised infrared image to visible light image conversion method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the unsupervised infrared image to visible light image conversion method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the unsupervised infrared image to visible light image conversion method according to any one of claims 1 to 5.

Images