A Temperature Adaptive Infrared Image Colorization Method

Abstract

The present invention discloses a temperature - adaptive infrared image colorization method, belonging to the field of image processing. The method includes: capturing infrared images and visible - light images at different ambient temperatures; establishing a temperature mapping relationship; constructing paired temperature - mapped image - infrared normalized image - visible - light normalized image data pairs, and dividing the image data pairs into an image training set and an image test set; constructing a temperature - adaptive network, including a temperature module and an infrared colorization module, and completing the training process of the temperature - adaptive network; inputting the image test set into the trained temperature - adaptive network to generate infrared colorized images. The purpose of the present invention is to enhance the robustness of the network affected by the ambient temperature. By inputting infrared images and temperature - mapped images into the network, high - quality color images can be accurately reconstructed.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method for temperature - adaptive infrared image colorization. Background Art

[0002] Traditional infrared image colorization methods usually rely on specific color mapping techniques. Based on manually preset parameters, they use mathematical models and feature extraction to adjust the color balance and contrast of images. For example, the scribble - based colorization method allows researchers to manually annotate color information in grayscale images and achieve color diffusion by optimizing the global energy function. This method provides an effective solution for the visual presentation of infrared images through manual intervention and can generate colorized images that meet expectations in specific scenarios.

[0003] Deep - learning - based infrared image colorization methods utilize convolutional neural network architectures. By stacking multiple convolutional layers and using different convolutional kernels, they extract multi - level features in images. These methods can automatically learn the complex mapping relationship between infrared images and visible - light images, thus generating more natural and realistic colorized images. The introduction of deep learning has significantly improved the automation level and processing efficiency of infrared image colorization, providing new possibilities for image processing in complex scenarios.

[0004] However, neither traditional methods nor deep - learning methods take into account the impact of environmental temperature changes on infrared image acquisition. In practical applications, changes in environmental temperature can cause significant differences in the thermal radiation intensity of the same object in the same scene, thereby affecting the gray - level distribution of pixel points in infrared images. This change in gray - level distribution will further affect the color mapping effect during the colorization process, resulting in color distortion or detail loss in the generated images. Summary of the Invention

[0005] Aiming at the deficiencies of the above - mentioned prior art, the purpose of the present invention is to propose a method for temperature - adaptive infrared image colorization. By constructing a temperature - adaptive network, establishing the mapping relationship between infrared images and temperature - mapping images to visible - light images, enhancing the robustness of the network affected by environmental temperature, and inputting infrared images and temperature - mapping images into the network, high - quality color images can be accurately reconstructed.

[0006] Technical Solution: To solve the above - mentioned technical problems, the present invention is implemented by adopting the following technical solutions:

[0007] Step 1: At different environmental temperatures, capture infrared images and visible - light images.

[0008] Step 2: Use an infrared camera to photograph a blackbody radiation source with adjustable temperature, obtain infrared images of the blackbody radiation source at different temperatures, and at the same time use a temperature gun to record the surface temperature of the corresponding blackbody radiation source to establish a temperature mapping relationship;

[0009] Step 3: Utilize the temperature mapping relationship in Step 2 to perform temperature mapping processing on the infrared images taken at different ambient temperatures to obtain temperature mapping images; perform image preprocessing on the infrared images taken at different ambient temperatures to obtain infrared normalized images; perform image preprocessing on the visible light images taken at different ambient temperatures to obtain visible light normalized images; form paired temperature mapping image - infrared normalized image - visible light normalized image data pairs, and divide the image data pairs into an image training set and an image test set;

[0010] Step 4: Construct a temperature adaptive network, where the temperature adaptive network includes a temperature module and an infrared colorization module; use the infrared normalized images and temperature mapping images in the image training set as the input quantities of the temperature adaptive network, and the visible light normalized images as the labels of the temperature adaptive network to complete the training process of the temperature adaptive network;

[0011] Step 5: Input the temperature mapping images and infrared normalized images in the image test set into the trained temperature adaptive network to generate infrared colorized images.

[0012] In Step 1, the infrared images and visible light images are taken by an infrared camera and a visible light camera at the same time and in the same scene.

[0013] In Step 2, the temperature mapping relationship means that after accumulating all pixel values in each infrared image of the blackbody radiation source and taking the pixel average value, linearly fit the pixel average value with the surface temperature of the corresponding blackbody radiation source to establish a corresponding relationship in the form of a linear function between the pixel value and the temperature, that is, the temperature mapping relationship; where the linear function form between the pixel value and the temperature is T = 0.0045G - 76.82.

[0014] In Step 3, the temperature mapping processing means that using the temperature mapping relationship, convert the pixel value of each pixel point in each infrared image through the linear function form between the pixel value and the temperature into a temperature mapping image, where the value of each pixel point in the temperature mapping image represents the temperature value.

[0015] The image preprocessing means performing a normalization operation on the image to be processed and further performing a cropping operation; where the normalization operation linearly maps the values of the pixel points in the image to the interval from 0 to 1; where the cropping operation performs central cropping on the normalized image so that the resolution of the image is uniformly adjusted to a fixed size.

[0016] In step 4, the temperature module is used to extract information from the temperature mapping image in the training set, and the temperature module outputs a temperature feature image; further, the temperature feature image and the infrared normalized image in the training set are added with their corresponding pixel values to obtain an infrared enhanced image;

[0017] The temperature module includes a temperature downsampling module and a temperature upsampling module; the temperature downsampling module contains eight convolutional layers, and after each convolutional layer, an activation function ReLU and a normalization layer are connected in sequence; the temperature upsampling module contains eight convolutional layers, and after each convolutional layer, an activation function ReLU and a normalization layer are connected in sequence;

[0018] The infrared colorization module includes a Unet network and a visible light discriminator; the infrared enhanced image is input into the encoder in the Unet network, and the encoder extracts the edge and texture features of the infrared enhanced image to obtain intermediate enhanced features, and then the intermediate enhanced features pass through the decoder of the Unet network to reconstruct the predicted visible light image;

[0019] The visible light discriminator contains five convolutional layers, and a Leaky ReLU activation function is used after the first four convolutional layers; the visible light discriminator is used to distinguish the predicted visible light image and the visible light normalized image in the training set.

[0020] The normalization function, activation function ReLU, and activation function Leaky ReLU used in the present invention are as follows:

[0021]

[0022] Compared with the prior art, the present invention has the following beneficial technical effects:

[0023] At present, neither traditional algorithms nor deep learning algorithms take into account the influence of environmental temperature changes on infrared image acquisition, resulting in missing image information in the restored images. The present invention is a temperature adaptive infrared image colorization method, which aims to enhance the robustness to environmental temperature changes, so as to more accurately reconstruct high-quality color images and realize infrared image colorization.

[0024] The structure of the present invention is simple, easy to implement, has a wide application range, and strong versatility. Description of the Drawings

[0025] Figure 1 is a step diagram of a temperature adaptive infrared image colorization method of the present invention.

[0026] Figure 2 is the sample points recorded by the pixel average value and the surface temperature of the corresponding blackbody radiation source in the example of the present invention, as well as the corresponding relationship in the form of a linear function between the pixel value and the temperature.

[0027] Figure 3 is the temperature adaptive network flowchart of the present invention.

[0028] Figure 4 are four infrared normalized images in the image test set of the present invention example.

[0029] Figure 5 are four temperature mapping images in the image test set of the present invention example.

[0030] Figure 6 are four infrared colorized images generated in the present invention example.

[0031] Figure 7 are four visible light normalized images in the image test set of the present invention example. Detailed implementation manners

[0032] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0033] As Figure 1 shown, step 1: At different ambient temperatures of 20, 25, 30, and 35 degrees Celsius, capture infrared images and visible light images. The infrared images are 16-bit and in TIFF format; the visible light images are 8-bit and in JPG format. The infrared images and visible light images are captured by an infrared camera (model FLIR Boson320) and a visible light camera (model Nikon Z30) at the same time and in the same scene.

[0034] Step 2: Use the infrared camera to capture the temperature adjustable blackbody radiation source (model JQ-D70Z), obtain the infrared images of the blackbody radiation source at different temperatures, and at the same time use a temperature gun to record the surface temperature of the corresponding blackbody radiation source to establish a temperature mapping relationship.

[0035] Step 3: Utilize the temperature mapping relationship in step 2 to perform temperature mapping processing on the infrared images captured at different ambient temperatures to obtain temperature mapping images; after normalizing the infrared images captured at different ambient temperatures, the image size is centered and cropped to a fixed size of 256×256 to obtain infrared normalized images; after normalizing the visible light images captured at different ambient temperatures, the image size is centered and cropped to a fixed size of 256×256 to obtain visible light normalized images; form paired temperature mapping image-infrared normalized image-visible light normalized image data pairs, and divide the image data pairs into an image training set and an image test set according to 9:1.

[0036] Step 4: Construct a temperature adaptive network, which includes a temperature module and an infrared colorization module; use the infrared normalized images and temperature mapping images in the image training set as the input of the temperature adaptive network, and the visible light normalized images as the labels of the temperature adaptive network to complete the training process of the temperature adaptive network;

[0037] Step 5: Input the temperature mapping images and infrared normalized images in the image test set into the trained temperature adaptive network to generate infrared colorized images.

[0038] In Step 2, the temperature mapping relationship means that for each infrared image of the blackbody radiation source, the pixel values of all pixels are accumulated and then the pixel average value is taken. The pixel average value is linearly fitted with the surface temperature of the corresponding blackbody radiation source to establish a corresponding relationship in the form of a linear function between the pixel value and the temperature, that is, the temperature mapping relationship; where the linear function form between the pixel value and the temperature is;

[0039] As Figure 2 shown are the sample points recorded of the pixel average value and the surface temperature of the corresponding blackbody radiation source in the example of the present invention, as well as the corresponding relationship in the form of a linear function between the pixel value and the temperature; Figure 3 The abscissa represents the pixel average value, and the ordinate represents the recorded surface temperature of the blackbody radiation source; among them, the surface temperature of the blackbody radiation source starts to be recorded from 20 degrees Celsius, and is recorded once every 5 degrees Celsius increase using a temperature measuring gun, and finally recorded up to 65 degrees Celsius.

[0040] In Step 3, the temperature mapping process means that using the temperature mapping relationship, the pixel value of each pixel point in each infrared image is converted into a temperature mapping image through the linear function form between the pixel value and the temperature, where the value of each pixel point in the temperature mapping image represents the temperature value.

[0041] In Step 4, as Figure 3 shown, the temperature module is used to extract the information in the temperature mapping images in the training set, and the temperature module outputs a temperature feature image; further, the temperature feature image and the infrared normalized images in the training set are added with their corresponding pixel values to obtain an infrared enhanced image;

[0042] The temperature module includes a temperature downsampling module and a temperature upsampling module; the temperature downsampling module contains eight convolutional layers, and each convolutional layer is sequentially connected with an activation function ReLU and a normalization layer; the temperature upsampling module contains eight convolutional layers, and each convolutional layer is sequentially connected with an activation function ReLU and a normalization layer;

[0043] The parameters of the eight convolutional layers in the temperature downsampling module are inputs of 3, 64, 128, 256, 512, 1024, 1024, and 1024 channels respectively, and outputs of 64, 128, 256, 512, 1024, 1024, 1024, and 1024 channels respectively; the parameters of the eight convolutional layers in the temperature upsampling module are inputs of 1024, 1024, 1024, 1024, 512, 256, 128, and 64 channels respectively, and outputs of 1024, 1024, 1024, 512, 256, 128, 64, and 3 channels respectively;

[0044] The infrared colorization module includes a Unet network and a visible light discriminator; the Unet network is a convolutional neural network including an encoder and a decoder; as Figure 3 shown, the infrared enhanced image is input into the encoder in the Unet network, and the encoder extracts the edge and texture features of the infrared enhanced image to obtain intermediate enhanced features. Then, the intermediate enhanced features pass through the decoder to reconstruct the predicted visible light image;

[0045] As Figure 3 shown, the visible light discriminator includes five convolutional layers, and a Leaky ReLU activation function is used after the first four convolutional layers; the visible light discriminator is used to distinguish the predicted visible light image from the visible light normalized image in the training set;

[0046] The parameters of the five convolutional layers in the visible light discriminator are inputs of 6, 64, 128, 256, and 512 channels respectively, and outputs of 64, 128, 256, 512, and 1 channel respectively.

[0047] The normalization function, the linear function form of the pixel value and temperature, the activation function ReLU, and the activation function Leaky ReLU used in the present invention are as follows:

[0048] Figure 4 are four infrared normalized images in the image test set of the present invention example, Figure 4 In the scenes photographed from left to right, the ambient temperatures are 30, 25, 35, and 20 degrees Celsius respectively, Figure 5 are four temperature mapping images in the image test set of the present invention example, Figure 6 are four infrared colorized images generated in the present invention example, Figure 7 are four visible light normalized images in the image test set of the present invention example.

[0049] In the present invention example, the peak signal-to-noise ratio is used as the evaluation index:

[0050] Among them, the average peak signal-to-noise ratio of the four infrared normalized images in the image test set of the present invention example and the four visible light normalized images in the image test set of the present invention example is 20.27, and the average peak signal-to-noise ratio of the four infrared colorized images generated in the present invention example and the four visible light normalized images in the image test set of the present invention example is 30.67;

[0051] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the patent and not to limit them. Without departing from the principle of this patent, those of ordinary skill in the art can also make several modifications and improvements, which should also be regarded as the protection scope of this patent.

Claims

1. A temperature - adaptive infrared image colorization method, characterized in that, The method includes the following steps: Step 1: At different ambient temperatures, capture infrared images and visible light images; Step 2: Use an infrared camera to capture a blackbody radiation source with adjustable temperature, obtain infrared images of the blackbody radiation source at different temperatures, and at the same time use a temperature gun to record the surface temperature of the corresponding blackbody radiation source to establish a temperature mapping relationship; Step 3: Utilize the temperature mapping relationship in Step 2 to perform temperature mapping processing on the infrared images captured at different ambient temperatures to obtain temperature mapping images; the temperature mapping processing refers to using the temperature mapping relationship to convert the pixel value of each pixel point in each infrared image into a temperature mapping image through a linear function form of pixel value and temperature, where the value of each pixel point in the temperature mapping image represents the temperature value; perform image preprocessing on the infrared images captured at different ambient temperatures to obtain infrared normalized images; perform image preprocessing on the visible light images captured at different ambient temperatures to obtain visible light normalized images; form paired temperature mapping image - infrared normalized image - visible light normalized image data pairs, and divide the image data pairs into an image training set and an image test set; Step 4: Construct a temperature adaptive network and complete the training process of the temperature adaptive network; the temperature adaptive network includes a temperature module and an infrared colorization module; the infrared colorization module includes a Unet network and a visible light discriminator; the visible light discriminator contains five convolutional layers, and after the first four convolutional layers, a Leaky ReLU activation function is passed; the temperature module is used to extract the information in the temperature mapping images in the training set, and the temperature module outputs a temperature feature image; further add the corresponding pixel values of the temperature feature image and the infrared normalized images in the training set to obtain an infrared enhanced image; Input the infrared enhanced image into the encoder in the Unet network. The encoder extracts the edge and texture features of the infrared enhanced image to obtain intermediate enhanced features, and then the intermediate enhanced features pass through the decoder of the Unet network to reconstruct the predicted visible light image; the visible light discriminator is used to distinguish the predicted visible light image and the visible light normalized images in the training set; Step 5: Input the temperature mapping images and infrared normalized images in the image test set into the trained temperature adaptive network to generate infrared colorized images.

2. The temperature adaptive infrared image colorization method according to claim 1, characterized in that The infrared images and visible light images are captured by an infrared camera and a visible light camera at the same time and in the same scene.

3. A temperature adaptive infrared image colorization method according to claim 1, characterized in that, The temperature mapping relationship refers to taking the pixel average after accumulating all pixel values in each infrared image of the blackbody radiation source, linearly fitting the pixel average with the surface temperature of the corresponding blackbody radiation source, and establishing a corresponding relationship in the form of a linear function of pixel value and temperature, that is, the temperature mapping relationship; where the linear function form of pixel value and temperature is: T = 0.0045G - 76.

82.

4. A temperature adaptive infrared image colorization method according to claim 1, characterized in that, Image preprocessing refers to performing a normalization operation on the image to be processed and further performing a cropping operation. The normalization operation linearly maps the values of the pixels in the image to the interval from 0 to 1. The cropping operation is to perform central cropping on the normalized image so that the resolution of the image is uniformly adjusted to a fixed size.

5. A temperature adaptive infrared image colorization method according to claim 1, characterized in that, The temperature module includes a temperature downsampling module and a temperature upsampling module. The temperature downsampling module contains eight convolutional layers, and after each convolutional layer, an activation function ReLU and a normalization layer are connected in sequence. The temperature upsampling module contains eight convolutional layers, and after each convolutional layer, an activation function ReLU and a normalization layer are connected in sequence.

Images