Temperature-adaptive infrared image colorization method

Abstract

The invention discloses a temperature-adaptive infrared image colorization method, and belongs to the field of image processing. The method comprises the following steps: shooting to obtain an infrared image and a visible light image at different environment temperatures; establishing a temperature mapping relation; constructing a paired temperature mapping image-infrared normalized image-visible light normalized image data pair, and dividing the image data pair into an image training set and an image test set; a temperature self-adaptive network is constructed and comprises a temperature module and an infrared colorization module, and the training process of the temperature self-adaptive network is completed; and inputting the image test set into the trained temperature adaptive network to generate an infrared colorized image. According to the method, the robustness of the network influenced by the environment temperature is enhanced, and the high-quality color image is accurately reconstructed by inputting the infrared image and the temperature mapping image into the network.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method for temperature - adaptive infrared image colorization. Background Art

[0002] Traditional infrared image colorization methods usually rely on specific color mapping techniques. Based on manually preset parameters, mathematical models and feature extraction are used to adjust the color balance and contrast of the image. For example, the scribble - based colorization method allows researchers to manually annotate color information in grayscale images and realizes color diffusion by optimizing the global energy function. This method provides an effective solution for the visual presentation of infrared images through manual intervention and can generate colorized images that meet expectations in specific scenarios.

[0003] Deep - learning - based infrared image colorization methods utilize convolutional neural network architectures. By stacking multiple convolutional layers and using different convolutional kernels, multi - level features in the image are extracted. These methods can automatically learn the complex mapping relationship between infrared images and visible - light images, thereby generating more natural and realistic colorized images. The introduction of deep learning has significantly improved the automation level and processing efficiency of infrared image colorization, providing new possibilities for image processing in complex scenarios.

[0004] However, neither traditional methods nor deep - learning methods take into account the impact of environmental temperature changes on infrared image acquisition. In practical applications, changes in environmental temperature will cause significant differences in the thermal radiation intensity of the same object in the same scene, thereby affecting the gray - scale distribution of pixel points in the infrared image. This change in gray - scale distribution will further affect the color mapping effect during the colorization process, resulting in color distortion or detail loss in the generated image. Summary of the Invention

[0005] Aiming at the deficiencies of the above - mentioned existing technologies, the purpose of the present invention is to propose a method for temperature - adaptive infrared image colorization. By constructing a temperature - adaptive network, establishing the mapping relationship between infrared images and temperature - mapped images to visible - light images, enhancing the robustness of the network affected by environmental temperature, and inputting infrared images and temperature - mapped images into the network, a high - quality color image can be accurately reconstructed.

[0006] Technical Solution: To solve the above - mentioned technical problems, the present invention is implemented by adopting the following technical solutions: Step 1: At different environmental temperatures, capture infrared images and visible - light images. Step 2: Use an infrared camera to photograph a black - body radiation source with adjustable temperature, obtain infrared images of the black - body radiation source at different temperatures, and at the same time use a temperature gun to record the surface temperature of the corresponding black - body radiation source to establish a temperature mapping relationship. Step 3: Using the temperature mapping relationship in Step 2, perform temperature mapping processing on the infrared images taken at different ambient temperatures to obtain temperature mapping images; perform image preprocessing on the infrared images taken at different ambient temperatures to obtain infrared normalized images; perform image preprocessing on the visible light images taken at different ambient temperatures to obtain visible light normalized images; form paired temperature mapping image-infrared normalized image-visible light normalized image data pairs, and divide the image data pairs into an image training set and an image test set; Step 4: Construct a temperature adaptive network, where the temperature adaptive network includes a temperature module and an infrared colorization module; use the infrared normalized images and temperature mapping images in the image training set as the input of the temperature adaptive network, and the visible light normalized image as the label of the temperature adaptive network to complete the training process of the temperature adaptive network; Step 5: Input the temperature mapping images and infrared normalized images in the image test set into the trained temperature adaptive network to generate infrared colorized images.

[0007] In Step 1, the infrared images and visible light images are taken by an infrared camera and a visible light camera at the same time and in the same scene.

[0008] In Step 2, the temperature mapping relationship means that for each infrared image of the blackbody radiation source, after accumulating all pixel values and taking the pixel average value, linearly fit the pixel average value with the surface temperature of the corresponding blackbody radiation source to establish a corresponding relationship in the form of a linear function between the pixel value and the temperature, that is, the temperature mapping relationship; where the linear function form between the pixel value and the temperature is T = 0.0045G - 76.82.

[0009] In Step 3, the temperature mapping processing means that using the temperature mapping relationship, convert the pixel value of each pixel point of each infrared image through the linear function form between the pixel value and the temperature into a temperature mapping image, where the value of each pixel point in the temperature mapping image represents the temperature value.

[0010] The image preprocessing means performing a normalization operation on the image to be processed and further performing a cropping operation; where the normalization operation linearly maps the pixel values in the image to the interval from 0 to 1; where the cropping operation performs a central crop on the normalized image to uniformly adjust the resolution of the image to a fixed size.

[0011] In Step 4, the temperature module is used to extract the information in the temperature mapping images in the training set, and the temperature module outputs a temperature feature image; further add the corresponding pixel values of the temperature feature image and the infrared normalized images in the training set to obtain an infrared enhanced image; The temperature module includes a temperature downsampling module and a temperature upsampling module; the temperature downsampling module contains eight convolutional layers, and an activation function ReLU and a normalization layer are sequentially connected after each convolutional layer; the temperature upsampling module contains eight convolutional layers, and an activation function ReLU and a normalization layer are sequentially connected after each convolutional layer; The infrared colorization module includes a Unet network and a visible light discriminator; the infrared enhanced image is input into the encoder in the Unet network, and the encoder extracts the edge and texture features of the infrared enhanced image to obtain intermediate enhanced features. Then, the intermediate enhanced features pass through the decoder of the Unet network to reconstruct the predicted visible light image; The visible light discriminator contains five convolutional layers, and a Leaky ReLU activation function is used after the first four convolutional layers; the visible light discriminator is used to distinguish the predicted visible light image and the visible light normalized image in the training set.

[0012] The normalization function, activation function ReLU, and activation function Leaky ReLU used in the present invention are as follows:

[0013] Compared with the prior art, the present invention has the following beneficial technical effects: At present, neither traditional algorithms nor deep learning algorithms take into account the impact of environmental temperature changes on infrared image acquisition, resulting in missing image information in the restored images. The present invention is a temperature adaptive infrared image colorization method, which aims to enhance the robustness to environmental temperature changes, thereby more accurately reconstructing high-quality color images and realizing infrared image colorization.

[0014] The structure of the present invention is simple, easy to implement, has a wide range of applications, and strong versatility. Description of the Drawings

[0015] Figure 1 is a step diagram of a temperature adaptive infrared image colorization method of the present invention.

[0016] Figure 2 are the sample points recorded by the pixel average value and the surface temperature of the corresponding blackbody radiation source in the example of the present invention, as well as the corresponding relationship in the form of a linear function between the pixel value and the temperature.

[0017] Figure 3 is a flowchart of the temperature adaptive network of the present invention.

[0018] Figure 4 are four infrared normalized images in the image test set of the example of the present invention.

[0019] Figure 5 are four temperature mapping images in the image test set of the example of the present invention.

[0020] Figure 6 These are four infrared colorized images generated in the embodiments of the present invention.

[0021] Figure 7 These are four visible light normalized images in the image test set of the embodiments of the present invention. Specific Embodiments

[0022] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.

[0023] As Figure 1 shown, Step 1: At different ambient temperatures of 20, 25, 30, and 35 degrees Celsius, capture infrared images and visible light images. The infrared images are 16-bit, in TIFF format; the visible light images are 8-bit, in JPG format. The infrared images and visible light images are captured by an infrared camera (model FLIR Boson320) and a visible light camera (model Nikon Z30) at the same time and in the same scene. Step 2: Use the infrared camera to capture the blackbody radiation source with adjustable temperature (model JQ-D70Z), obtain the infrared images of the blackbody radiation source at different temperatures, and at the same time use a temperature measuring gun to record the surface temperature of the corresponding blackbody radiation source to establish a temperature mapping relationship. Step 3: Utilize the temperature mapping relationship in Step 2 to perform temperature mapping processing on the infrared images captured at different ambient temperatures to obtain temperature mapping images; after normalizing the infrared images captured at different ambient temperatures, the image size is centrally cropped to a fixed size of 256×256 to obtain infrared normalized images; after normalizing the visible light images captured at different ambient temperatures, the image size is centrally cropped to a fixed size of 256×256 to obtain visible light normalized images; form paired temperature mapping image-infrared normalized image-visible light normalized image data pairs, and divide the image data pairs into an image training set and an image test set according to 9:1. Step 4: Construct a temperature adaptive network, and the temperature adaptive network includes a temperature module and an infrared colorization module; use the infrared normalized images and temperature mapping images in the image training set as the input quantities of the temperature adaptive network, and the visible light normalized images as the labels of the temperature adaptive network to complete the training process of the temperature adaptive network. Step 5: Input the temperature mapping images and infrared normalized images in the image test set into the trained temperature adaptive network to generate infrared colorized images.

[0024] In the second step, the temperature mapping relationship means that for each infrared image of the blackbody radiation source, after accumulating all pixel values and taking the pixel average value, a linear fitting is performed between the pixel average value and the surface temperature of the corresponding blackbody radiation source to establish a corresponding relationship in the form of a first-order function between the pixel value and the temperature, that is, the temperature mapping relationship; the form of the first-order function between the pixel value and the temperature is; As Figure 2 shown are the sample points recorded for the pixel average value and the surface temperature of the corresponding blackbody radiation source in the example of the present invention, as well as the corresponding relationship in the form of a first-order function between the pixel value and the temperature; Figure 3 The abscissa represents the pixel average value, and the ordinate represents the surface temperature of the recorded blackbody radiation source; among them, the surface temperature of the blackbody radiation source starts to be recorded from 20 degrees Celsius, and is recorded once every 5 degrees Celsius increase using a temperature measuring gun, and finally recorded up to 65 degrees Celsius.

[0025] In the third step, the temperature mapping process means that using the temperature mapping relationship, the pixel value of each pixel point in each infrared image is converted into a temperature mapping image through the first-order function form between the pixel value and the temperature, where the value of each pixel point in the temperature mapping image represents the temperature value.

[0026] In the fourth step, as Figure 3 shown, the temperature module is used to extract information from the temperature mapping images in the training set, and the temperature module outputs a temperature feature image; further, the temperature feature image and the infrared normalized image in the training set are added with their corresponding pixel values to obtain an infrared enhanced image; The temperature module includes a temperature downsampling module and a temperature upsampling module; the temperature downsampling module contains eight convolutional layers, and after each convolutional layer, an activation function ReLU and a normalization layer are connected in sequence; the temperature upsampling module contains eight convolutional layers, and after each convolutional layer, an activation function ReLU and a normalization layer are connected in sequence; The parameters of the eight convolutional layers in the temperature downsampling module are inputs of 3, 64, 128, 256, 512, 1024, 1024, and 1024 channels in sequence, and outputs of 64, 128, 256, 512, 1024, 1024, 1024, and 1024 channels; the parameters of the eight convolutional layers in the temperature upsampling module are inputs of 1024, 1024, 1024, 1024, 512, 256, 128, and 64 channels in sequence, and outputs of 1024, 1024, 1024, 512, 256, 128, 64, and 3 channels; The infrared colorization module includes a Unet network and a visible light discriminator; the Unet network is a convolutional neural network including an encoder and a decoder; as Figure 3As shown, the infrared enhanced image is input into the encoder of the Unet network. The encoder extracts the edge and texture features of the infrared enhanced image to obtain intermediate enhanced features. Then, the intermediate enhanced features pass through the decoder to reconstruct the predicted visible light image. As Figure 3 shown, the visible light discriminator includes five convolutional layers, and a Leaky ReLU activation function is used after the first four convolutional layers. The visible light discriminator is used to distinguish the predicted visible light image from the visible light normalized image in the training set. The parameters of the five convolutional layers in the visible light discriminator are inputs with 6, 64, 128, 256, and 512 channels respectively, and outputs with 64, 128, 256, 512, and 1 channel respectively.

[0027] The normalization function, the linear function form of pixel value and temperature, the activation function ReLU, and the activation function Leaky ReLU used in the present invention are as follows:

[0028] Figure 4 are four infrared normalized images in the image test set of the present invention example. Figure 4 In the scenes photographed from left to right, the ambient temperatures are 30, 25, 35, and 20 degrees Celsius respectively. Figure 5 are four temperature mapping images in the image test set of the present invention example. Figure 6 are four infrared colorized images generated in the present invention example. Figure 7 are four visible light normalized images in the image test set of the present invention example.

[0029] In the present invention example, the peak signal-to-noise ratio is used as the evaluation index: Among them, the average peak signal-to-noise ratio of the four infrared normalized images in the image test set of the present invention example and the four visible light normalized images in the image test set of the present invention example is 20.27, and the average peak signal-to-noise ratio of the four infrared colorized images generated in the present invention example and the four visible light normalized images in the image test set of the present invention example is 30.67. Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the patent and not to limit them. Without departing from the principles of this patent, those of ordinary skill in the art can still make several variations and improvements, which should also be regarded as the protection scope of this patent.

Claims

1. A temperature adaptive infrared image colorization method, characterized in that The method includes the following steps: Step 1: Capture infrared images and visible light images at different ambient temperatures; Step 2: Use an infrared camera to capture infrared images of a blackbody radiation source with adjustable temperature, obtain the infrared images of the blackbody radiation source at different temperatures, and at the same time use a temperature gun to record the surface temperature of the corresponding blackbody radiation source, and establish a temperature mapping relationship; Step 3: Utilize the temperature mapping relationship in Step 2 to perform temperature mapping processing on the infrared images captured at different ambient temperatures to obtain temperature mapping images; perform image preprocessing on the infrared images captured at different ambient temperatures to obtain infrared normalized images; perform image preprocessing on the visible light images captured at different ambient temperatures to obtain visible light normalized images; form paired temperature mapping image-infrared normalized image-visible light normalized image data pairs, and divide the image data pairs into an image training set and an image test set; Step 4: Construct a temperature adaptive network, where the temperature adaptive network includes a temperature module and an infrared colorization module; use the infrared normalized images and temperature mapping images in the image training set as the input of the temperature adaptive network, and the visible light normalized image as the label of the temperature adaptive network to complete the training process of the temperature adaptive network; Step 5: Input the temperature mapping images and infrared normalized images in the image test set into the trained temperature adaptive network to generate infrared colorized images.

2. The temperature adaptive infrared image colorization method according to claim 1, characterized in that The infrared images and visible light images are captured by an infrared camera and a visible light camera at the same time and in the same scene.

3. A temperature adaptive infrared image colorization method according to claim 1, characterized in that, The temperature mapping relationship means that for each infrared image of the blackbody radiation source, after accumulating all pixel values and taking the pixel average value, linearly fit the pixel average value with the surface temperature of the corresponding blackbody radiation source to establish a corresponding relationship in the form of a linear function between the pixel value and the temperature, that is, the temperature mapping relationship; where the linear function form between the pixel value and the temperature is: T = 0.0045G - 76.

82.

4. A temperature adaptive infrared image colorization method according to claim 1, characterized in that, The temperature mapping processing means that using the temperature mapping relationship, convert the pixel value of each pixel point in each infrared image through the linear function form between the pixel value and the temperature into a temperature mapping image, where the value of each pixel point in the temperature mapping image represents the temperature value.

5. A temperature adaptive infrared image colorization method according to claim 1, characterized in that Image preprocessing means performing a normalization operation on the image to be processed and further performing a cropping operation; where the normalization operation linearly maps the pixel values in the image to the interval of 0 to 1; where the cropping operation performs a central crop on the normalized image to uniformly adjust the resolution of the image to a fixed size.

6. A temperature adaptive infrared image colorization method according to claim 1, characterized in that, The temperature module is used to extract information from the temperature mapping images in the training set, and the temperature module outputs a temperature feature image; further add the corresponding pixel values of the temperature feature image and the infrared normalized images in the training set to obtain an infrared enhanced image.

7. A temperature adaptive infrared image colorization method according to claim 6, characterized in that, The temperature module includes a temperature downsampling module and a temperature upsampling module; the temperature downsampling module contains eight convolutional layers, and each convolutional layer is sequentially connected to an activation function ReLU and a normalization layer; the temperature upsampling module contains eight convolutional layers, and each convolutional layer is sequentially connected to an activation function ReLU and a normalization layer.

8. A temperature adaptive infrared image colorization method according to claim 1, characterized in that The infrared colorization module includes a Unet network and a visible light discriminator; The infrared enhanced image is input into the encoder in the Unet network. The encoder extracts the edge and texture features of the infrared enhanced image to obtain intermediate enhanced features, and then the intermediate enhanced features pass through the decoder of the Unet network to reconstruct the predicted visible light image.

9. A temperature adaptive infrared image colorization method according to claim 8, characterized in that, The visible light discriminator contains five convolutional layers, and a Leaky ReLU activation function is used after the first four convolutional layers; the visible light discriminator is used to distinguish the predicted visible light image from the visible light normalized image in the training set.

Images