An infrared image colorization method based on attention mask

Abstract

The application discloses an infrared image colorization method based on an attention mask, wherein an infrared image input preliminary mapping module generates a preliminary color image; the preliminary color image is input into an attention mask module to generate a content mask, a foreground attention mask and a background attention mask; the foreground attention mask and the background attention mask are fused with the corresponding content mask to enhance the color and details of a specific area, and a target image is obtained at the same time; a cycle consistency loss of the mask generator is determined to ensure that the image generated after each input image is cycled can be restored to the original input image; a perception loss is determined to improve the perception quality of the generated image; and a color image is output. The application can complete the infrared image colorization task while significantly improving the colorization effect of important information contained in the foreground content of the infrared image, so that the details of the infrared image after colorization can be more clearly and completely displayed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image translation, and particularly relates to an infrared image colorization method based on attention mask. BACKGROUND

[0002] In recent years, with the continuous improvement of people's living standards, the number of cars has also become more and more, in addition, with the vigorous development of artificial intelligence technology, automatic driving has also been more and more widely concerned by people. In order to collect more valuable images during driving, infrared technology is often used to collect the situation during vehicle driving, because infrared images are almost not affected by environmental changes, and can still stably collect images in bad weather or low visibility conditions, which has a much larger application range than visible light cameras which are extremely susceptible to lighting conditions, and performs more stably. At the same time, the shortcomings of infrared images are also obvious, it is a single-channel grayscale image formed according to the temperature emitted by the object, and does not have rich color information, details and important content are not easy for people to understand and further processed by computers. Therefore, how to colorize the infrared images collected in automatic driving into RGB images with rich color information has very important value and significance.

[0003] On the problem of colorization of infrared images, early research is a kind of manual design of color space as a reference system to colorize different semantic information contained in infrared images; with the emergence and popularity of deep learning, a large number of models based on neural network structure have emerged to colorize infrared images. These methods can be divided into two categories, namely supervised learning and unsupervised learning methods. Supervised learning needs paired infrared-visible light image data for training, but in the field of automatic driving, the change of traffic scene is very rapid, so it is a very difficult work to collect accurate paired infrared-visible light image data. Therefore, it is more suitable to use unsupervised learning to complete the colorization of infrared images by translating non-paired infrared and visible light images, but the existing unsupervised model pays more attention to the overall coloring of the infrared image after colorization, and does not pay much attention to some important information contained in the infrared image, for example, in the infrared image collected in automatic driving, the road situation in front and the surrounding vehicle situation convey more important foreground information, while the sky information in the distance belongs to secondary background information. Therefore, how to colorize the infrared image while clearly translating the foreground content containing more important information is a new concern. Based on this problem, an excellent deep learning model in the field of image translation, namely the model based on generative adversarial network, has not proposed an effective method, and the generated result cannot make some important contents in the foreground correctly and clearly colored. SUMMARY

[0004] The technical problem to be solved by the present application is to overcome the defects of the prior art, provide an attention mask-based infrared image colorization method for fine control of different regions in an image, especially in emphasizing foreground content and important information, and improve the quality and detail performance of the colorized image.

[0005] The technical solution adopted to solve the above technical problems is: an attention mask-based infrared image colorization method, comprising the following steps:

[0006] Step 1. An initial color image is generated by an infrared image input preliminary mapping module, and the preliminary mapping module is composed of an encoder, a conversion layer, and a decoder connected in series;

[0007] Step 1.1. The infrared image is input to the encoder for two-dimensional convolution operation to generate an initial feature map, and the initial feature map is sequentially subjected to normalization operation and first activation function processing, and the output result is output to the conversion layer;

[0008] Step 1.2. The conversion layer further deepens the features and outputs them to the decoder;

[0009] Step 1.3. The decoder performs deconvolution operation on the output of the conversion layer to generate a feature map closer to the original image size, and sequentially generates an initial color image through normalization operation and second activation function processing;

[0010] Step 2. The initial color image is input to an attention mask module to generate a content mask, a foreground attention mask, and a background attention mask;

[0011] The attention mask module is composed of a parameter sharing encoder and a mask generator connected in series, and the mask generator includes a content mask generator and an attention mask generator connected in parallel;

[0012] Step 2.1. The initial color image is input to the parameter sharing encoder for feature extraction to generate a feature map m, which is input to the mask generator;

[0013] Step 2.2. The content mask generator of the mask generator is used to extract the foreground content of the image to generate a plurality of content masks;

[0014] Step 2.3. The attention mask generator generates a foreground attention mask and a background attention mask;

[0015] Step 3. The foreground attention mask and the background attention mask are fused with the corresponding content mask to enhance the color and details of a specific region, and a target image is obtained;

[0016] Step 4. The cycle consistency loss of the mask generator is determined according to the following formula, to ensure that the image generated after the cycle of each input image can be restored to the original input image;

[0017]

[0018] In the formula, is the cycle consistency loss function, x is the input infrared domain image, y is the visible light domain image corresponding to x, G(x) is the y domain image generated by the generator G, F(G(x)) is the image restored to the source infrared domain from the y domain image generated by the generator G, and E x～Pdata(x) [·] is the L1 normalized expectation of the difference between the input image x and the image after twice generator reconstruction, E y～Pdata(y) [·] is the L1 normalized expectation of the difference between the input y domain image and the image after twice generator reconstruction; F(y) is the generated x domain image, G(F(y)) is the image restored to the source visible light domain from the x domain image generated by the generator F, and ‖·‖1 is the L1 norm;

[0019] Step 5. The perceptual loss is determined according to the following formula, to improve the perceptual quality of the generated image;

[0020]

[0021] In the formula, is the perceptual loss, is the deep feature of the image extracted using the neural network pre-training model, and ‖·‖2 is the L2 norm;

[0022] Step 6. Output the color target image.

[0023] As a preferred technical solution, in step 1.1, the first activation function is a ReLU activation function.

[0024] As a preferred technical solution, in step 1.2, the conversion layer is composed of 9 ResNET modules with the same structure, and the ResNET module is a five-layer convolutional neural network.

[0025] As a preferred technical solution, in step 1.3, the second activation function is a Tanh activation function.

[0026] As a preferred technical solution, in step 2.2, the content mask generator is generated according to the following formula,

[0027]

[0028] In the formula, is the content mask, tanh(·) is the activation process, m is the parameter shared encoder generated feature map, is the weight of each position in the content mask, is the coordinate value of each position in the content mask, n represents the number of generated masks;

[0029] As a preferred technical solution, in step 2.3, the attention mask generator generates the foreground attention mask and the background attention mask according to the following formula,

[0030]

[0031] In the formula, is the foreground attention mask, Softmax(·) is a channel-level function for normalization, is the weight of each position in the attention mask, is the coordinate value of each position in the attention mask, n is the number of generated attention masks, is the background attention mask.

[0032] As a preferred technical solution, in step 3, the foreground attention mask and the background attention mask are fused with the corresponding content mask according to the following formula to obtain the target image,

[0033]

[0034] In the formula, G(X) is the target image.

[0035] The beneficial effects of the present application are as follows:

[0036] The present application can clearly display the foreground content in the infrared image through the attention mask mechanism, and generate a color image that is visually closer to the real visible light image by using the cycle consistency loss and the perception loss, thereby improving the naturalness and realism of the image.

[0037] The present application reduces the complexity and computational resource consumption of the attention mask module while maintaining the efficiency and accuracy of feature extraction, thereby improving the generalization ability of the model.

[0038] Compared with the prior art, the present application pays more attention to the clear translation of the foreground content containing important information in the infrared image, thereby improving the information value of the colorized image and maintaining a high colorization effect under different environments and conditions, especially under low light or harsh weather conditions, the stability and reliability of the infrared image are higher.

[0039] The present application is not only suitable for autonomous driving, but also can be extended to other fields such as medical imaging, military reconnaissance, industrial detection, etc., and has a wide application prospect. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the infrared image colorization method based on attention masking according to the present invention.

[0041] Figure 2 This is a schematic diagram of the structure of the preliminary mapping module of the present invention.

[0042] Figure 3 This is a schematic diagram of the process of generating a color image from an infrared image according to the present invention.

[0043] Figure 4 This is a line graph comparing the SSIM results in the experiments of this invention.

[0044] Figure 5 This is a line graph comparing the PSNR results in the experiments of this invention. Detailed Implementation

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0046] exist Figure 1 The infrared image colorization method based on attention masking in this embodiment includes the following steps:

[0047] Step 1. The infrared image input to the preliminary mapping module generates a preliminary color image. The preliminary mapping module consists of an encoder, a conversion layer, and a decoder connected in series, as shown below. Figure 2 ;

[0048] Step 1.1. The infrared image is input into the encoder for two-dimensional convolution operation to generate a preliminary feature map. The preliminary feature map is then normalized and processed by the ReLU activation function. The output result is sent to the transformation layer.

[0049] Step 1.2. The transformation layer further refines the features and outputs them to the decoder. The transformation layer consists of 9 ResNet modules with the same structure. Each ResNet module is a five-layer convolutional neural network.

[0050] Step 1.3. The decoder performs a deconvolution operation on the output of the conversion layer to generate a feature map that is closer to the size of the original image, and then processes it through normalization and Tanh activation function to generate a preliminary color image.

[0051] Step 2. The initial color image is input into the attention mask module to generate content mask, foreground attention mask and background attention mask. The attention mask module is composed of a parameter-shared encoder and a mask generator connected in series. The mask generator includes a content mask generator and an attention mask generator connected in parallel.

[0052] Step 2.1. The preliminary color image is input to the parameter-shared encoder for feature extraction to generate feature map m, which is input to the mask generator;

[0053] Step 2.2. The content mask generator of the mask generator extracts the foreground content of the image, and generates a plurality of content masks according to the following formula,

[0054]

[0055] In the formula, is the content mask, tanh(·) is the activation process, m is the feature map generated by the parameter-shared encoder, is the weight of each position in the content mask, is the coordinate value of each position in the content mask, and n represents the number of generated masks;

[0056] Step 2.3. The attention mask generator generates foreground attention masks and background attention masks according to the following formula,

[0057]

[0058] In the formula, is the foreground attention mask, Softmax(·) is a channel-level function for normalization, is the weight of each position in the attention mask, is the coordinate value of each position in the attention mask, and n is the number of generated attention masks, is the background attention mask;

[0059] Step 3. The foreground attention mask and the background attention mask are fused with the corresponding content mask according to the following formula to enhance the color and details of specific regions, and the target image G(X) is obtained;

[0060]

[0061] Step 4. The cycle consistency loss of the mask generator is determined according to the following formula to ensure that the image generated after the cycle of each input image can be restored to the original input image;

[0062]

[0063] In the formula, is the cycle consistency loss function, x is the input infrared domain image, y is the visible light domain image corresponding to x, G(x) is the y domain image generated by the generator G, F(G(x)) is the image restored to the source infrared domain from the y domain image generated by the generator G, and E x～Pdata(x) [·] is the L1 normalized expectation of the difference between the input image x and the image after two generator reconstructions, E y～Pdata(y)[·] is the L1 normalized expectation of the difference between the input y domain image and the image after twice generator reconstruction; F(y) is the generated x domain image, G(F(y)) is the image recovered to the source visible light domain for the x domain image generated by the generator F, ‖·‖1 is the L1 norm;

[0064] Step 5. The perceptual loss is determined according to the following formula for improving the perceptual quality of the generated image;

[0065]

[0066] In the formula, is the perceptual loss, is the deep feature of the image extracted using the neural network pre-training model, and ‖·‖2 is the L2 norm;

[0067] Step 6. Output the color target image, such as Figure 3 .

[0068] Experiment

[0069] In order to verify the beneficial effects of the present application, the inventors made the following comparative experiments using the method of the present application and the prior art method:

[0070] 1) Constructing a data set

[0071] From the known IRVI data set, 16999 infrared images in the traffic field image data and the corresponding number of visible light images were selected to form a data set;

[0072] 2) Experimental method

[0073] The method of the present application was used to colorize the infrared images in the data set, and the results were compared with several existing classic image translation methods (such as CycleGAN, TodayGAN, I2V-GAN, AttentionGAN). SSIM (Structural Similarity Index) and PSNR (Peak Signal to Noise Ratio) were used as objective evaluation indicators to measure the similarity and quality between the generated image and the real visible light image, such as Figure 4 、 5 The results are shown in Table 1 below, which shows that the method of the present application is superior to the comparative methods in both SSIM and PSNR indicators, showing better colorization effect and image quality.

[0074] Table 1 Test results of different models on the data set

[0075]

[0076] The experimental results show that the method has significant technical advantages in infrared image colorization, especially in enhancing the details and color performance of the foreground content, and effectively improves the perceptual quality and cycle stability of the generated image.

Claims

1. An infrared image colorization method based on attention masking, characterized in that, Includes the following steps: Step 1. The infrared image input to the preliminary mapping module generates a preliminary color image. The preliminary mapping module consists of an encoder, a conversion layer, and a decoder connected in series. Step 1.

1. The infrared image is input into the encoder for two-dimensional convolution operation to generate a preliminary feature map. The preliminary feature map is then normalized and processed by the first activation function. The output result is sent to the transformation layer. Step 1.

2. The transformation layer further refines the features and outputs them to the decoder; Step 1.

3. The decoder performs a deconvolution operation on the output of the conversion layer to generate a feature map that is closer to the size of the original image, and then processes it through normalization and the second activation function to generate a preliminary color image. Step 2. The initial color image is input into the attention mask module to generate content mask, foreground attention mask, and background attention mask; The attention mask module consists of a parameter-shared encoder and a mask generator connected in series. The mask generator includes a content mask generator and an attention mask generator connected in parallel. Step 2.

1. The initial color image is input into the parameter-shared encoder for feature extraction, generating a feature map m which is then input into the mask generator; Step 2.

2. The content mask generator is used to extract the foreground content of the image and generate multiple content masks; Step 2.

3. The attention mask generator generates foreground and background attention masks; Step 3. Fuse the foreground attention mask and background attention mask with the corresponding content mask to enhance the color and detail of specific areas while obtaining the target image; Step 4. Determine the cycle consistency loss of the mask generator according to the following formula to ensure that the image generated after each input image cycle can be restored to the original input image; In the formula, It is a cycle consistency loss function, where x is the input infrared domain image, y is the corresponding visible domain image, G(x) is the y-domain image generated by generator G, and F(G(x)) is the image restored from the source infrared domain to the y-domain image generated by generator G. x～Pdata(x) [·] is the L1 normalized expectation of the difference between the input image x and the image reconstructed by the generator twice, Ε y～Pdata(y) [·] is the L1 normalized expectation of the difference between the input y-domain image and the image reconstructed by the generator twice; F(y) is the generated x-domain image, G(F(y)) is the image restored from the x-domain image generated by the generator F to the source visible light domain, and ||·||1 is the L1 norm. Step 5. Determine the perceptual loss according to the following formula to improve the perceptual quality of the generated image; In the formula, It is perceived loss. It uses a pre-trained neural network model to extract deep features of an image, where ||·|2 is the L2 norm. Step 6. Output the color target image.

2. The infrared image colorization method based on attention mask according to claim 1, characterized in that, In step 1.1, the first activation function is the ReLU activation function.

3. The infrared image colorization method based on attention mask according to claim 1, characterized in that, In step 1.2, the conversion layer consists of nine ResNET modules with the same structure. Each ResNET module is a five-layer convolutional neural network.

4. The infrared image colorization method based on attention mask according to claim 1, characterized in that, In step 1.3, the second activation function is the Tanh activation function.

5. The infrared image colorization method based on attention mask according to claim 1, characterized in that, In step 2.2, the content mask generator generates a content mask according to the following formula: In the formula, Here, tanh(·) is the content mask, tanh(·) is the activation process, and m is the feature map generated by the parameter-shared encoder. It is the weight of each position in the content mask. These are the coordinates of each position in the content mask, and n represents the number of masks generated.

6. The infrared image colorization method based on attention mask according to claim 5, characterized in that, In step 2.3, the attention mask generator generates a foreground attention mask and a background attention mask according to the following formula: In the formula, It is a foreground attention mask, and Softmax(·) is a channel-level function used for normalization. These are the weights at each location in the attention mask. These are the coordinates of each position in the attention mask, and n is the number of attention masks generated. It is a background attention mask.

7. The infrared image colorization method based on attention mask according to claim 1, characterized in that, In step 3, the foreground attention mask and background attention mask are fused with the corresponding content mask according to the following formula to obtain the target image. In the formula, G(X) is the target image.

Images