Infrared image generation method based on multi-scale perception

Abstract

The invention relates to an infrared image generation method based on multi-scale perception. Comprising the steps that image data are acquired and preprocessed, and the image data comprise a visible light image and an infrared image; constructing an infrared image generator and an infrared image discriminator; constructing a visible light image generator and a visible light image discriminator; a deep learning method is utilized to train an infrared image generator, an infrared image discriminator, a light image generator and a visible light image discriminator based on acquired image data. And determining the trained infrared image generator as an infrared image generation model. The method can generate a high-quality infrared image from a visible light image, and can be widely applied to the fields of night vision monitoring, remote sensing detection and the like.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and computer vision, and in particular relates to a method for generating an infrared image from a visible light image based on multi-scale perception. Background Art

[0002] Infrared images have a wide range of applications in night vision monitoring, medical imaging, remote sensing detection and other fields. However, obtaining high-quality infrared images usually requires expensive infrared imaging equipment, and is difficult to obtain due to environmental conditions. With the development of deep learning, the image style transfer method based on Generative Adversarial Network (GAN) provides a new idea for generating infrared images from visible light images. The traditional CycleGAN model can achieve conversion between image domains on unpaired datasets, but when processing complex scenes, the generated infrared images may have problems such as missing details and blurred textures. This is mainly due to the model's shortcomings in extracting and utilizing multi-scale features, and it cannot fully capture key information at different scales in the image. To solve the above problems, a model that can make full use of multi-scale features is needed to enhance the capture of global and local information of the image, thereby generating high-quality, detail-rich infrared images. Summary of the invention

[0003] In view of the shortcomings of the prior art, the present invention provides an infrared image generation method based on multi-scale perception. By introducing a multi-scale perception module into the discriminator of the CycleGAN model, the model's ability to learn features of different scales is improved, thereby guiding the infrared image generator to generate high-quality, detail-rich infrared images.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0005] A method for generating an infrared image based on multi-scale perception comprises the following steps:

[0006] 1) Obtain visible light image data and infrared image data and perform preprocessing to construct an image data training set;

[0007] 2) Building a CycleGAN model based on generative adversarial networks;

[0008] 3) Use the training set to train the CycleGAN model;

[0009] 4) Generate infrared images using the trained CycleGAN model.

[0010] The pre-processing is specifically as follows:

[0011] Perform geometric correction and consistency processing on the acquired images.

[0012] The CycleGAN model includes an infrared image generator, a visible light image generator, an infrared image discriminator and a visible light image discriminator, wherein a multi-scale perception module for acquiring and fusing image features of different scales is added between the backbone network and the output layer of the infrared image discriminator and the visible light image discriminator.

[0013] The multi-scale perception module includes a dimensionality reduction module, three parallel dilated convolution modules and a feature fusion module connected in sequence, wherein:

[0014] The convolution kernel size of the dimensionality reduction module is 1×1, the number of input channels and the number of output channels are both 128, and the stride is 1;

[0015] The convolution kernel size of the first dilated convolution module is 3×3, the dilation rate is 1, the number of input channels and the number of output channels are both 128, the stride is 1, and it is followed by the Leaky ReLU activation function and instance normalization;

[0016] The convolution kernel size of the second atrous convolution module is 3×3, the atrous rate is 3, the number of input channels and the number of output channels are both 128, the stride is 1, and it is followed by the Leaky ReLU activation function and instance normalization;

[0017] The convolution kernel size of the third atrous convolution module is 3×3, the atrous rate is 5, the number of input channels and the number of output channels are both 128, the stride is 1, and it is followed by a Leaky ReLU activation function and instance normalization.

[0018] The step 2) comprises the following steps:

[0019] 2.1) Input the visible light image data into the infrared image generator to generate the corresponding infrared style image;

[0020] 2.2) Using the infrared image discriminator to judge the input infrared image, wherein the input infrared image is a real infrared image in the training set or an infrared style image generated by the infrared image generator;

[0021] 2.3) Fix the infrared image discriminator model weights and update the infrared image generator parameters based on the adversarial loss function; Fix the infrared image generator model weights and update the infrared image discriminator parameters based on the adversarial loss function;

[0022] 2.4) Input the infrared image data into the visible light image generator to generate the corresponding visible light style image;

[0023] 2.5) judging the input visible light image based on the visible light image discriminator, wherein the input visible light image is a real visible light image in the training set or a visible light style image generated by the visible light image generator;

[0024] 2.6) Fix the weights of the visible light image discriminator model and update the parameters of the visible light image generator based on the adversarial loss function; Fix the weights of the visible light image generator model and update the parameters of the visible light image discriminator based on the adversarial loss function.

[0025] The adversarial loss function L GAN (G x ,D Y ,X,Y), specifically:

[0026]

[0027] Among them, X and Y represent visible light datasets or infrared datasets, x and y represent image samples in X and Y, and p data is the real image data distribution, G is the generator mapping from X to Y, and D Y It represents the discriminator that determines whether the image sample y comes from the data set Y, and E is the mathematical expectation.

[0028] The step 3) comprises the following steps:

[0029] 3.1) Inputting the visible light image data into the infrared image generator to generate a pseudo infrared image corresponding to the infrared style, and inputting the generated pseudo infrared image into the visible light image generator to generate a pseudo visible light image corresponding to the visible light style;

[0030] 3.2) Based on the cycle consistency loss function and the real visible light image and the pseudo visible light image, the loss is calculated and the parameters of the visible light image generator and the infrared image generator are updated;

[0031] 3.3) Inputting the infrared image data into a visible light image generator to generate a pseudo visible light image corresponding to the visible light style, and inputting the generated pseudo visible light image into an infrared image generator to generate a pseudo infrared image corresponding to the infrared style;

[0032] 3.4) Based on the cycle consistency loss function and the real infrared image and the pseudo infrared image, the loss is calculated and the parameters of the visible light image generator and the infrared image generator are updated.

[0033] The cycle consistency loss function L cyc (X,Y), specifically:

[0034]

[0035] Among them, ‖·‖1 is the 1-norm.

[0036] An infrared image generation system based on multi-scale perception, comprising:

[0037] A training set building module is used to acquire visible light image data and infrared image data and perform preprocessing to build an image data training set;

[0038] CycleGAN model building module, used to build CycleGAN model based on generative adversarial network;

[0039] Model training module, used to train the CycleGAN model using the training set;

[0040] Image generation module, used to generate infrared images using the trained CycleGAN model.

[0041] The present invention has the following beneficial effects and advantages:

[0042] 1. By introducing a multi-scale perception mechanism, the model’s ability to learn features of different scales is enhanced, and the detail expression and texture fidelity of the generated images are improved.

[0043] 2. The model does not require paired visible light and infrared image data for training, which reduces the difficulty of data acquisition and expands the scope of application of the model.

[0044] 3. A general infrared image generation method is provided, which can be widely used in night vision monitoring, medical imaging, remote sensing detection and other fields, and has good application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Build a schematic for your model.

[0046] Figure 2 This is the structural diagram of the infrared image generator.

[0047] Figure 3 This is the structure diagram of the infrared image discriminator.

[0048] Figure 4 This is the structure diagram of the multi-scale perception module.

[0049] Figure 5 This is the structure diagram of the visible light image generator.

[0050] Figure 6 This is the structure diagram of the visible light image discriminator. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the invention specification herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0053] A method for generating an infrared image based on multi-scale perception comprises the following steps:

[0054] Step 1: Obtain visible light image data and infrared image data as model training sets, and preprocess the data including geometric correction and data consistency processing.

[0055] Step 2: Construct a CycleGAN model including an infrared image generator, a visible light image generator, an infrared image discriminator, and a visible light image discriminator, and introduce a multi-scale perception module into the infrared image discriminator and the visible light image discriminator to obtain and fuse image features of different scales;

[0056] Step 2: Input the visible light image data into the infrared image generator to generate the corresponding infrared style image.

[0057] Step 3: Determine the input infrared image based on the infrared image discriminator. The input infrared image is a real infrared image in the training set or a pseudo infrared image generated by the infrared image generator in step 1.

[0058] Step 4: Fix the infrared image discriminator model weights and update the infrared image generator parameters based on the preset loss function. Fix the infrared image generator model weights and update the infrared image discriminator parameters based on the preset loss function.

[0059] Step 5: Input the infrared image data into the visible light image generator to generate the corresponding visible light style image.

[0060] Step 6: Determine the input visible light image based on the visible light image discriminator. The input visible light image is a real visible light image in the training set or a pseudo visible light image generated by the visible light image generator in step 5.

[0061] Step 7: Fix the visible light image discriminator model weights and update the parameters of the visible light image generator based on the preset loss function. Fix the visible light image generator model weights and update the parameters of the visible light image discriminator based on the preset loss function.

[0062] Step 8: Input the visible light image data into the infrared image generator to generate a pseudo infrared image of the corresponding infrared style, and input the generated pseudo infrared image into the visible light image generator to generate a pseudo visible light image of the corresponding visible light style. Based on the preset loss function and the real visible light image and the pseudo visible light image, calculate the loss and update the parameters of the visible light image generator and the infrared image generator.

[0063] Step 9: Input the infrared image data into the visible light image generator to generate a pseudo visible light image corresponding to the visible light style, and input the generated pseudo visible light image into the infrared image generator to generate a pseudo infrared image corresponding to the infrared style. Based on the preset loss function and the real infrared image and the pseudo infrared image, calculate the loss and update the parameters of the visible light image generator and the infrared image generator.

[0064] like Figure 1 The following is a schematic diagram of the model construction:

[0065] Data acquisition: Acquire multiple visible light images and infrared images, where the visible light images and infrared images can be taken using a camera and an infrared imaging device without pairing.

[0066] Infrared image generator: Figure 2 The figure shows the structure of the infrared image generator, which maps the visible light image to the corresponding infrared style image. Its structure includes: input layer, convolution downsampling layer, deconvolution upsampling layer and output layer. The input layer input is a three-channel RGB image. The convolution downsampling layer includes three convolution layers. The convolution kernel size of the first convolution layer is 7×7, the stride is 1, and the output channel is 64. The convolution kernel size used in the next two convolution layers is 3×3, the stride is 2, and the number of output channels is 128 and 256 respectively (each layer halves the spatial size of the feature map). All the above convolution layers are followed by instance normalization and ReLU activation function. The deconvolution upsampling layer includes two deconvolution layers, each with a stride of 2, which upsamples the feature map. The convolution kernel size of the deconvolution layer is 3×3, which restores the feature map size to the same size as the input image. The first deconvolution layer upsamples the feature map from 256 channels to 128 channels, and the second deconvolution layer upsamples from 128 channels to 64 channels. All the above deconvolution layers are followed by instance normalization and ReLU activation function. The output layer uses a 7×7 convolution kernel with a stride of 1, and uses the tanh activation function to limit the pixel values ​​of the output image to [-1,1], outputting a single-channel pseudo infrared image.

[0067] The infrared image generator inputs the visible light image into the infrared image generator to obtain an infrared image of the infrared image style corresponding to the visible light image.

[0068] Infrared image discriminator: Figure 3The figure shows the structure diagram of the infrared image discriminator, which determines whether the input image is a real infrared image or a pseudo infrared image generated by the infrared image generator. Its structure includes: input layer, backbone network, multi-scale perception module, and output layer. The input layer inputs a single-channel grayscale image. The backbone network includes four convolutional layers, of which the first layer has a convolution kernel size of 4×4, 64 channels, and a stride of 2, followed by a Leaky ReLU activation function. The second layer has a convolution kernel size of 4×4, 128 channels, and a stride of 2, followed by a Leaky ReLU activation function and instance normalization. The third layer has a convolution kernel size of 4×4, 256 channels, and a stride of 2, followed by a Leaky ReLU activation function and instance normalization. The fourth layer has a convolution kernel size of 4×4, 512 channels, and a stride of 1, followed by a Leaky ReLU activation function and instance normalization. As shown Figure 4 The figure shows the structure of the multi-scale perception module, including: dimensionality reduction operation, three parallel dilated convolution modules and feature fusion operation. The convolution kernel size used in the dimensionality reduction operation is 1×1, the input channel is 128, the output channel is 128, the stride is 1, followed by the LeakyReLU activation function and instance normalization. The three parallel dilated convolution modules are:

[0069] 1) Convolution kernel size 3×3, dilation rate 1, input channels 128, output channels 128, stride 1, followed by Leaky ReLU activation function and instance normalization;

[0070] 2) Convolution kernel size 3×3, dilation rate 3, input channels 128, output channels 128, stride 1, followed by Leaky ReLU activation function and instance normalization;

[0071] 3) Convolution kernel size 3×3, dilation rate 5, input channels 128, output channels 128, stride 1, followed by Leaky ReLU activation function and instance normalization;

[0072] The feature fusion operation fuses the outputs of three parallel dilated convolution modules together through feature addition operation to obtain a feature map with 128 channels. The output layer includes two convolution layers with parameters of convolution kernel size 4×4, number of channels 512, stride 1, followed by Leaky ReLU activation function and instance normalization, and convolution kernel size 4×4, number of channels 1, stride 1, and output true and false discrimination matrix.

[0073] Visible light image generator: Figure 5The figure shows the structure of the visible light image generator, which maps the infrared image to the image of the corresponding visible light style. Its structure includes: input layer, convolution downsampling layer, deconvolution upsampling layer and output layer. The input layer input is a single-channel grayscale image. The convolution downsampling layer includes three convolution layers. The convolution kernel size of the first convolution layer is 7×7, the stride is 1, and the output channel is 64. The convolution kernel size used in the next two convolution layers is 3×3, the stride is 2, and the number of output channels is 128 and 256 respectively (each layer halves the spatial size of the feature map). All the above convolution layers are followed by instance normalization and ReLU activation function. The deconvolution upsampling layer includes two deconvolution layers, each with a stride of 2, which upsamples the feature map. The convolution kernel size of the deconvolution layer is 3×3, which restores the feature map size to the same size as the input image. The first deconvolution layer upsamples the feature map from 256 channels to 128 channels, and the second deconvolution layer upsamples from 128 channels to 64 channels. All the above deconvolution layers are followed by instance normalization and ReLU activation function. The output layer uses a 7×7 convolution kernel with a stride of 1, and uses the tanh activation function to limit the pixel values ​​of the output image to [-1,1], outputting a three-channel RGB pseudo visible light image.

[0074] The visible light image generator inputs the infrared image into the visible light image generator to obtain a visible light image of the style of the visible light image corresponding to the infrared image.

[0075] The infrared image discriminator discriminates the real infrared image in the sample set and the pseudo infrared image generated by the infrared image generator to obtain authenticity discrimination results of the real infrared image and the generated pseudo infrared image, including: dividing the real infrared image and the pseudo infrared image into a plurality of sub-images; based on the infrared image discriminator, making authenticity judgment on each sub-image; and determining the final judgment result according to the judgment results of all sub-images of the real infrared image or the pseudo infrared image.

[0076] Visible light image discriminator: Figure 6The figure shows the structure of the visible light image discriminator, which determines whether the input image is a real visible light image or a pseudo visible light image generated by the visible light image generator. Its structure includes: input layer, backbone network, multi-scale perception module, and output layer. The input layer inputs a single-channel grayscale image. The backbone network includes four convolutional layers, of which the first layer has a convolution kernel size of 4×4, 64 channels, and a stride of 2, followed by a Leaky ReLU activation function. The second layer has a convolution kernel size of 4×4, 128 channels, and a stride of 2, followed by a Leaky ReLU activation function and instance normalization. The third layer has a convolution kernel size of 4×4, 256 channels, and a stride of 2, followed by a Leaky ReLU activation function and instance normalization. The fourth layer has a convolution kernel size of 4×4, 512 channels, and a stride of 1, followed by a Leaky ReLU activation function and instance normalization. The multi-scale perception module includes: dimensionality reduction operation, three parallel dilated convolution modules and feature fusion operation. The convolution kernel size used in the dimensionality reduction operation is 1×1, the input channel is 128, the output channel is 128, the stride is 1, followed by the Leaky ReLU activation function and instance normalization. The three parallel dilated convolution modules are:

[0077] 1) Convolution kernel size 3×3, dilation rate 1, input channels 128, output channels 128, stride 1, followed by LeakyReLU activation function and instance normalization;

[0078] 2) Convolution kernel size 3×3, dilation rate 3, input channels 128, output channels 128, stride 1, followed by LeakyReLU activation function and instance normalization;

[0079] 3) Convolution kernel size 3×3, dilation rate 5, input channels 128, output channels 128, stride 1, followed by LeakyReLU activation function and instance normalization;

[0080] The feature fusion operation fuses the outputs of three parallel dilated convolution modules together through feature addition operation to obtain a feature map with 128 channels. The output layer includes two convolution layers with parameters of convolution kernel size 4×4, number of channels 512, stride 1, followed by Leaky ReLU activation function and instance normalization, and convolution kernel size 4×4, number of channels 1, stride 1, and output true and false discrimination matrix.

[0081] The visible light image discriminator discriminates the real visible light image in the sample set and the pseudo visible light image generated by the visible light image generator to obtain authenticity discrimination results of the real visible light image and the generated pseudo visible light image, including: dividing the real visible light image and the pseudo visible light image into a plurality of sub-images; based on the visible light image discriminator, making authenticity judgment on each sub-image; and determining the final judgment result according to the judgment results of all sub-images of the visible light infrared image or the pseudo visible light image.

[0082] During model training, first fix the infrared image generator parameters, input the visible light image into the infrared image generator to output a pseudo infrared image, then input the pseudo infrared image and the real infrared image into the infrared image discriminator, and update the infrared image discriminator parameters according to the preset loss function. Then fix the infrared image discriminator parameters, and update the infrared image generator parameters according to the discrimination result of the pseudo infrared image. Then input the infrared image into the visible light image generator to output a pseudo visible light image, then input the pseudo visible light image and the real visible light image into the visible light image discriminator, and update the visible light image discriminator parameters according to the preset loss function. Then fix the visible light image discriminator parameters, and update the visible light image generator parameters according to the discrimination result of the pseudo visible light image. Output the pseudo infrared image through the infrared image generator, then input the pseudo infrared image into the visible light image generator to obtain a cyclic pseudo visible light image, calculate the loss between the cyclic pseudo visible light image and the real visible light image, and update the parameters of the visible light image generator and the infrared image generator. Similarly, the real infrared image is outputted as a pseudo visible light image through the visible light generator, and then the pseudo visible light image is inputted into the infrared image generator to obtain a cyclic pseudo infrared image. The loss between the cyclic pseudo infrared image and the real infrared image is calculated, and the parameters of the visible light image generator and the infrared image generator are updated.

[0083] The selected loss functions include: adversarial loss function and cycle consistency loss function. The adversarial loss function formula is:

[0084]

[0085] Among them, X and Y represent visible light datasets or infrared datasets, x and y represent image samples in X and Y, and p data is the real image data distribution, G is the generator mapping from X to Y, and D Y It represents the discriminator that determines whether the image sample y comes from the data set Y, and E is the mathematical expectation. The formula of the cycle consistency loss function is:

[0086]

[0087] Among them, ‖·‖1 is the 1-norm.

[0088] After multiple rounds of the above training process, the infrared image generator obtained is an infrared image generation model.

[0089] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A method for generating infrared images based on multi-scale perception, characterized in that: The following steps are involved: 1) Obtain visible light image data and infrared image data and perform preprocessing to construct an image data training set; 2) Building a CycleGAN model based on generative adversarial networks; 3) Use the training set to train the CycleGAN model; 4) Generate infrared images using the trained CycleGAN model.

2. The infrared image generation method based on multi-scale perception according to claim 1 is characterized in that: The pre-processing is specifically as follows: Perform geometric correction and consistency processing on the acquired images.

3. The infrared image generation method based on multi-scale perception according to claim 1, characterized in that: The CycleGAN model includes an infrared image generator, a visible light image generator, an infrared image discriminator and a visible light image discriminator, wherein a multi-scale perception module for acquiring and fusing image features of different scales is added between the backbone network and the output layer of the infrared image discriminator and the visible light image discriminator.

4. The infrared image generation method based on multi-scale perception according to claim 1, characterized in that: The multi-scale perception module includes a dimensionality reduction module, three parallel dilated convolution modules and a feature fusion module connected in sequence, wherein: The convolution kernel size of the dimensionality reduction module is 1×1, the number of input channels and the number of output channels are both 128, and the stride is 1; The convolution kernel size of the first dilated convolution module is 3×3, the dilation rate is 1, the number of input channels and the number of output channels are both 128, the stride is 1, and it is followed by the Leaky ReLU activation function and instance normalization; The convolution kernel size of the second atrous convolution module is 3×3, the atrous rate is 3, the number of input channels and the number of output channels are both 128, the stride is 1, and it is followed by the Leaky ReLU activation function and instance normalization; The convolution kernel size of the third atrous convolution module is 3×3, the atrous rate is 5, the number of input channels and the number of output channels are both 128, the stride is 1, and it is followed by a Leaky ReLU activation function and instance normalization.

5. The infrared image generation method based on multi-scale perception according to claim 1, characterized in that: The step 2) comprises the following steps: 2.1) Input the visible light image data into the infrared image generator to generate the corresponding infrared style image; 2.2) Using the infrared image discriminator to judge the input infrared image, wherein the input infrared image is a real infrared image in the training set or an infrared style image generated by the infrared image generator; 2.3) Fix the infrared image discriminator model weights and update the infrared image generator parameters based on the adversarial loss function; Fix the infrared image generator model weights and update the infrared image discriminator parameters based on the adversarial loss function; 2.4) Input the infrared image data into the visible light image generator to generate the corresponding visible light style image; 2.5) judging the input visible light image based on the visible light image discriminator, wherein the input visible light image is a real visible light image in the training set or a visible light style image generated by the visible light image generator; 2.6) Fix the weights of the visible light image discriminator model and update the parameters of the visible light image generator based on the adversarial loss function; Fix the weights of the visible light image generator model and update the parameters of the visible light image discriminator based on the adversarial loss function.

6. The infrared image generation method based on multi-scale perception according to claim 5, characterized in that: The adversarial loss function L GAN (G x ,D Y ,X,Y), specifically: Among them, X and Y represent visible light datasets or infrared datasets, x and y represent image samples in X and Y, and p data is the real image data distribution, G is the generator mapping from X to Y, and D Y It represents the discriminator that determines whether the image sample y comes from the data set Y, and E is the mathematical expectation.

7. The infrared image generation method based on multi-scale perception according to claim 1, characterized in that: The step 3) comprises the following steps: 3.1) Inputting the visible light image data into the infrared image generator to generate a pseudo infrared image corresponding to the infrared style, and inputting the generated pseudo infrared image into the visible light image generator to generate a pseudo visible light image corresponding to the visible light style; 3.2) Based on the cycle consistency loss function and the real visible light image and the pseudo visible light image, the loss is calculated and the parameters of the visible light image generator and the infrared image generator are updated; 3.3) Inputting the infrared image data into a visible light image generator to generate a pseudo visible light image corresponding to the visible light style, and inputting the generated pseudo visible light image into an infrared image generator to generate a pseudo infrared image corresponding to the infrared style; 3.4) Based on the cycle consistency loss function and the real infrared image and the pseudo infrared image, the loss is calculated and the parameters of the visible light image generator and the infrared image generator are updated.

8. The infrared image generation method based on multi-scale perception according to claim 7 is characterized in that: The cycle consistency loss function L cyc (X,Y), specifically: Among them, ‖·‖1 is the 1-norm.

9. An infrared image generation system based on multi-scale perception, characterized in that: include: A training set building module is used to acquire visible light image data and infrared image data and perform preprocessing to build an image data training set; CycleGAN model building module, used to build CycleGAN model based on generative adversarial network; Model training module, used to train the CycleGAN model using the training set; Image generation module, used to generate infrared images using the trained CycleGAN model.

Images