Method and system for generating infrared image from visible light image

Abstract

The invention provides a method and system for generating an infrared image from a visible light image, and belongs to the technical field of image generation, and the method comprises the steps: constructing pairing input data composed of the infrared image and the visible light image; constructing a semantic segmentation model, and inputting the visible light image in the paired input data into the semantic segmentation model to obtain semantic features of the visible light image; constructing a generator for generating an infrared image, inputting the visible light image in the paired input data, and decoding to obtain the generated infrared image; constructing a discriminator and a loss function, and inputting the infrared image in the paired input data into the discriminator and the loss function to obtain an infrared image generation model; and obtaining a real visible light image, and inputting the real visible light image into the final infrared image generation model, thereby obtaining an infrared image corresponding to the real visible light image.

Description

Technical Field

[0001] The present application belongs to the field of image generation technology, and in particular relates to a method and system for generating infrared images from visible light images. Background Art

[0002] Infrared target images are very important for military purposes, but they are relatively scarce. To address the problem of the lack of infrared target images, most existing technologies use data augmentation technology based on generative adversarial network models to process visible light images to generate near-infrared images, thereby expanding the number of infrared samples. However, in generative adversarial networks, Pix2Pix and CycleGAN provide a general network framework for converting visible light images to infrared images, but the infrared images generated by these general methods have problems with unclear textures and missing structures to varying degrees. Summary of the invention

[0003] The purpose of the present application is to provide a method and system for generating an infrared image from a visible light image, so as to solve or alleviate at least one problem in the background technology.

[0004] On the one hand, the technical solution of the present application is: a method for generating an infrared image from a visible light image, comprising: Construct paired input data consisting of infrared images and visible light images with the same image size and a quantity ratio of 1:1; Construct a semantic segmentation model, and input the visible light image in the paired input data into the semantic segmentation model to obtain the semantic features of the visible light image; A generator for generating an infrared image is constructed, wherein the generator includes an encoder, a decoder, and a dilated convolution model. A visible light image in paired input data is input into the encoder for feature extraction to obtain feature encoding of the visible light image. The semantic features of the visible light image and the feature encoding are then merged in the dilated convolution model, and the generated infrared image is obtained after being decoded by the decoder. Construct a discriminator and a loss function, input the infrared image in the paired input data into the discriminator and the loss function, the discriminator and the loss function perform loss discrimination on the infrared image in the paired input data and the infrared image generated by the generator, and then provide feedback to optimize the generator, thereby obtaining an infrared image generation model; A real visible light image is obtained, and the real visible light image is input into the infrared image generation model, so as to obtain an infrared image corresponding to the real visible light image.

[0005] In a preferred embodiment of the present application, the process of constructing paired input data is: A drone equipped with an infrared sensor and a visible light sensor is used to collect infrared and visible light video sequences at different times and in different scenes. After adjusting the sizes of the infrared and visible light video sequences to be consistent, image pairing is performed to construct paired input data of 1:1 paired infrared images and visible light images.

[0006] In a preferred embodiment of the present application, the semantic segmentation model is a UNet network with a depth of 5. The UNet network includes 5 encoding blocks and 5 decoding blocks. Each encoding block is composed of a downsampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the downsampling convolution layers corresponding to the 5 encoding blocks are [64, 128, 256, 512, 1024], respectively. Each decoding block is composed of a deconvolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the deconvolution layers corresponding to the 5 decoding blocks are [512, 256, 128, 64, 3], respectively.

[0007] In a preferred embodiment of the present application, the encoder of the generator is an encoder based on a deep residual network, which includes multiple convolution modules and residual modules. The encoder of the deep residual network first performs a convolution operation on the input visible light image, where the convolution kernel size is 7×7, the step size is 2, and the padding size is 3×3; then, a four-layer residual network based on ResNet34 is used as a pre-trained network structure, and a jump connection is used to enhance the generalization representation ability of the regional block. The visible light image of the paired input data is input into the constructed encoder to obtain the feature encoding of the visible light image, including shallow features and underlying features.

[0008] In a preferred embodiment of the present application, the dilated convolution model of the generator is a serial and parallel connection network. The dilated convolution model fuses the semantic features of the visible light image with the feature coding to obtain the spatial feature information of the visible light image, and inputs it into a subsequent decoder.

[0009] In a preferred embodiment of the present application, the decoder of the generator adopts a bottleneck connection structure of a residual network, uses a 1×1 convolution kernel to improve the computational efficiency of the network, uses transposed convolution upsampling to increase the side length to 32 times the original, and restores the size of the original visible light image to obtain the generated infrared image.

[0010] In a preferred embodiment of the present application, the discriminator is composed of a five-layer convolutional network. The discriminator first performs five convolution operations on the infrared image generated by the generator to obtain a 30×30 pixel feature map, and then inputs the obtained feature map into the cross entropy function to determine whether it conforms to the target image distribution.

[0011] In a preferred embodiment of the present application, the loss function includes a generative adversarial loss function, an L2 loss function, and a semantic segmentation loss function. The generated adversarial loss function is: {min}_{G}{max}_{D}{L}_{adv}(G,D)=\sum{[logD(y)]}+\sum{[1-logD(G(x))]} In the formula, G For the generator, D is the discriminator, x is a real visible light image, y is a real infrared image. G(x) Infrared images generated for adversarial networks; The L2 loss function is: In the formula, G For the generator, x is a real visible light image, G(x) Infrared image generated for the adversarial network.

[0012] The semantic segmentation loss function is: {L}^{r}_{seg}={E}_{{x,c}^{s}}[-{c}^{s}log(S(x))] in, is the scene category, S represents the semantic encoding branch, and x is the real visible light image.

[0013] On the other hand, the technical solution provided by the present application is: a system for generating an infrared image from a visible light image, comprising: A data input module is used to construct paired input data consisting of infrared images and visible light images with the same image size and a quantity ratio of 1:1; A semantic segmentation model, used for inputting the visible light image in the paired input data into the semantic segmentation model to obtain semantic features of the visible light image; A generator, for generating an infrared image, comprising an encoder, a decoder and a dilated convolution model, wherein a visible light image in paired input data is input into the encoder for feature extraction to obtain feature encoding of the visible light image, and then the semantic features of the visible light image and the feature encoding are merged in the dilated convolution model, and the generated infrared image is obtained after being decoded by the decoder; A discriminator and loss function module, wherein the infrared image in the paired input data is input into the discriminator and loss function module, and the discriminator and loss function module performs loss discrimination on the infrared image in the paired input data and the infrared image generated by the generator, and then feeds back to optimize the generator, thereby obtaining an infrared image generation model; The infrared image generation model is used to obtain a real visible light image. The real visible light image is input into the infrared image generation model to obtain an infrared image corresponding to the real visible light image.

[0014] In a preferred embodiment of the present application, the process of constructing the paired input data by the data input module is as follows: An unmanned aerial vehicle with an infrared sensor and a visible light sensor is used to collect infrared and visible light video sequences at different times and in different scenes. After adjusting the sizes of the infrared and visible light video sequences to be consistent, image pairing is performed to construct paired input data of paired infrared images and visible light images with consistent image sizes and a quantity ratio of 1:1.

[0015] In a preferred embodiment of the present application, the semantic segmentation model is a UNet network with a depth of 5, which includes 5 encoding blocks and 5 decoding blocks. Each encoding block is composed of a downsampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the downsampling convolution layer corresponding to the 5 encoding blocks are [64, 128, 256, 512, 1024] respectively; each decoding block is composed of a deconvolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the deconvolution layer corresponding to the 5 decoding blocks are [512, 256, 128, 64, 3] respectively.

[0016] In a preferred embodiment of the present application, the encoder of the generator is an encoder based on a deep residual network, which includes multiple convolution modules and residual modules. The encoder of the deep residual network first performs a convolution operation on the input visible light image, where the convolution kernel size is 7×7, the step size is 2, and the padding size is 3×3; then, a four-layer residual network based on ResNet34 is used as a pre-trained network structure, and a jump connection is used to enhance the generalization representation ability of the regional block. The visible light image of the paired input data is input into the constructed encoder to obtain the feature encoding of the visible light image, including shallow features and underlying features.

[0017] In a preferred embodiment of the present application, the dilated convolution model of the generator is a serial and parallel connection network. The dilated convolution model fuses the semantic features of the visible light image with the feature coding to obtain the spatial feature information of the visible light image, and inputs it into a subsequent decoder.

[0018] In a preferred embodiment of the present application, the decoder of the generator adopts a bottleneck connection structure of a residual network, uses a 1×1 convolution kernel to improve the computational efficiency of the network, uses transposed convolution upsampling to increase the side length to 32 times the original, and restores the size of the original visible light image to obtain the generated infrared image.

[0019] In a preferred embodiment of the present application, the discriminator is composed of a five-layer convolutional network. The discriminator first performs five convolution operations on the infrared image generated by the generator to obtain a 30×30 pixel feature map, and then inputs the obtained feature map into the cross entropy function to determine whether it conforms to the target image distribution.

[0020] In a preferred embodiment of the present application, the loss function module includes a generative adversarial loss function module, an L2 loss function module and a semantic segmentation loss function module. The generation of adversarial loss function module is performed as follows: {min}_{G}{max}_{D}{L}_{adv}(G,D)=\sum{[logD(y)]}+\sum{[1-logD(G(x))]} In the formula, G For the generator, D is the discriminator, x is a real visible light image, y is a real infrared image. G(x) Infrared images generated for adversarial networks; The L2 loss function module is performed as follows: In the formula, x is a real visible light image, G For the generator, G(x) Infrared images generated for the Generative Adversarial Network; The semantic segmentation loss function module is performed as follows: {L}^{r}_{seg}={E}_{{x,c}^{s}}[-{c}^{s}log(S(x))] in, is the scene category, S represents the semantic encoding branch, x It is a real visible light image.

[0021] In a third aspect, the technical solution provided by the present application is: an electronic device, comprising: a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the method of generating an infrared image from a visible light image as described in any of the above items.

[0022] The last aspect is a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for generating an infrared image from a visible light image as described in any of the above items is implemented.

[0023] The method and system for generating infrared images from visible light images provided in the present application adopt a segmentation-guided network to learn the data distribution mapping between visible light images and infrared images, realize migration from the visible light domain to the infrared domain, and achieve the purpose of infrared data augmentation while ensuring that the overall image content features remain unchanged; by adding a semantic segmentation model, adding scene semantic features, designing a semantic segmentation loss function, and semantically constraining the network training process, it is ensured that the expanded infrared image has the same scene information as the original infrared image. The receptive field of the network is enlarged by a generator with a hole convolution, ensuring that the generated infrared image has a high degree of authenticity in local details and is consistent with the texture details of the real visible light image. Therefore, the expansion of small samples of infrared images can be achieved by migrating from the visible light domain to the infrared domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution provided by the present application, the following is a brief introduction to the accompanying drawings. Obviously, the accompanying drawings described below are only some embodiments of the present application.

[0025] Figure 1 A schematic diagram of the method for generating an infrared image from a visible light image of the present application.

[0026] Figure 2 A schematic diagram showing the comparison between a visible light image, a generated infrared image and a real infrared image according to an embodiment of the present application.

[0027] Figure 3 A diagram showing the composition of a system for generating infrared images from visible light images of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the implementation of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below in conjunction with the drawings in the embodiments of this application.

[0029] In order to compensate for the number of infrared image sample libraries and improve the clarity / realism of infrared images, the present application provides a method for converting visible light images into infrared images.

[0030] like Figure 1 As shown, the method for generating an infrared image from a visible light image provided by the present application includes the following process: Step S100 , constructing paired input data consisting of an infrared image and a visible light image with the same image size and a quantity ratio of 1:1.

[0031] In some embodiments of the present application, a drone with an infrared sensor and a visible light sensor is used to collect infrared and visible light video sequences at different times and in different scenes. After adjusting the sizes of the infrared and visible light video sequences to be consistent, image pairing is performed to construct 1:1 paired input data of infrared images and visible light images. Afterwards, the paired input data can be divided according to a certain ratio (for example, 7:3 or 8:2) to obtain a training data set and a test data set. For example, in this embodiment of the present application, 6,000 paired input data of infrared images and visible light images are selected as the training data set, and 1,000 paired input data of infrared images and visible light images are selected as the test data set.

[0032] Step S200: construct a semantic segmentation model, and input the visible light image in the paired input data into the semantic segmentation model to obtain semantic features of the visible light image.

[0033] In this embodiment of the present application, the semantic segmentation model is a UNet network with a depth of 5, and the UNet network includes 5 encoding blocks and 5 decoding blocks. Each encoding block consists of a downsampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the downsampling convolution layer corresponding to the 5 encoding blocks are [64, 128, 256, 512, 1024] respectively. Each decoding block consists of a deconvolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the deconvolution layer corresponding to the 5 decoding blocks are [512, 256, 128, 64, 3] respectively.

[0034] Step 300, construct a generator for generating an infrared image, which includes an encoder, a decoder, and a dilated convolution model. The visible light image in the paired input data is input into the encoder for feature extraction to obtain feature encoding of the visible light image. The semantic features of the visible light image and the feature encoding are then merged in the dilated convolution model, and the generated infrared image is obtained after decoding by the decoder.

[0035] In some embodiments of the present application, the encoder of the generator is an encoder based on a deep residual network (ResNet), which includes multiple convolution modules and residual modules. The encoder of the deep residual network first performs a convolution operation on the input visible light image, where the convolution kernel size is 7×7, the step size is 2, and the padding size is 3×3. Afterwards, the residual module uses a four-layer residual network based on ResNet34 (the four-layer residual networks are ResNet layer1, ResNetlayer2, ResNet layer3 and ResNe layer4) as the pre-trained network structure, and uses jump connections to enhance the generalization representation ability of the regional blocks. The visible light image of the paired input data is input into the constructed encoder to obtain the feature encoding of the visible light image, including shallow features and underlying features.

[0036] In some embodiments of the present application, the dilated convolution model of the generator is a serial and parallel connection network. The dilated convolution model is used to enhance the network recognition ability, expand the receptive field, and fuse multi-scale information. The dilated convolution model fuses the semantic features of the visible light image with the feature coding to obtain the spatial feature information of the visible light image and input it into the subsequent decoder.

[0037] In a preferred embodiment of the present application, the atrous convolution model is stacked in a cascade mode, which consists of three cascades of atrous convolutions with gradually increasing numbers. The convolution rates of each part are [1, 2, 4], and the receptive fields are [3, 7, 9], respectively, to cover the output infrared image.

[0038] In this application, the calculation formula for the size of the receptive field of the dilated convolution is: ; in, r is the dilation rate of the dilated convolution, f s is the size of the convolution kernel, and y is the size of the receptive field. By fusing dilated convolutions with different expansion rates, the dilated convolution cascade mode can collect dense pixel attention information in the encoder output, thereby extracting target features of different sizes.

[0039] In some embodiments of the present application, the decoder of the generator adopts a bottleneck connection structure of a residual network, which uses a 1×1 convolution kernel to improve the computational efficiency of the network, and uses transposed convolution upsampling to increase the side length to 32 times the original size, restoring the size of the original visible light image to obtain the generated infrared image.

[0040] Step 400, construct a discriminator and a loss function, wherein the loss function includes a generative adversarial loss function, an L2 loss function, and a semantic segmentation loss function. By inputting the infrared image in the paired input data into the discriminator and the loss function, the discriminator and the loss function perform loss discrimination on the infrared image in the paired input data and the infrared image generated by the generator, and then feed back to optimize the generator, thereby obtaining an infrared image generation model.

[0041] In this application, the discriminator is composed of a five-layer convolutional network. The discriminator first performs five convolution operations on the infrared image generated by the generator to obtain a 30×30 pixel feature map, and then inputs the obtained feature map into the cross entropy function to determine whether it conforms to the target image distribution.

[0042] In one embodiment of the present application, the discriminator includes 6 down-sampling convolution blocks and 2 output branches, each of which is composed of a down-sampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, and a Relu activation layer. The output channels of the 6 down-sampling convolution blocks are [64, 128, 256, 512, 1024, 2048], respectively. The "true / false" discrimination output branch is composed of a convolution layer with a convolution kernel size of 3, a step size of 2, and a padding size of 1.

[0043] In this application, the overall network is optimized by constructing a variety of loss functions to work together, and the generator is feedback optimized and iterated by generating adversarial loss function, L2 loss function and semantic segmentation loss function to train the overall generative adversarial network and obtain the final infrared image generation model.

[0044] Among them, the generative adversarial loss function can enable the generator network to learn the distribution of the target domain image. In this application, the generative adversarial loss function is: {min}_{G}{max}_{D}{L}_{adv}(G,D)=\sum{[logD(y)]}+\sum{[1-logD(G(x))]} ; In the formula, G For the generator, D is the discriminator, x is a real visible light image, y is a real infrared image, G(x) is an infrared image generated by the adversarial network, and the network is able to learn the distribution of the target domain image by generating adversarial losses.

[0045] The L2 loss function can make the generated infrared image as close as possible to the real infrared image. In this application, the L2 loss function is: ; In the formula, G For the generator,x is a real visible light image, G(x) Infrared image generated for the adversarial network.

[0046] The semantic segmentation loss function can make the generated infrared image more accurate in detail expression and local mapping. In this application, the semantic segmentation loss function is: {L}^{r}_{seg}={E}_{{x,c}^{s}}[-{c}^{s}log(S(x))] in, is the scene category, S represents the semantic encoding branch, and x is the real visible light image. For the training of the semantic segmentation branch, a labeled supervised learning method is adopted, and the cross entropy loss function is used to train the network.

[0047] Step 500: Acquire a real visible light image, and input the real visible light image into the above infrared image generation model, so as to obtain an infrared image corresponding to the real visible light image.

[0048] like Figure 2 As shown, in this embodiment of the present application, the real visible light image is tested using the visible light image in the test data set in step S10, and the infrared image in the test data set is only used as a reference for comparison, and does not participate in the correction and optimization of the above-mentioned model or network. In actual use, the real visible light image is the visible light image that needs to be converted.

[0049] The method for generating infrared images from visible light images provided in the present application adopts a segmentation-guided network to learn the data distribution mapping between visible light images and infrared images, realizes migration from the visible light domain to the infrared domain, and achieves the purpose of infrared data augmentation while ensuring that the overall image content features remain unchanged; by adding a semantic segmentation model, adding scene semantic features, designing a semantic segmentation loss function, and semantically constraining the network training process, it is ensured that the expanded infrared image has the same scene information as the original infrared image. The receptive field of the network is enlarged by a generator with a hole convolution, ensuring that the generated infrared image has a high degree of authenticity in local details and is consistent with the texture details of the real visible light image, so that the expansion of small samples of infrared images can be achieved by migrating from the visible light domain to the infrared domain.

[0050] On this basis, the present application also provides a system for generating infrared images from visible light images, such as Figure 3 As shown, the system includes: The data input module 11 is used to construct paired input data consisting of infrared images and visible light images with the same image size and a quantity ratio of 1:1; A semantic segmentation model 12, used for inputting the visible light image in the paired input data into the semantic segmentation model to obtain semantic features of the visible light image; A generator 13 is used to generate an infrared image. The generator includes an encoder, a decoder, and a dilated convolution model. The visible light image in the paired input data is input into the encoder for feature extraction to obtain feature encoding of the visible light image. The semantic features of the visible light image and the feature encoding are then merged in the dilated convolution model, and the generated infrared image is obtained after being decoded by the decoder. The discriminator 14 and the loss function module 15 input the infrared image in the paired input data into the discriminator and the loss function module, and the discriminator and the loss function module perform loss discrimination on the infrared image in the paired input data and the infrared image generated by the generator, and then feed back to optimize the generator, thereby obtaining an infrared image generation model; The infrared image generation model 16 is used to obtain a real visible light image, and input the real visible light image into the infrared image generation model to obtain an infrared image corresponding to the real visible light image.

[0051] In the preferred embodiment of the present application, the process of the data input module 11 constructing the paired input data is as follows: An unmanned aerial vehicle with an infrared sensor and a visible light sensor is used to collect infrared and visible light video sequences at different times and in different scenes. After adjusting the sizes of the infrared and visible light video sequences to be consistent, image pairing is performed to construct paired input data of paired infrared images and visible light images with consistent image sizes and a quantity ratio of 1:1.

[0052] In a preferred embodiment of the present application, the semantic segmentation model 12 is a UNet network with a depth of 5, which includes 5 encoding blocks and 5 decoding blocks. Each encoding block is composed of a downsampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the downsampling convolution layers corresponding to the 5 encoding blocks are [64, 128, 256, 512, 1024] respectively; each decoding block is composed of a deconvolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the deconvolution layers corresponding to the 5 decoding blocks are [512, 256, 128, 64, 3] respectively.

[0053] In a preferred embodiment of the present application, the encoder of the generator 13 is an encoder based on a deep residual network, which includes multiple convolution modules and residual modules. The encoder of the deep residual network first performs a convolution operation on the input visible light image, where the convolution kernel size is 7×7, the step size is 2, and the padding size is 3×3; then, a four-layer residual network based on ResNet34 is used as a pre-trained network structure, and a jump connection is used to enhance the generalization representation ability of the regional block. The visible light image of the paired input data is input into the constructed encoder to obtain the feature encoding of the visible light image, including shallow features and underlying features.

[0054] In a preferred embodiment of the present application, the dilated convolution model of the generator 13 is a serial and parallel connection network. The dilated convolution model is used to enhance network recognition capability, expand the receptive field, and fuse multi-scale information. The dilated convolution model fuses the semantic features of the visible light image with the feature coding to obtain the spatial feature information of the visible light image and input it into the subsequent decoder.

[0055] In a preferred embodiment of the present application, the decoder of the generator 13 adopts a bottleneck connection structure of a residual network, uses a 1×1 convolution kernel to improve the computational efficiency of the network, uses transposed convolution upsampling to increase the side length to 32 times the original, and restores the size of the original visible light image to obtain the generated infrared image.

[0056] In a preferred embodiment of the present application, the discriminator 14 is composed of a five-layer convolutional network. The discriminator first performs five convolution operations on the infrared image generated by the generator to obtain a 30×30 pixel feature map, and then inputs the obtained feature map into the cross entropy function to determine whether it conforms to the target image distribution.

[0057] In a preferred embodiment of the present application, the loss function module 16 includes a generation adversarial loss function module, an L2 loss function module and a semantic segmentation loss function module. The generation of adversarial loss function module is performed as follows: {min}_{G}{max}_{D}{L}_{adv}(G,D)=\sum{[logD(y)]}+\sum{[1-logD(G(x))]} ; In the formula, G For the generator, D is the discriminator, x is a real visible light image, y is a real infrared image. G(x) Infrared images generated for adversarial networks; The L2 loss function module is implemented as follows: ; In the formula, x is a real visible light image, G For the generator, G(x) Infrared images generated for adversarial networks; The semantic segmentation loss function module is performed as follows: {L}^{r}_{seg}={E}_{{x,c}^{s}}[-{c}^{s}log(S(x))] ; in, is the scene category, S represents the semantic encoding branch, x It is a real visible light image.

[0058] In addition, the present application also provides an electronic device, which includes: a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement the above-mentioned method of generating an infrared image from a visible light image.

[0059] Finally, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method for generating an infrared image from a visible light image is implemented.

[0060] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating an infrared image from a visible light image, characterized in that: include: Construct paired input data consisting of infrared images and visible light images with the same image size and a quantity ratio of 1:1; Construct a semantic segmentation model, and input the visible light image in the paired input data into the semantic segmentation model to obtain the semantic features of the visible light image; A generator for generating an infrared image is constructed, wherein the generator includes an encoder, a decoder, and a dilated convolution model. A visible light image in paired input data is input into the encoder for feature extraction to obtain feature encoding of the visible light image. The semantic features of the visible light image and the feature encoding are then merged in the dilated convolution model, and the generated infrared image is obtained after being decoded by the decoder. Construct a discriminator and a loss function, input the infrared image in the paired input data into the discriminator and the loss function, the discriminator and the loss function perform loss discrimination on the infrared image in the paired input data and the infrared image generated by the generator, and then provide feedback to optimize the generator, thereby obtaining an infrared image generation model; A real visible light image is obtained, and the real visible light image is input into the infrared image generation model, so as to obtain an infrared image corresponding to the real visible light image.

2. The method for generating an infrared image from a visible light image as claimed in claim 1, characterized in that: The process of constructing paired input data is: An unmanned aerial vehicle with an infrared sensor and a visible light sensor is used to collect infrared and visible light video sequences at different times and in different scenes. After adjusting the sizes of the infrared and visible light video sequences to be consistent, image pairing is performed to construct paired input data of paired infrared images and visible light images with consistent image sizes and a quantity ratio of 1:

1.

3. The method for generating an infrared image from a visible light image as claimed in claim 1, characterized in that: The semantic segmentation model is a UNet network with a depth of 5, which includes 5 encoding blocks and 5 decoding blocks. Each encoding block consists of a downsampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the downsampling convolution layers corresponding to the 5 encoding blocks are [64, 128, 256, 512, 1024] respectively; each decoding block consists of a deconvolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the deconvolution layers corresponding to the 5 decoding blocks are [512, 256, 128, 64, 3] respectively.

4. The method for generating an infrared image from a visible light image according to claim 1, characterized in that: The encoder of the generator is an encoder based on a deep residual network, which includes multiple convolution modules and residual modules. The encoder of the deep residual network first performs a convolution operation on the input visible light image, where the convolution kernel size is 7×7, the step size is 2, and the padding size is 3×3; then, a four-layer residual network based on ResNet34 is used as the pre-trained network structure, and a jump connection is adopted to enhance the generalization representation ability of the regional block. The visible light image in the paired input data is input into the constructed encoder to obtain the feature encoding of the visible light image, including shallow features and underlying features.

5. The method for generating an infrared image from a visible light image as claimed in claim 4, characterized in that: The dilated convolution model of the generator is a serial and parallel connection network. The dilated convolution model fuses the semantic features of the visible light image with the feature coding to obtain the spatial feature information of the visible light image and input it into the subsequent decoder.

6. The method for generating an infrared image from a visible light image as claimed in claim 5, characterized in that: The decoder of the generator adopts a bottleneck connection structure of a residual network, uses a 1×1 convolution kernel to improve the computational efficiency of the network, uses transposed convolution upsampling to increase the side length to 32 times the original, and restores the size of the original visible light image to obtain the generated infrared image.

7. The method for generating an infrared image from a visible light image as claimed in claim 1, characterized in that: The discriminator is composed of a five-layer convolutional network. The discriminator first performs five convolution operations on the infrared image generated by the generator to obtain a 30×30 pixel feature map, and then inputs the obtained feature map into the cross entropy function to determine whether it conforms to the target image distribution.

8. The method for generating an infrared image from a visible light image as claimed in claim 7, characterized in that: The loss functions include generating adversarial loss function, L2 loss function and semantic segmentation loss function. The generated adversarial loss function is: {min}_{G}{max}_{D}{L}_{adv}(G,D)=\sum{[logD(y)]}+\sum{[1-logD(G(x))]} ; In the formula, G For the generator, D is the discriminator, x is a real visible light image, y is a real infrared image, G(x) Infrared images generated for adversarial networks; The L2 loss function is: ； In the formula, x is a real visible light image, G For the generator, G(x) Infrared images generated for the Generative Adversarial Network; The semantic segmentation loss function is: {L}^{r}_{seg}={E}_{{x,c}^{s}}[-{c}^{s}log(S(x))] ; in, is the scene category, S represents the semantic encoding branch, x It is a real visible light image.

9. A system for generating infrared images from visible light images, characterized in that: include: A data input module is used to construct paired input data consisting of infrared images and visible light images with the same image size and a quantity ratio of 1:1; A semantic segmentation model, used for inputting the visible light image in the paired input data into the semantic segmentation model to obtain semantic features of the visible light image; A generator, for generating an infrared image, comprising an encoder, a decoder and a dilated convolution model, wherein a visible light image in paired input data is input into the encoder for feature extraction to obtain feature encoding of the visible light image, and then the semantic features of the visible light image and the feature encoding are merged in the dilated convolution model, and the generated infrared image is obtained after being decoded by the decoder; A discriminator and loss function module, wherein the infrared image in the paired input data is input into the discriminator and loss function module, and the discriminator and loss function module performs loss discrimination on the infrared image in the paired input data and the infrared image generated by the generator, and then feeds back to optimize the generator, thereby obtaining an infrared image generation model; The infrared image generation model is used to obtain a real visible light image. The real visible light image is input into the infrared image generation model to obtain an infrared image corresponding to the real visible light image.

10. The system for generating infrared images from visible light images as claimed in claim 9, characterized in that: The process of constructing paired input data by the data input module is as follows: An unmanned aerial vehicle with an infrared sensor and a visible light sensor is used to collect infrared and visible light video sequences at different times and in different scenes. After adjusting the sizes of the infrared and visible light video sequences to be consistent, image pairing is performed to construct paired input data of paired infrared images and visible light images with consistent image sizes and a quantity ratio of 1:

1.

11. The system for generating infrared images from visible light images as claimed in claim 9, characterized in that: The semantic segmentation model is a UNet network with a depth of 5, which includes 5 encoding blocks and 5 decoding blocks. Each encoding block consists of a downsampling convolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the downsampling convolution layers corresponding to the 5 encoding blocks are [64, 128, 256, 512, 1024] respectively; each decoding block consists of a deconvolution layer with a convolution kernel size of 4, a step size of 2, and a padding size of 1, a batch normalization layer, and a Relu activation layer. The output channels of the deconvolution layers corresponding to the 5 decoding blocks are [512, 256, 128, 64, 3] respectively.

12. The system for generating infrared images from visible light images as claimed in claim 9, characterized in that: The encoder of the generator is an encoder based on a deep residual network, which includes multiple convolution modules and residual modules. The encoder of the deep residual network first performs a convolution operation on the input visible light image, where the convolution kernel size is 7×7, the step size is 2, and the padding size is 3×3; then, a four-layer residual network based on ResNet34 is used as the pre-trained network structure, and a jump connection is adopted to enhance the generalization representation ability of the regional block. The visible light image of the paired input data is input into the constructed encoder to obtain the feature encoding of the visible light image, including shallow features and underlying features.

13. The system for generating infrared images from visible light images as claimed in claim 12, characterized in that: The dilated convolution model of the generator is a serial and parallel connection network. The dilated convolution model fuses the feature coding and semantic features of the visible light image to obtain the spatial feature information of the visible light image, and inputs it into the subsequent decoder.

14. The system for generating infrared images from visible light images as claimed in claim 13, characterized in that: The decoder of the generator adopts a bottleneck connection structure of a residual network, uses a 1×1 convolution kernel to improve the computational efficiency of the network, uses transposed convolution upsampling to increase the side length to 32 times the original, and restores the size of the original visible light image to obtain the generated infrared image.

15. The system for generating infrared images from visible light images as claimed in claim 9, characterized in that: The discriminator is composed of a five-layer convolutional network. The discriminator first performs five convolution operations on the infrared image generated by the generator to obtain a 30×30 pixel feature map, and then inputs the obtained feature map into the cross entropy function to determine whether it conforms to the target image distribution.

16. The system for generating infrared images from visible light images as claimed in claim 15, characterized in that: The loss function module includes a generation adversarial loss function module, an L2 loss function module and a semantic segmentation loss function module. The generation of adversarial loss function module is performed as follows: {min}_{G}{max}_{D}{L}_{adv}(G,D)=\sum{[logD(y)]}+\sum{[1-logD(G(x))]} ; In the formula, G For the generator, D is the discriminator, x is a real visible light image, y is a real infrared image. G(x) Infrared images generated for adversarial networks; The L2 loss function module is performed as follows: ； In the formula, x is a real visible light image, G For the generator, G(x) Infrared images generated for the Generative Adversarial Network; The semantic segmentation loss function module is performed as follows: {L}^{r}_{seg}={E}_{{x,c}^{s}}[-{c}^{s}log(S(x))] ; in, is the scene category, S represents the semantic encoding branch, x It is a real visible light image.

17. An electronic device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the method for generating an infrared image from a visible light image according to any one of claims 1 to 8.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for generating an infrared image from a visible light image according to any one of claims 1 to 8 is implemented.

Images