A method for batch generating pseudo infrared thermal imaging images from visible light images

Abstract

The present invention discloses a method for batch generating pseudo infrared thermal imaging images from visible light images, belonging to the field of artificial intelligence and image generation technology. The method comprises: designing and constructing the vi2ir_GAN neural network of the present invention, adopting a multi-layer backbone structure with multiple local and global jump connections; adding attention mechanism modules between multiple corresponding levels of the encoder and the decoder, and the multi-layer attention mechanism can extract and fuse image feature information at different scale levels of the image to pay more attention to the significant features in the image; adding a function constraint term for calculating the structured loss in the loss function to drive the image generator to learn in a direction that makes the generated pseudo infrared image and the real infrared image more similar in terms of structural similarity; adding a multi-version output function, which can generate multiple versions of infrared images and enhance diversity. The present invention has the advantage of being able to quickly generate realistic pseudo infrared thermal imaging images in batches from visible light images.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, machine deep learning and image generation technology, and specifically to technologies such as improved neural networks and algorithms for training generation models of infrared images. More specifically, it is a method for batch generating pseudo infrared thermal imaging images from visible light images, which is used to provide sufficient and rich infrared image data resources for machine learning to identify target scene objects at night. Background Art

[0002] In the field of machine learning in artificial intelligence, training a model that can identify target objects at night requires a large amount of infrared thermal imaging image data. However, obtaining real infrared thermal imaging image data often faces some difficulties and challenges. One reason is cost considerations, and the other is the limited diversity of infrared image scenes and types. It is often costly, time-consuming, or unrealistic to deploy a large number of infrared thermal imaging equipment and have the opportunity to capture a wide variety of target objects and scenes.

[0003] Relatively speaking, visible light image resources are easier to obtain and are rich in variety. The lack of real infrared thermal imaging image data has created new technical requirements, namely, the need for low-cost, fast, and large-scale technology to automatically generate pseudo infrared thermal imaging images from visible light images. The effect of the pseudo infrared thermal imaging images generated by this technology needs to be as similar as possible to the real infrared thermal imaging images, so that the object recognition model trained by machine learning from the former (pseudo infrared images) can recognize the scenes and target objects of the latter (real infrared images) with high performance;

[0004] Generative Adversarial Networks (GAN) is an effective technology for image generation. Some studies have used general generative GAN (e.g., pix2pix) to generate target domains (e.g., infrared images) from source domains (e.g., visible light images). The quality and effects of the generated images (e.g., the structure of image scene elements and the distribution of light and dark in thermal imaging) are often significantly different from those taken by real infrared cameras. Thermal infrared imaging in the physical world is not based on the intensity difference of reflected light from different parts of the scene within the shooting range, but rather on the corresponding expression of light and dark in the image of the infrared camera based on the different intensities of thermal radiation from different parts. Some existing pseudo infrared images generated based on GAN are usually grayish overall. Even if there is actually a higher thermal radiation part and a lower thermal radiation part, there is no significant contrast difference in the corresponding parts of the generated pseudo infrared images. These generated pseudo infrared image data will adversely affect the performance of the target recognition model trained with this;

[0005] Therefore, it is necessary to develop a method for batch generating pseudo infrared thermal imaging images from visible light images in order to improve and optimize the general generative GAN technology and generate more realistic pseudo infrared thermal imaging image data to improve the performance of training recognition models. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the background technology and provide a method for batch generating pseudo infrared thermal imaging images from visible light images, which is a machine learning method and an artificial neural network, so as to be able to quickly generate realistic pseudo infrared thermal imaging images in batches from visible light images to meet the needs of machine learning for infrared thermal imaging image data and improve the performance of training recognition models; to solve the problem that the pseudo infrared images generated by the existing neural network model technology are usually overall darker, have a large difference in light and dark distribution characteristics from real infrared thermal imaging images, and the target recognition model trained with such pseudo infrared images has poor performance when actually identifying scene objects in low illumination at night.

[0007] In order to achieve the above object, the technical solution of the present invention is: a method for batch generating pseudo infrared thermal imaging images from visible light images, characterized in that it includes the following steps:

[0008] Step 1: Build the vi2ir_GAN neural network;

[0009] The vi2ir_GAN neural network consists of a neural network of an image generator G and a neural network of an image discriminator D;

[0010] Step 2: Dataset preparation;

[0011] Collect visible light images and the real infrared thermal imaging images corresponding to the visible light images, pre-process the data to make each pair of visible light and infrared images consistent in size and perform registration between the scene elements corresponding to the images, and form a data set of visible light and infrared image pairs;

[0012] The above registered visible light and infrared image pairs are divided into the training data set, validation data set and test data set of the vi2ir_GAN model according to a certain ratio;

[0013] Collect a visible light image dataset to be converted to generate a pseudo infrared image;

[0014] Step 3: Model training and test evaluation;

[0015] Input the training and validation data sets into the vi2ir_GAN neural network for training to obtain the vi2ir_GAN generative model; use the test data set to test and evaluate the performance of the vi2ir_GAN generative model;

[0016] Step 4: Generation of batch pseudo infrared image data:

[0017] The visible light image data to be converted is input into the vi2ir_GAN generative model obtained through the above training to generate and expand a large amount of pseudo infrared image data corresponding to these visible light image data for use in machine learning training of recognition models for identifying daytime and / or nighttime targets.

[0018] In the above technical solution, in step 1, the image generator G includes an improved neural network structure, a multi-layer spatial attention mechanism module, a multi-version output module, and a loss function used to train the image generator;

[0019] The improved neural network is a multi-layer structure consisting of an encoder and a decoder with multiple skip connections;

[0020] A multi-layer spatial attention mechanism module is added to the skip connections between multiple corresponding layers of the encoder and decoder to extract and fuse image feature information at different scale levels of the image;

[0021] The loss function used for image generator training is a combination of multiple loss functions, which encourages the vi2ir_GAN generative model to train and learn to generate more realistic pseudo infrared images.

[0022] The multi-version output module is attached to the decoder of the neural network of the image generator G and is used to generate multiple versions of infrared images to enhance diversity and expand the infrared data set required for training the object recognition model to enhance the generalization ability and universality of the recognition model.

[0023] The present invention facilitates the forward or backward propagation of image feature information through the network structure of the above-mentioned image generator, thereby facilitating the training of the model to generate a high-quality pseudo infrared image of the target domain.

[0024] In the above technical solution, the neural network model of the image generator G is a five-layer (from top to bottom, 0 / 1 / 2 / 3 / 4 layers, for ease of description) network of a U-shaped structure composed of an encoder and a decoder, and the components of the encoder on the left and the decoder on the right correspond to each other at each layer;

[0025] At the top level (level 0) of the U-shaped structure, set the input layer of the encoder part and the output layer of the decoder part;

[0026] At each level 1 / 2 / 3 below the top level, shortcut jump connections connect the corresponding parts of the encoder and the right decoder;

[0027] The above three jump connections are in the decoder part on the right. An attention mechanism module is set on each connection. Each attention mechanism module receives the input of the encoder on the left and the input of the encoder part of the layer below, and outputs it to the decoder part on the right of this layer after processing.

[0028] At each level 1 / 2 / 3, there is a residual network unit inside each part of the encoder and decoder to perform data processing transformation;

[0029] Image data flows in the neural network of the above structure. In the encoder part, image features are extracted layer by layer, the resolution is halved layer by layer, and the number of channels is doubled layer by layer. In the decoder part, images are generated layer by layer, the number of channels is halved layer by layer, and the resolution is doubled layer by layer.

[0030] In the above technical solution, in step 4, the visible light image data to be converted is input into the vi2ir_GAN generative model obtained by the above training to generate a pseudo infrared image. The specific processing method includes:

[0031] (1) At the encoder input layer (level 0), a 3-channel color visible light image is processed by a 3×3 convolution operation to form a 64-channel image; it is then processed by a 2×2 pooling operation to halve the image resolution and the result is input to the next layer (level 1);

[0032] (2) At level 1, the encoder receives 64 channels of data from level 0 above, processes it through a residual unit, and the result flows into two branches: one branch is transmitted to the decoder part of the same level through the skip connection of the current layer (via the attention module), and the other branch is processed by a 2×2 pooling operation, the image resolution is halved, and the result is input to the next level (level 2);

[0033] (3) At level 2 and level 3, the encoder processes data in the same way as level 1, but the resolution and number of channels change layer by layer, with the resolution decreasing and the number of channels increasing.

[0034] (4) At level 4, the encoder obtains data with 256 channels from the previous level 3, processes it through a residual unit, and directly outputs it to the decoder at this level.

[0035] (5) After the decoder receives data at level 4, it flows to two branches and is processed by a 3×3 deconvolution operation, halving the number of channels and doubling the image resolution. Then, on the one hand, the output enters the attention module of the previous level (level 3), and on the other hand, it is directly output to the decoder part of the previous level (level 3);

[0036] (6) The decoder receives the processed output of the attention module on the skip connection at level 3, as well as the direct output of the previous level (level 4), cascades and merges the two, and then processes them through a residual unit. The result flows to two branches to the previous level (level 2).

[0037] (7) The decoder processes the data at layer 2 and layer 1 in a similar way to layer 3, but the resolution and number of channels change layer by layer, with the resolution increasing and the number of channels decreasing.

[0038] (8) The decoder at level 1 processes a 64-channel image, which is then processed by a 3×3 deconvolution operation, with the resolution doubled and output to the output layer (i.e., level 0).

[0039] (9) At the output layer (i.e., layer 0), the decoder processes the 64-channel image received from layer 1 through a 3×3 convolution operation to obtain a single-channel resolution-doubled image, i.e., the generated pseudo infrared image;

[0040] The neural network model of the image discriminator D is designed with reference to the conventional VGGNet network structure. Its function is to compare the pseudo infrared image generated by the image generator G with the real infrared image during the training phase, and to judge whether the generated pseudo infrared image is real or fake based on the degree of similarity, so as to control the training process of the model.

[0041] In the above technical solution, the multi-layer spatial attention mechanism module in the neural network structure of the image generator G realizes the extraction and fusion of image feature information at different scale levels of the image through external connection mode and internal network structure;

[0042] The external connection mode of the attention mechanism module is as follows: the attention mechanism module receives two inputs x and g; x is the image feature information extracted from the spatial scale level transmitted from the jump connection of the encoder, and g is the image feature signal extracted from the spatial scale level transmitted from the next layer (i.e., the previous layer) of the decoder; after the two input information x and g are merged, the image feature information obtained after the four steps of processing on the neural network structure of the attention mechanism module is once again fused with the information x to obtain the output result, which is used as the input information of the neural network node of the decoder of the corresponding layer of the image generator G and participates in the processing of subsequent neural network nodes;

[0043] Internal network structure: The internal neural network structure of the attention mechanism module that implements the above data processing flow is that the two inputs x and g are respectively subjected to a Conv1×1 convolution operation and then The operation is merged, and then the result is once again outer-producted with the information x through four steps: ReLU linear rectification function, Conv1×1 convolution, Sigmoid activation function and Resampler resampler. Operational fusion.

[0044] In the above technical solution, the loss function used for training the improved image generator includes a function constraint term Loss_strucsim for calculating the structured loss, so as to drive and induce the improved image generator to learn in a direction that makes the generated pseudo infrared image and the real infrared image more similar in structure, so as to generate a more realistic thermal imaging image;

[0045] The calculation formula of the structural loss is Loss_strucsim = 1 - strucsim(x,y), where strucsim(x,y) represents the structural similarity metric, which is used to measure the similarity between the generated pseudo infrared image and its corresponding real infrared image.

[0046] Calculation; This includes three sub-items, illumination similarity l(x, y), contrast similarity c(x, y) and structural similarity s(x, y), which are calculated by the following formulas:

[0047] ;

[0048] In the above formula, μ x , μ y , σ x , σ y and σ xy are represents the local mean, standard deviation and cross-covariance of images x and y, and other symbols represent exponents or constants;

[0049] The comprehensive loss function of the image generator of vi2ir_GAN in the present invention (i.e., the loss function used for image generator training) loss_G consists of three parts: adversarial loss Loss_GAN, L1 loss Loss_L1, and structural loss Loss_strucsim:

[0050] Loss_G=aLoss_GAN+bLoss_L1+cLoss_strucsim

[0051] The above coefficients a, b, and c represent the weights of each component;

[0052] Among them, the adversarial loss Loss_GAN:

[0053] ;

[0054] L1 loss Loss_L1:

[0055] ;

[0056] In the above formula, G and D represent the image generator and image discriminator respectively, E represents the expected value, x and y represent the visible light and real infrared images participating in the training, and x' represents the generated pseudo infrared image;

[0057] The loss_strucsim sub-item added to the loss function in the present invention is used to repeatedly iteratively compare the differences in structure and brightness between the generated pseudo infrared image and the real infrared image during the training of the generative model, so as to induce the generative model to learn from the real infrared image, which can significantly improve the quality of the generated pseudo infrared image.

[0058] In the above technical solution, in step 4, the multi-version output module Vm attached to the neural network of the image generator G makes the pseudo infrared images generated and output in batches finally have diversity through a variety of image transformation algorithms, so as to more widely simulate the imaging effects of different thermal imaging cameras in real environments (such as different seasonal climates and temperature environments);

[0059] The image transformation algorithm at least includes a defogging algorithm, and a contrast, brightness, and sharpness adjustment algorithm;

[0060] Each visible light image is processed by the vi2ir_GAN image generation model to generate a corresponding pseudo infrared image, which is then processed by the multi-version output module Vm to generate multiple versions of pseudo infrared images.

[0061] Object recognition models trained using these diversely augmented pseudo-infrared datasets have better generalization, universality, and robustness.

[0062] The present invention has the following advantages:

[0063] (1) Setting up a neural network that improves the image generator;

[0064] The present invention designs a multi-layer network structure with multiple skip connections for the encoder and decoder in the image generator. These local or trunk-level skip connections can play an active role in local and global feature extraction, cross-level transmission and fusion of images: they can expand the receptive field of the neural network, so as to take into account the extraction of local details of the image and global semantic features; they can facilitate the forward or backward propagation of image feature information and facilitate the training of the model, so as to generate high-quality pseudo infrared images of the target domain;

[0065] (2) Introduced a multi-level spatial attention mechanism;

[0066] Based on the backbone neural network of the improved image generator, the present invention adds attention mechanism modules to the jump connections between the multiple corresponding levels of the encoder and the decoder therein, and the neural network inside each attention module receives the image feature information extracted by the image encoder at the image scale of the level and transmitted by the jump connection, and receives the feature signal transmitted by the image decoder at the previous level of the level. After being processed by multiple steps of the neural network inside the module, the image feature information transmitted by the jump connection of the current level is once again fused to obtain the result as the input information of the network node of the decoder at the current level, and the subsequent neural network nodes of the decoder are processed; this multi-layer attention mechanism neural network that can extract and fuse image feature information at different scale levels of the image can pay more attention to the significant features in the image (such as foreground object elements) and ignore other areas that are not related to the specific target task (such as image background); this helps to generate more realistic pseudo infrared images for significant foreground objects in the image, so that the generated pseudo infrared images can present significant brightness and contrast differences similar to those generated by real infrared thermal radiation for different objects or background elements in the scene;

[0067] (3) Improved loss function;

[0068] For this specific application task of generating pseudo infrared thermal imaging images from visible light, the present invention has made improvements and customizations based on the loss function of general GAN ​​(such as pix2pix), and constructed a combination of multiple loss functions to improve the performance of the trained model for generating pseudo infrared images. The specific improvements and customizations are as follows: adding structured comparative learning of image objects in image pairs of learning samples to generate more realistic thermal imaging images; adding a function constraint item in the loss function of the generator: structural loss loss_strucsim, to drive and induce the generator to learn in the direction of making the generated pseudo infrared image and the real infrared image more similar in terms of structural similarity, which can significantly improve the quality of the generated pseudo infrared image and make the generated pseudo infrared image present significant brightness and contrast differences for objects of different categories, such as the structure of image scene elements and the light and dark distribution of thermal imaging are more similar to real infrared thermal imaging images, which conforms to the physical characteristics of the actual infrared thermal radiation intensity differences of various parts or regions of the target object in the real scene;

[0069] (4) Added multi-version output module;

[0070] The multi-version output module can generate multiple versions of infrared images to enhance diversity, so as to expand the infrared data set required for training the object recognition model, so as to enhance the generalization ability and universality of the recognition model; in the multi-version output module attached hereto, the present invention designs and combines multiple image transformation algorithms, such as multiple defogging algorithms, contrast, brightness, sharpness adjustment algorithms, etc., so that the final output pseudo-infrared image has diversity, so as to more widely simulate the imaging effects of different thermal imaging cameras in the real environment (such as different seasonal climates and temperature environments) (the present invention obtains more pseudo-infrared images with different effects through the multi-version output module, which are used to train the recognition model), so that the generated pseudo-infrared image can present significant brightness and contrast differences similar to those produced by real infrared thermal radiation for different objects or background elements in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 Schematic diagram of the system structure and workflow of the vi2ir_GAN neural network in the present invention;

[0072] Figure 2 The specific neural network structure and image processing example flow chart of the encoder and decoder included in the image generator G of the present invention;

[0073] Figure 3 A specific neural network structure diagram of the attention mechanism module in the present invention;

[0074] Figure 4 A flowchart for generating multiple versions of pseudo infrared images for the multi-version module in the present invention;

[0075] Figure 5 1 is a sample example diagram of a visible light image and a generated pseudo infrared image in an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The following is a detailed description of the implementation of the present invention in conjunction with the accompanying drawings, but they do not constitute a limitation of the present invention and are only used as examples. At the same time, the advantages of the present invention are made clearer and easier to understand through the description.

[0077] The present invention designs and constructs the vi2ir_GAN neural network of the present invention, adopts a multi-layer backbone structure with multiple local and global jump connections; adds attention mechanism modules between multiple corresponding layers of the encoder and the decoder, and the multi-layer attention mechanism can extract and fuse image feature information at different scale levels of the image to pay more attention to significant features in the image; adds a function constraint term for calculating structured loss in the loss function to drive the image generator to learn in a direction that makes the generated pseudo infrared image and the real infrared image more similar in terms of structural similarity; adds a multi-version output function, can generate multiple versions of infrared images, enhance diversity, and expand the infrared data set required for training object recognition models;

[0078] The present invention designs vi2ir_GAN (visible to infrared Generative Adversarial Network), which is based on a general generative adversarial network (such as pix2pix), and involves adding or improving related algorithms and neural network structures. The model trained by vi2ir_GAN can quickly generate realistic pseudo infrared thermal imaging images from visible light images in batches. The machine learning method and artificial neural network (system formed by the method) described in the present invention can also be applied to other more generalized fields of image generation from source domain to target domain;

[0079] It should be pointed out that the name of the system of the present invention, vi2ir_GAN (as well as the names of the technical components contained therein or the terms involved), is used in the application documents of the present invention only to facilitate the description of the content, technology and embodiments of the present invention, and all technologies created by the present invention are not essentially limited by these names;

[0080] The present invention makes a series of improvements, optimizations and enhancements based on the generative adversarial network pix2pix, which specifically involves the following aspects and technical components:

[0081] 1. Improved the neural network of the image generator;

[0082] A multi-layer network structure with multiple skip connections is designed for the encoder and decoder in the image generator. These local or backbone-level skip connections can play a positive role in local and global feature extraction, cross-level transmission and fusion of images: they can expand the receptive field of the neural network, so that both local details of the image and global semantic features can be extracted; they can facilitate the forward or backward propagation of image feature information to facilitate the training of the model, so as to generate high-quality pseudo infrared images of the target domain;

[0083] 2. Introduced a multi-level spatial attention mechanism;

[0084] On the basis of the backbone neural network of the improved image generator described above, attention mechanism modules are added to the jump connections between multiple corresponding levels of the encoder and decoder. The neural network inside each attention module receives the image feature information extracted by the image encoder at the image scale of the level and transmitted by the jump connection, and receives the feature signal transmitted by the image decoder at the previous level of the level. After multiple steps of processing by the neural network inside the module, the image feature information transmitted by the jump connection of the current level is once again integrated to obtain the result as the input information of the network node of the decoder at the current level, and the decoder is processed by the subsequent neural network nodes;

[0085] This multi-layer attention mechanism neural network, which can extract and fuse image feature information at different scale levels of the image, can pay more attention to the salient features in the image (such as foreground object elements) and ignore other areas that are not relevant to the specific target task (such as the image background). This helps to generate more realistic infrared images for salient foreground objects in the image;

[0086] 3. Improved loss function;

[0087] The loss function (also called objective function, cost function or error function) is an indicator used to measure the difference between the model prediction and the actual observation value. It can play many roles in the model training process: measuring model performance, training optimization, back propagation to adjust model parameters, inducing and encouraging the model to learn towards the preset performance target, etc. For the specific application task of generating pseudo infrared thermal imaging images from visible light, the present invention improves and customizes the loss function of general GAN ​​(such as pix2pix), and constructs a combination of multiple loss functions to improve the performance of the trained model for generating pseudo infrared images;

[0088] Added comparative learning of the structure of image objects in the image pairs of learning samples to generate more realistic thermal imaging images. This is to add a function constraint term in the loss function of the generator: structural loss Loss_strucsim, to drive and induce the generator to learn in the direction of making the generated pseudo infrared image closer to the real infrared image in terms of structural similarity, which can significantly improve the quality of the generated pseudo infrared image, for example, the structure of the image scene elements and the light and dark distribution of the thermal imaging are closer to the real infrared thermal imaging images;

[0089] The calculation formula of structural loss is Loss_strucsim=1-strucsim(x,y), where strucsim(x,y) represents the structural similarity metric, which is an indicator to measure the similarity between two images (here refers to the generated pseudo-infrared image and its corresponding real infrared image). Calculation. This includes three sub-items, illumination similarity l(x, y), contrast similarity c(x, y) and structural similarity s(x, y), which are calculated by the following formulas:

[0090] ;

[0091] In the above formula, μ x , μ y , σ x , σ y and σ xy are represents the local mean, standard deviation and cross-covariance of images x and y, and other symbols represent exponents or constants;

[0092] Combined with the loss function of general GAN ​​(such as pix2pix), the comprehensive loss function loss_G of the image generator of vi2ir_GAN consists of three parts: adversarial loss Loss_GAN, L1 loss Loss_L1, and structural loss Loss_strucsim:

[0093] Loss_G = aLoss_GAN+bLoss_L1+cLoss_strucsim

[0094] The above coefficients a, b, and c represent the weights of each component;

[0095] Among them, the adversarial loss Loss_GAN:

[0096] ;

[0097] And L1 loss Loss_L1:

[0098] ;

[0099] In the above formula, G and D represent the image generator and image discriminator respectively, E represents the expected value, x and y represent the visible light and real infrared images participating in the training, and x' represents the generated pseudo infrared image;

[0100] 4. Added multi-version output module;

[0101] Its function is to be able to generate multiple versions of infrared images to enhance diversity in order to expand the infrared data set required for training object recognition models, so as to enhance the generalization ability and universality of the recognition model;

[0102] In this additional multi-version output module, a variety of image transformation algorithms are designed and combined, such as a variety of defogging algorithms, contrast, brightness, sharpness adjustment algorithms, etc., so that the final output pseudo-infrared image has diversity to more widely simulate the imaging effects of different thermal imaging cameras in real environments (such as different seasonal climates and temperature environments). Experiments have shown that the object recognition model trained with these diverse expanded pseudo-infrared datasets has better generalization ability and has higher robustness in identifying dynamically changing low-light target objects (such as at night).

[0103] The embodiment section of this specification will specifically illustrate the technical components and related details involved in the present invention in conjunction with the accompanying drawings, and take the training of image generation models and the rapid batch generation of pseudo infrared image data of specific types of ships as an example to illustrate and verify the relevant technologies, functions and performances of the vi2ir_GAN of the present invention;

[0104] like Figure 1 As shown in FIG. 1 , the architecture of the vi2ir_GAN of the present invention includes the main components of an image generator G (which consists of an encoder and a decoder) and an image authenticity discriminator D (Discriminator). In addition, an additional multi-version output module V m (like Figure 4 As shown) is also part of the vi2ir_GAN architecture;

[0105] exist Figure 1 The image generator G, which consists of an encoder and a decoder, and the multi-layer attention module marked in the figure, have specific neural network structures and workflows in them. Figure 2 and Figure 3 Provide detailed illustrations;

[0106] The image generator G transforms the source domain (visible light) image X i Transformed into the target domain (infrared) image Y' i The encoder is used to extract the spatial features of the image from the input visible light image, while the decoder is used to generate a pseudo infrared image based on the spatial features of the image extracted by the former. The image authenticity discriminator D receives the generator G and generates the output pseudo infrared image Y' i , and X i The corresponding real infrared image Y i , for Y' i and Y iCompare the similarities between the two and make a judgment Y' according to a certain probability i Is the conclusion true or false?

[0107] The loss function (objective function) Loss_G=αLoss_GAN+βLoss_L1+γLoss_strucsim designed and constructed by the invention vi2ir_GAN can drive and encourage the image generator G to learn in the direction of making the generated pseudo infrared image more realistic and similar to the real infrared image (such as more similar structural similarity, more realistic distribution of light and dark areas of thermal imaging and other image features) during the training process of the neural network model.

[0108] Figure 1 The training process of the vi2ir_GAN architecture shown in the figure can be formally expressed as: i =G(X i , Y i ), i=1,2,3,......, where X and Y are the visible light image of the source domain and the corresponding real infrared thermal imaging image of the target domain, respectively, Y' is the pseudo infrared image of the target domain generated by the image generator G, and i represents the number of training rounds. The weight model of the neural network parameters of the image generator G and the authenticity discriminator D is repeatedly trained and optimized with paired visible light-infrared images as input. In each round of training, the generator G calculates the loss function Loss_G to minimize its convergence as much as possible; while the discriminator D is the infrared image Y' generated by G. i and the real infrared image Y i Compare the two and make a judgment based on the degree of similarity Y' i The conclusion is true or false. G and D are trained repeatedly in this way (until the set number of training rounds is completed; or until the pseudo infrared image generated by G is realistic enough that D cannot distinguish its authenticity, that is, Y' i ≈Y i ), the neural network parameters are repeatedly learned and optimized, thereby forming a weight model of the neural network parameters of the trained image generator G and image discriminator D;

[0109] Figure 2The left and right parts of the diagram respectively illustrate the specific neural network structures and processes of the encoder and decoder contained in the image generator G. The present invention introduces a hybrid neural network structure that integrates the characteristics of Unet and ResNet for the image generator G design, as the neural network of the encoder and decoder. It is characterized in that some layers in the U-net structure are replaced by pre-activated residual blocks of ResNet, so that the advantages of the two network structures can be comprehensively obtained; and a multi-layer attention mechanism module is introduced to promote the generation of high-quality pseudo infrared images;

[0110] Figure 2 The legend in the lower left and right corners indicates Figure 2 The functional components or processes represented by the relevant graphics elements in the figure. The key components include 7 residual network units / blocks, which have a skip connection mechanism inside; the skip connection (left and right sides) of the Unet network itself, the connection mechanism of the corresponding layers of the encoder and decoder; and the attention mechanism modules added to the 3 skip connections directly connecting the left and right sides. These two skip connection mechanisms bring many advantages: they can build deep neural networks without the problem of gradient disappearance or explosion; they can facilitate the forward or backward propagation of image feature information and facilitate model training; they can expand the receptive field of the neural network, so as to take into account the extraction of local details and global semantic features of the image; in addition, the added 3 attention modules implement a multi-layer attention mechanism, which can extract and fuse image feature information at different scale levels of the image;

[0111] exist Figure 2 In the figure, an example is given of a process in which a certain visible light image (resolution 1024*1024*3 color channels) is generated into a pseudo infrared image (grayscale image with resolution 1024*1024*1 color channel) through the processing flow of the neural network of the encoder and decoder of the image generator G; it should be pointed out that the image resolution of this example is only for illustrating the workflow of the neural network in this embodiment, and the vi2ir_GAN of the present invention also supports images of other resolutions;

[0112] Figure 3 The specific diagram is shown (in Figure 2 The neural network structure diagram inside the attention mechanism module (in the jump connection between the corresponding layers of the encoder and decoder shown in Figure 3In the example, Conv is a convolution operation; ReLU is a linear rectification function (an activation function); Sigmoid is an activation function; and Resampler is a resampler). The attention mechanism module receives two inputs x and g; x is the image feature information extracted from the spatial scale level transmitted from the jump connection of the encoder, and g is the feature signal from the next layer (i.e., the previous layer) of the decoder. After the two input information x and g are merged, the image feature information obtained after several steps of neural network processing is once again fused with the information x to obtain the output result, which is used as the input information of the neural network node of the decoder at this corresponding level and participates in the processing of subsequent neural network nodes. In summary, combined with Figure 2 and Figure 3 As can be seen from the diagram of the workflow of the attention module in the vi2ir_GAN system of the present invention, this multi-layer attention mechanism is an important technical component, which can prompt the neural network to pay more attention to the significant features in the image (such as foreground object elements) and ignore other irrelevant areas (such as image background). This helps to generate more realistic infrared images for significant foreground objects in the image;

[0113] Figure 4 The multi-version output module V included in the technical components of the present invention is illustrated. m The function of is to use the weight model of the neural network parameters obtained by training (referred to as vi2ir_GAN image generation model) to generate pseudo infrared images (i.e., the inference stage of the model) after the image generator G is trained. Each visible light image is generated by the vi2ir_GAN image generation model to generate a corresponding pseudo infrared image, which is then output through this multi-version output module V. m Multiple versions of pseudo infrared images are generated by processing. m In the process, a variety of image transformation algorithms are designed and combined, such as a variety of defogging algorithms, contrast, brightness, sharpness adjustment algorithms, etc., so that the final output pseudo infrared image has diversity. This diversity enhancement can expand the infrared data set required for subsequent training object recognition models to more widely simulate the imaging effects of different thermal imaging cameras in actual environments (such as different seasonal climate and temperature environments), so as to enhance the generalization ability, universality and robustness of the recognition model trained by these data.

[0114] Based on the technical implementation described above, the specific steps of an embodiment of using the vi2ir_GAN of the present invention to train a model for generating pseudo infrared images from visible light images on a sample data set will be described below through an embodiment. This also has a guiding role in applying the present invention to other target scenes and generating pseudo infrared thermal imaging images from visible light images in batches.

[0115] Embodiment: The application scenario of this embodiment is: in a certain business of supervising illegal acts and incidents involving rivers and lakes, based on the real-time images obtained by video surveillance, an AI visual recognition model that can identify a certain type of ship (such as sand mining and transportation ships, referred to as special ships here) around the clock (day and night) is needed. The only available data sets for training such a model are daytime visible light image data sets of this special ship and a small amount of real infrared thermal imaging data sets, as well as a relatively sufficient number of visible light and corresponding infrared thermal imaging data sets of general types of ships (referred to as generalized types here). However, the above-mentioned image data cannot directly meet the requirements for training an AI visual recognition model that can accurately identify the special ship at night, because there are only a small amount of infrared thermal imaging data sets about this special ship available for machine learning training;

[0116] In this embodiment, the present invention uses the image data of generalized ships to train the vi2ir_GAN generative model so that it can generate a pseudo infrared image dataset of such special ships. The relevant process steps are as follows:

[0117] Step 1) Collect and organize the visible light images of generalized ships and their corresponding real infrared images, pre-process the data to make the corresponding images consistent in size and perform registration between the scene elements corresponding to the images, and form a data set of visible light and infrared image pairs of generalized ship types;

[0118] Step 2) The datasets of visible light and infrared image pairs of these generalized types of ships are divided into training datasets and validation datasets of the vi2ir_GAN model in a ratio of 80% to 20%;

[0119] Step 3) Model training: The training dataset and the validation dataset of the paired images of the generalized types of ships divided in step 2) above are trained using the vi2ir_GAN system (such as Figure 1 As shown in the figure, the neural network weight model of the image generator of vi2ir_GAN is obtained (referred to as vi2ir_GAN generative model);

[0120] Step 4) collect and organize the image data of this type of special ships, and form image pairs with both visible light images and corresponding real infrared thermal imaging images. Preprocess the data to make the corresponding images consistent in size and perform registration between the scene elements corresponding to the images to form a test data set for this type of special ships; and use other image data with only visible light images (but without corresponding real infrared thermal imaging images) as a production data set of pseudo infrared images of this type of special ships;

[0121] Step 5) Model testing and index evaluation: Input the visible light image data in the test data set of this type of special ship type compiled in step 4) above into the vi2ir_GAN generative model trained in step 3) above to generate a pseudo infrared image data set corresponding to this test data set; then calculate the evaluation indexes (such as LPIPS, FID, PSNR, SSIM, etc.) for the real infrared thermal imaging image data set in this test data set and the pseudo infrared image data set generated in this step to verify the quality of the pseudo infrared images generated by the vi2ir_GAN generative model and evaluate the performance of the vi2ir_GAN technology;

[0122] Step 6) Generation of batch pseudo infrared image data: The production dataset of this type of special ship collected in step 4) above (visible light images without corresponding real infrared thermal imaging images) is input into the vi2ir_GAN generative model trained in step 3) and its multi-version generation function (such as Figure 4 As shown in the figure), a large amount of pseudo infrared image data corresponding to these visible light image data (about this type of special ships) can be generated and expanded, which can be used for machine learning to train models for identifying night targets; Figure 5 Sample examples of visible light images of this type of special ship and their corresponding generated pseudo-infrared images are shown ( Figure 5 The left side is a visible light color image during the day, and the right side is an infrared image generated by the neural network model vi2ir_GAN of the present invention that simulates the night (which can be used to train a recognition model for identifying low-light scenes at night);

[0123] The evaluation index calculated in step 5) and the image effect generated in step 6) above (such as Figure 5 Example) The image quality effect of the vi2ir_GAN neural network structure and related algorithms for generating pseudo infrared thermal imaging images from visible light images was analyzed and verified from the aspects of quantitative indicators and visual perception, proving the effectiveness and authenticity of the vi2ir_GAN technology of the present invention;

[0124] The analysis results of this embodiment are as follows:

[0125] First, compare visually;

[0126] The pseudo infrared images generated by general adversarial neural network models (such as pix2pix) are usually overall dark and cannot truly show the significant differences in the intensity of thermal radiation of different objects or background elements in the scene; while the pseudo infrared images generated by vi2ir_GAN of the present invention are more realistic and closer to the brightness and darkness of images taken by real thermal infrared cameras. Figure 5In the example of several sample images of pseudo infrared images generated by vi2ir_GAN of the present invention, the left image is a visible light image, and the right image is pseudo infrared image data generated by the vi2ir_GAN generative model trained by visible light image input; Figure 5 The pseudo infrared image in the right image is of high quality, and the structure of the image scene elements and the light and dark distribution of the thermal imaging are more similar to the real infrared thermal imaging image. Figure 5 In the figure, no matter whether the steel structure of the hull is bright or dark in the visible light image on the left, it can show a relatively high brightness (with a significant brightness contrast difference) in the (generated) pseudo-infrared image on the right, which is consistent with the physical characteristic of the hull having more thermal radiation in the actual scene; the thermal imaging images taken by the real thermal infrared camera also show this characteristic. Therefore, from an intuitive observation and comparison, the effect of the pseudo-infrared image generated by the vi2ir_GAN generative model of the present invention is significantly better than that of some other current GAN-based models for generating pseudo-infrared images, because the pseudo-infrared images generated by some other current GAN-based models are usually overall darker, even if the actual physically higher thermal radiation parts and lower thermal radiation parts (such as the background part), the corresponding parts in the generated pseudo-infrared image do not have a significant brightness contrast difference;

[0127] Secondly, the quantitative evaluation indexes were calculated to verify the effect more objectively;

[0128] On the test data set, for general GAN ​​(such as pix2pix) and vi2ir_GAN of the present invention, the pseudo infrared images generated respectively and their corresponding real images taken by thermal infrared cameras were calculated with multiple evaluation indicators, such as LPIPS (Learned Perceptual Image Patch Similarity, the smaller the value, the more similar), FID (Fréchet Inception Distance, the smaller the value, the better the quality), PSNR (Peak Signal-to-Noise Ratio, the larger the value, the better the quality), SSIM (Structural SIMilarity, the larger the value, the better the quality) and other quantitative indicators, and then compared. These evaluation indicators calculated on the test data set (as shown in Table 1 below) have significant advantages over existing general GAN ​​(such as pix2pix) and the present invention, which verifies that compared with existing general GAN ​​(such as pix2pix), the vi2ir_GAN of the present invention can achieve better image effects;

[0129] Table 1 Evaluation indicators

[0130] Model indicators LPIPS FID PSNR(dB) SSIM Pix2Pix 0.078 69.35 24.61 0.782 vi2ir_GAN 0.052 51.23 35.78 0.976

[0131] Moreover, the various image transformation algorithms designed and combined in the additional multi-version output module of the present invention (such as various defogging algorithms, contrast, brightness, and sharpness adjustment algorithms) can diversely expand the infrared data set required for the subsequent training object recognition model to more widely simulate the imaging effects of different thermal imaging cameras in actual environments (such as different seasonal climate temperatures or illumination environments) to enhance the generalization ability, universality, and robustness of the recognition model trained by these data.

[0132] The advantageous effect of vi2ir_GAN in the present invention in generating pseudo infrared thermal imaging images is due to its improved neural network structure and algorithm modules, including the backbone neural network structure of the image generator improved on the basis of the general generative network, the added multi-layer spatial attention mechanism module, the improved loss function, and the additional multi-version output module and other technical components, making it more suitable for generating high-quality images of the target domain.

[0133] In summary, it should be pointed out that the above embodiments of training a model for generating pseudo infrared thermal imaging image data of a specific type of ship are intended to illustrate and verify the techniques and beneficial effects of the present invention;

[0134] In fact, vi2ir_GAN can also be used in other fields of generating infrared image scenes, as well as in more general fields of generating images from images (i.e., image generation that is not necessarily related to infrared images), such as generating color images from grayscale images, image style transfer, bidirectional conversion between real scene images and cartoon images, etc.

[0135] It should be understood that the above description is only an exemplary embodiment of the present invention and is not intended to limit the protection scope of the present invention.

[0136] Other parts not described belong to the prior art.

Claims

1. A method for batch generating pseudo infrared thermal imaging images from visible light images, characterized in that: The steps include: Step 1: Build the vi2ir_GAN neural network; The vi2ir_GAN neural network consists of a neural network of an image generator G and a neural network of an image discriminator D; In step 1, the image generator G includes an improved neural network structure, a multi-layer spatial attention mechanism module, a multi-version output module, and a loss function used to train the image generator; The improved neural network is a multi-layer structure consisting of an encoder and a decoder with multiple skip connections; A multi-layer spatial attention mechanism module is added to the skip connections between multiple corresponding layers of the encoder and decoder to extract and fuse image feature information at different scale levels of the image; The loss function used for image generator training is a combination of multiple loss functions, which encourages the vi2ir_GAN generative model to train and learn to generate more realistic pseudo infrared images. The loss function loss_G used for image generator training consists of three parts: adversarial loss Loss_GAN, L1 loss Loss_L1, and structural loss Loss_strucsim; Loss_G=aLoss_GAN+bLoss_L1+cLoss_strucsim The above coefficients a, b, and c represent the weights of each component; Among them, the adversarial loss Loss_GAN: L1 loss Loss_L1: In the above formula, G and D represent the image generator and image discriminator respectively, E represents the expected value, x and y represent the visible light and real infrared images participating in the training, and x' represents the generated pseudo infrared image; The structural loss Loss_strucsim is a function constraint for calculating the structured loss, which is used to drive and induce the improved image generator to learn in the direction of making the generated pseudo infrared image and the real infrared image more similar in terms of structural similarity, so as to generate more realistic thermal imaging images; the calculation formula of the structural loss is Loss_strucsim=1-strucsim(x,y), where strucsim(x,y) represents the structural similarity measure, which is used to measure the similarity between the generated pseudo infrared image and its corresponding real infrared image. strucsim(x,y)=[l(x,y)]α·[c(x,y)]β·[s(x,y)] γ Calculation; This includes three sub-items, illumination similarity l(x,y), contrast similarity c(x,y) and structural similarity s(x,y), which are calculated by the following formulas: In the above formula, μ x ,μ y ,σ x ,σ y and σ xy are represents the local mean, standard deviation and cross-covariance of images x and y; The multi-version output module is attached to the decoder of the neural network of the image generator G and is used to generate multiple versions of infrared images to enhance diversity and expand the infrared data set required for training the object recognition model to enhance the generalization ability and universality of the recognition model. Step 2: Dataset preparation; Collect visible light images and the real infrared thermal imaging images corresponding to the visible light images, pre-process the data to make each pair of visible light and infrared images consistent in size and perform registration between the scene elements corresponding to the images, and form a data set of visible light and infrared image pairs; The above registered visible light and infrared image pairs are divided into the training data set, validation data set and test data set of the vi2ir_GAN model according to a certain ratio; Collect a visible light image dataset to be converted to generate a pseudo infrared image; Step 3: Model training and test evaluation; Input the training and validation data sets into the vi2ir_GAN neural network for training to obtain the vi2ir_GAN generative model; use the test data set to test and evaluate the performance of the vi2ir_GAN generative model; Step 4: Generation of batch pseudo infrared image data: The visible light image data to be converted is input into the vi2ir_GAN generative model obtained through the above training to generate and expand a large amount of pseudo infrared image data corresponding to these visible light image data for use in machine learning training of recognition models for identifying daytime and / or nighttime targets.

2. The method for batch generating pseudo infrared thermal imaging images from visible light images according to claim 1, characterized in that: The neural network model of the improved image generator G is a five-layer network with a U-shaped structure consisting of an encoder and a decoder. The components of the encoder on the left and the decoder on the right correspond to each other at each layer. At the top level of the U-shaped structure, the input layer of the encoder part and the output layer of the decoder part are set; In levels 1, 2, and 3 below the top level, corresponding parts of the encoder and the right decoder are connected by shortcut jump connections; The above three jump connections are connected to the decoder part on the right. An attention mechanism module is set on each connection. Each attention mechanism module receives the input of the encoder on the left and the input of the encoder part of the layer below, and outputs it to the decoder part on the right of this layer after processing. In layers 1, 2, and 3, a residual network unit is provided inside each part of the encoder and decoder for data processing transformation; Image data flows in the neural network of the above structure. In the encoder part, image features are extracted layer by layer, the resolution is halved layer by layer, and the number of channels is doubled layer by layer. In the decoder part, images are generated layer by layer, the number of channels is halved layer by layer, and the resolution is doubled layer by layer.

3. The method for batch generating pseudo infrared thermal imaging images from visible light images according to claim 2, characterized in that: In step 4, the visible light image data to be converted is input into the vi2ir_GAN generative model trained above to generate a pseudo infrared image. The specific processing flow is as follows: (1) At the encoder input layer, a 3-channel color visible light image is processed by a 3×3 convolution operation to form a 64-channel image. It is then processed by a 2×2 pooling operation to halve the image resolution and the result is input to the next layer. (2) At level 1, the encoder receives 64 channels of data from the encoder input level above, processes it through a residual unit, and the result flows into two branches: one branch is transmitted to the decoder part of the same level through the skip connection of the current layer, and the other branch is processed by a 2×2 pooling operation, the image resolution is halved, and the result is input to the next level; (3) At level 2 and level 3, the encoder processes data in the same way as level 1, but the resolution decreases and the number of channels increases layer by layer; (4) At level 4, the encoder obtains data with 256 channels from the previous level 3, processes it through a residual unit, and directly outputs it to the decoder part of this level; (5) After the decoder receives data at level 4, it flows to two branches and is processed by a 3×3 deconvolution operation, which halves the number of channels and doubles the image resolution. Then, on the one hand, the output enters the attention module of the previous level, and on the other hand, it is directly output to the decoder part of the previous level. (6) The decoder receives the processed output of the attention module on the skip connection at level 3, as well as the direct output of the previous level, cascades and merges the two, and then processes them through a residual unit. The result flows to two branches to the previous level. (7) The decoder processes the data at layer 2 and layer 1 in a similar way to layer 3, but the resolution increases and the number of channels decreases at each layer; (8) The decoder processing result at level 1 is a 64-channel image, which is then processed by a 3×3 deconvolution operation, the resolution is doubled, and output to the output layer; (9) At the output layer, the decoder processes the 64-channel image received from level 1 through a 3×3 convolution operation to obtain a single-channel resolution-doubled image, i.e., the generated pseudo infrared image.

4. The method for batch generating pseudo infrared thermal imaging images from visible light images according to claim 3, characterized in that: The multi-layer spatial attention mechanism module extracts and fuses image feature information at different scale levels of the image through external connection structure and internal network structure; The external connection mode of the attention mechanism module is as follows: the attention mechanism module receives two inputs x and g; x is the image feature information extracted from the spatial scale level transmitted from the jump connection of the encoder, and g is the image feature signal extracted from the spatial scale level transmitted from the next level of the decoder; after the two input information x and g are merged, the image feature information obtained after being processed by the neural network structure of the attention mechanism module is once again fused with the information x to obtain the output result, which is used as the input information of the neural network node of the decoder of the corresponding level of the image generator G and participates in the processing of the subsequent neural network nodes; The internal network structure of the attention mechanism module that implements the above data processing flow is: the two inputs x and g are respectively subjected to a Conv1×1 convolution operation and the results are The operation is merged, and then the result is once again outer-producted with the information x through four steps: ReLU linear rectification function, Conv1×1 convolution, Sigmoid activation function and Resampler resampler. Operational fusion.

5. The method for batch generating pseudo infrared thermal imaging images from visible light images according to claim 4, characterized in that: In step 4, the multi-version output module Vm attached to the neural network of the image generator G uses a variety of image transformation algorithms to make the final batch generated output pseudo infrared images diverse, so as to more widely simulate the imaging effects of different thermal imaging cameras in the real environment; The image transformation algorithm at least includes a defogging algorithm, and a contrast, brightness, and sharpness adjustment algorithm; Each visible light image is generated through the vi2ir_GAN image generation model to generate a corresponding pseudo infrared image, which is then processed by the multi-version output module Vm to generate multiple versions of pseudo infrared images.

Images