A Near-Infrared Image Colorization Method Based on Cyclic Generative Adversarial Network

Abstract

A near-infrared image colorization method based on a cyclic generative adversarial network belongs to the field of image colorization technology. In order to solve the problem that the images obtained by the existing colorization methods lack fine details and do not conform to the visual effect of the human eye, and at the same time have high requirements on the data set and color mismatch, the method: prepare the data set: use the public near-infrared data set to train the entire generative adversarial network; build a network model; train the network model: input the prepared data set into the constructed network model for training; obtain the minimized loss function value; fine-tune the model; save the model. In the downsampling module, a combination of convolution blocks and dense residual blocks is used to replace the pooling operation, so that the details of the colorized image are finer. The use of a composite loss function helps to learn more detailed color and texture information of the image, which is more in line with human visual observation. The network architecture of the cyclic generative adversarial network is used to effectively avoid the dependence of the colorization network model on paired images.

Description

Technical Field

[0001] The invention relates to a near-infrared image colorization method based on a cyclic generative adversarial network, and belongs to the technical field of image colorization. Background Art

[0002] Image colorization technology has been widely used in the fields of art aesthetics, remote sensing satellites, medicine, and image preprocessing. Image colorization is a highly uncertain problem that requires mapping a real-valued grayscale image to a three-dimensional color image that has no unique solution. The imaging effect of near-infrared images is not affected by weather factors such as strong light and fog, and is insensitive to changes in the brightness of the scene. However, the lack of color information increases the difficulty for users to judge the information, so near-infrared images need to be colored. However, there are two key problems with existing near-infrared image colorization methods, namely, the lack of fine details and incorrect mapping.

[0003] The Chinese patent publication number is "CN114581560A", and the name is "Multi-scale neural network infrared image colorization method based on attention mechanism". This method first uses a composite loss function composed of pixel loss, edge loss and perceptual loss to train the infrared image colorization model; then, a two-dimensional convolutional neural network is used to extract features from the input infrared image pair at different resolution scales; then, the extracted high-dimensional feature information is refined through the attention mechanism; finally, the multi-scale information is fused to obtain the predicted colorized infrared image. The colorization result obtained by this method has blurred details, which does not meet the visual effect of the human eye. At the same time, it has high requirements on the data set and the colors do not match. Summary of the invention

[0004] In order to solve the problems that the images obtained by the existing colorization methods lack fine details and do not conform to the visual effect of the human eye, and at the same time have high requirements on the data set and color mismatch, the present invention provides a near-infrared image colorization method based on a recurrent generative adversarial network. The colorized image has fine details and is more in line with human visual observation. At the same time, the method proposed by the present invention has low requirements on the data set, does not need to rely on a large number of paired images, and has a higher degree of image color matching.

[0005] The solution to the technical problem of the present invention is:

[0006] A near-infrared image colorization method based on a recurrent generative adversarial network comprises the following steps:

[0007] Step 1, prepare the dataset: use the public near-infrared dataset to train the entire generative adversarial network, namely the RGB-NIR scene dataset;

[0008] Step 2, build the network model: the whole network consists of two generative adversarial networks, including generator 1, generator 2, discriminator 1 and discriminator 2; generator 1 consists of a downsampling module, a residual module, an upsampling module and a jump connection to achieve the conversion from infrared image to color image. Generator 2 removes the residual module on the basis of generator 1 to achieve the conversion from color image to infrared image. Both discriminator 1 and discriminator 2 use Markov discriminators to judge whether the generated fake image is real enough;

[0009] Step 3, training the network model: input the data set prepared in step 1 into the network model constructed in step 2 for training;

[0010] Step 4, obtain the minimized loss function value: optimize the generator and discriminator alternately by minimizing the loss functions of the generator and the discriminator until the confrontation between the generator and the discriminator reaches a balanced state, then the model parameters are considered to have been trained and the model parameters are saved;

[0011] Step 5, fine-tuning the model: Use near-infrared and visible light images to train and fine-tune the model to obtain stable and reliable model parameters, further improving the model's colorization capabilities;

[0012] Step 6, save the model: solidify the final model parameters. When the near-infrared image needs to be colorized, directly input the image into the network to obtain the final color infrared image.

[0013] In the step 2, the downsampling module consists of convolution blocks 1 to 5 and dense residual blocks 1 to 5, which are used to extract features from the input near-infrared image, wherein each convolution block reduces the feature map to half, and the dense residual block extracts and retains the information after each convolution block; the residual module consists of residual blocks 1 to 9, which are used to increase the network depth; the upsampling module consists of deconvolution blocks 1 to 6 and an output block, which is used for image reconstruction, wherein the deconvolution block decodes the information of the encoder and reconstructs the output image, and the output block outputs the reconstructed color image; the jump connection is used to directly map the feature details of each scale of the downsampling module to the upsampling module; the discriminator 1 and the discriminator 2 are both composed of six convolution blocks, and output true and false probability information to determine whether the input image is real.

[0014] In the step 4, the loss function used in the training process is a composite loss function of adversarial loss, cycle consistency loss, content loss and perceptual loss. The obtained color image is close to the real visible light image in terms of chromaticity and brightness, and can highlight the texture and details in the infrared image, improving the visual effect of the human eye.

[0015] The beneficial effects of the present invention are as follows:

[0016] 1. In the downsampling module, the combination of convolution blocks and dense residual blocks is used to replace the pooling operation, which can retain the detail information in the image to the maximum extent, thereby improving the feature extraction and expression capabilities of the network and making the colorized image details finer.

[0017] 2. Using a composite loss function helps to learn more detailed color and texture information of the image, making the generated image color more natural, the color information and content information in the image more matched, and more in line with human visual observation.

[0018] 3. The network architecture of the recurrent generative adversarial network makes the network training less demanding on the data set, effectively avoiding the dependence of the colorization network model on paired images. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a near-infrared image colorization method based on a cyclic generative adversarial network according to the present invention.

[0020] Figure 2 This is an overall network architecture diagram of a near-infrared image colorization method based on a cyclic generative adversarial network according to the present invention.

[0021] Figure 3 This is a network structure diagram of the generator described in the present invention.

[0022] Figure 4 This is a network structure diagram of the dense residual block in the generator network of the present invention.

[0023] Figure 5 This is a diagram of the network structure of the identifier described in the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, a near-infrared image colorization method based on a cyclic generative adversarial network comprises the following steps:

[0026] Step 1: Prepare the dataset. The entire generative adversarial network is trained using a public near-infrared dataset, namely the RGB-NIR scene dataset. Here, only the image resolution needs to be adjusted.

[0027] Step 2: Build a network model. The entire network consists of two generative adversarial networks, including Generator 1, Generator 2, Discriminator 1 and Discriminator 2. Generator 1 consists of a downsampling module, a residual module, an upsampling module and a jump connection to achieve the conversion from infrared images to color images. Generator 2 removes the residual module based on Generator 1 to achieve the conversion from color images to infrared images. Both Discriminator 1 and Discriminator 2 use Markov discriminators to determine whether the generated fake images are realistic enough. Among them, the downsampling module consists of convolution blocks 1 to 5 and dense residual blocks 1 to 5, which are used to extract features from the input near-infrared image, wherein each convolution block reduces the feature map to half, and the dense residual block extracts and retains the information after each convolution block; the residual module consists of residual blocks 1 to 9, which is used to increase the network depth; the upsampling module consists of deconvolution blocks 1 to 6 and an output block, which is used for image reconstruction, wherein the deconvolution block decodes the information of the encoder and then reconstructs the output image, and the output block outputs the reconstructed color image; the jump connection is used to directly map the feature details of each scale of the downsampling module to the upsampling module; both discriminators 1 and 2 use Markov discriminators, and output true and false probability information to determine whether the input image is real.

[0028] Step 3: Train the network model. Input the data set prepared in step 1 into the network model built in step 2 for training.

[0029] Step 4, obtain the minimized loss function value. By minimizing the loss function of the generator and the discriminator, the generator and the discriminator are optimized alternately until the confrontation between the generator and the discriminator reaches a balanced state. The model parameters can be considered to have been trained and the model parameters are saved. During the training process, the loss function uses a composite loss function of adversarial loss, cycle consistency loss, content loss, and perceptual loss. The obtained color image is close to the real visible light image in terms of chromaticity and brightness, which can highlight the texture and details in the infrared image and improve the visual effect of the human eye.

[0030] Step 5: Fine-tune the model. Use near-infrared and visible light images to train and fine-tune the model to obtain stable and reliable model parameters, further improving the model's colorization capabilities.

[0031] Step 6: Save the model. The final model parameters are solidified. When the near-infrared image needs to be colored, the image can be directly input into the network to obtain the final color infrared image.

[0032] Example:

[0033] like Figure 1 As shown, a near-infrared image colorization method based on a cyclic generative adversarial network comprises the following steps:

[0034] Step 1: Prepare the dataset. The near-infrared image colorization dataset uses the RGB-NIR scene dataset. The dataset contains 477 near-infrared images and 477 visible light images, and each image is cropped to 256×256.

[0035] Step 2: Build a network model. Figure 2 As shown in Figure 1, the network model structure consists of generator 1, generator 2, discriminator 1 and discriminator 2. The network structure diagram of generator 1 is shown in Figure 1. Figure 3 As shown in Figure 1, it consists of a convolution block, a dense residual block, a residual block, a deconvolution block, and an output block. The convolution block contains a convolution layer, instance normalization, and an activation function. The convolution kernel size is 3×3, the step size is 2, and the activation function selects the ReLU function. The ReLU function can converge quickly in the stochastic gradient descent algorithm and can effectively alleviate the problem of gradient disappearance. The dense residual block network structure diagram is shown in Figure 1. Figure 4 As shown in the figure, it includes convolutional layers, activation functions, splicing operations and skip connections. The convolution kernel size is 3×3, the step size and padding are both 1, the activation function selects the ReLU function, the first four splicing operations use concat, and the last splicing operation uses add. The residual block contains convolutional layers, activation functions, and splicing operations. The convolution kernel size of the first and third convolutional layers is 1×1, and the convolution kernel size of the second convolutional layer is 3×3. The step size and padding are both 1. The 1×1 convolution can reduce the number of channels, so that the 3×3 convolution can perform convolution operations with relatively low-dimensional inputs to improve computational efficiency. The activation function uses the ReLU function, and the splicing operation uses add. The deconvolution block contains deconvolution layers, activation functions, and instance normalization layers. The convolution kernel size is 3×3, the step size is 2, and the activation function uses the Leaky ReLU function. The Leaky ReLU function solves the neuron death problem of ReLU. The convolution kernel size of the convolution layer in the output block is 3×3, the step size and padding are both 1, and the activation function uses the Tanh function. The network structure diagram of the discriminator is shown in the figure below. Figure 5 As shown in the figure, it includes five convolution blocks. The first and fifth convolution blocks remove the batch normalization layer. The convolution kernel size is 4×4, the step size is 1, and the activation function uses the Leaky ReLU function. The use of the batch normalization layer and the Leaky ReLU function makes the model more stable and effectively realizes the back propagation of the gradient. The definitions of the ReLU function, the Leaky ReLU function, and the Tanh function are as follows:

[0036]

[0037]

[0038]

[0039] Step 3: Train the network model. Input the infrared image in the dataset into generator 1 to generate a false color image, and then input it into generator 2 to obtain a false infrared image. Through the cycle consistency loss, the false infrared image is restored to the input infrared image as much as possible. The other path is the same. At the same time, the generator and the discriminator are trained adversarially.

[0040] Step 4, obtain the minimized loss function value. The generator and discriminator are optimized alternately by minimizing the loss functions of the generator and the discriminator. The network parameters are updated by minimizing the loss function to obtain the optimal network model. The loss function is selected using a combination of adversarial loss, cycle consistency loss, content loss, and perceptual loss.

[0041] The secondary generator needs to restore the required image as much as possible within one cycle to maintain the consistency between the input image and the reconstructed output image. To this end, the cycle consistency loss is used, and its calculation formula is defined as:

[0042] L cyc =||x-G2(G1(x))||1

[0043] Among them, x represents the input infrared image, and G2(G1(x)) represents the fake infrared image generated by generator one and then generator two.

[0044] Content loss makes the generated image close to the real visible light image in terms of chromaticity and brightness, which helps the model to narrow the distance between the network output and the real image. Its essence is L1 loss. The calculation formula for defining content loss is:

[0045] L content =||y real -G(x)||1

[0046] Among them, y real represents the real visible light image, and G(x) represents the image generated by the generator.

[0047] Although the use of content loss is conducive to obtaining a high peak signal-to-noise ratio, the colorized result will lose detailed content. In order to encourage the network to output colorized results with more realistic details, adversarial loss is used. Adversarial loss is used to make the colorized result indistinguishable from the real visible light image. The calculation formula for defining adversarial loss is:

[0048] L adv =E x [-log(D(G(x),x))]

[0049] Where x is not only the input of the generator, but also the input of the discriminator as a conditional term.

[0050] Using adversarial loss can easily produce distorted textures. To alleviate this problem, the perceptual loss based on the ReLU function of the VGG-16 network is used. The perceptual loss is conducive to maintaining texture in high-level space. The calculation formula for perceptual loss is defined as:

[0051]

[0052] Among them, φ k (·) represents the feature representation of the kth max pooling layer in the VGG-16 network, C k H k W k Indicates the size of these feature representations.

[0053] Therefore, the total model loss L total for:

[0054] L total =λ cyc L cyc +L content +λ adv L adv +λ perceptual L perceptual

[0055] Among them, λ cyc , perceptual and λ adv They respectively represent the weights that control different loss shares in the entire total loss function. The setting of the weights is based on preliminary experiments on the training dataset.

[0056] The number of training times is set to 100, and the number of images input to the network each time is 16. This number can be determined according to the performance of the computer graphics processor. If the computer allows, the larger the number of inputs each time, the better, which can make the network more stable. The learning rate during training is set to 0.0002, which can ensure fast network fitting without causing network overfitting. The network parameter optimizer selects the stochastic gradient descent algorithm, which randomly selects a part from the entire training data set to participate in the model training. Therefore, as long as the size of the randomly selected data set is appropriate, there is no need to worry about the time cost of calculating the loss function and the complexity of model training. When the generator output image and the real image are input, the discriminator is difficult to distinguish between true and false, and the probability of judging true and false is one-half, so it can be considered that the network has been basically trained.

[0057] Step 5: Fine-tune the model. Use near-infrared and visible light images to train and fine-tune the model to obtain stable and reliable model parameters, further improving the model's colorization capabilities.

[0058] Step 6, save the model. After the network training is completed, all parameters in the network need to be saved. When the near-infrared image needs to be colorized, the image can be directly input into the network to obtain the final color infrared image.

[0059] Among them, the implementation of convolution, activation function, skip connection, splicing operation and residual structure are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0060] The present invention constructs a near-infrared image colorization network model based on a recurrent generative adversarial network, which can automatically colorize infrared images, avoid the tediousness of manual auxiliary colorization, and reduce the requirements for data sets. The feasibility and superiority of this method are further verified by calculating the relevant indicators of the image obtained by the existing method. The comparison of the relevant indicators of the existing technology and the method proposed in this invention is shown in Table 1:

[0061] Table 1 Comparison of relevant indicators of the prior art and the method proposed in the present invention

[0062]

[0063] It can be seen from the table that the method proposed in the present invention has two indicators of higher peak signal-to-noise ratio and structural similarity, which further illustrates that the present invention has better colorized image results.

Claims

1. A near-infrared image colorization method based on a recurrent generative adversarial network, characterized in that: The method comprises the following steps: Step 1, prepare the dataset: use the public near-infrared dataset to train the entire generative adversarial network, namely the RGB-NIR scene dataset; Step 2, build the network model: the whole network consists of two generative adversarial networks, including generator 1, generator 2, discriminator 1 and discriminator 2; generator 1 consists of a downsampling module, a residual module, an upsampling module and a jump connection to achieve the conversion from infrared image to color image. Generator 2 removes the residual module on the basis of generator 1 to achieve the conversion from color image to infrared image. Both discriminator 1 and discriminator 2 use Markov discriminators to judge whether the generated fake image is real enough; The downsampling module is composed of convolution blocks 1 to 5 and dense residual blocks 1 to 5, and is used to extract features from the input near-infrared image, wherein each convolution block reduces the feature map to half, and the dense residual block extracts and retains the information after each convolution block; the dense residual block includes a convolution layer, an activation function, a splicing operation and a jump connection, the convolution kernel size is 3×3, the step size and padding are both 1, the activation function selects the ReLU function, the first four splicing operations use concat, and the last splicing operation uses add; The residual module consists of residual blocks 1 to 9, and is used to increase the network depth; The upsampling module is composed of deconvolution blocks 1 to 6 and an output block, and is used for image reconstruction, wherein the deconvolution block decodes the information of the encoder to reconstruct the output image, and the output block outputs the reconstructed color image; The skip connection is used to directly map the feature details of each scale of the downsampling module to the upsampling module; The discriminator 1 and the discriminator 2 are both composed of six convolution blocks, and output true and false probability information to determine whether the input image is real; Step 3, training the network model: input the data set prepared in step 1 into the network model constructed in step 2 for training; Step 4, obtain the minimized loss function value: optimize the generator and discriminator alternately by minimizing the loss functions of the generator and the discriminator until the confrontation between the generator and the discriminator reaches a balanced state, which means that the model parameters have been trained and the model parameters are saved; The loss function uses a composite loss function of adversarial loss, cycle consistency loss, content loss and perceptual loss; the obtained color image is close to the real visible light image in terms of chromaticity and brightness, and can highlight the texture and details in the infrared image, thereby improving the visual effect of the human eye; Step 5, fine-tuning the model: Use near-infrared and visible light images to train and fine-tune the model to obtain stable and reliable model parameters, further improving the model's colorization capabilities; Step 6, save the model: solidify the final model parameters. When the near-infrared image needs to be colorized, directly input the image into the network to obtain the final color infrared image.

Images