Near-infrared image colorization method based on two-stage conditional generative adversarial network

Abstract

The present invention discloses a near-infrared image colorization method based on a two-stage conditional generative adversarial network. The near-infrared image colorization method based on the two-stage conditional generative adversarial network comprises the following steps: S1: obtaining a near-infrared grayscale image; S2: constructing a grayscale preprocessing module for the near-infrared image; S3: using the grayscale preprocessing module for the near-infrared image to preprocess the near-infrared grayscale image to obtain a grayscale image; S4: constructing an image colorization module; S5: using the image colorization module to colorize the grayscale image to obtain a colored generated image, wherein the image colorization module is further used to discriminate color feature vectors extracted from the colored generated image and a training image, and obtain a discrimination result for training the image colorization module. The present invention can solve the problem that existing image colorization methods lack corresponding grayscale processing for near-infrared images containing a large amount of noise information, resulting in the colored image having an overall visual effect of bright and dark flickering.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a near-infrared image colorization method based on a two-stage conditional generative adversarial network. Background Art

[0002] The purpose of image colorization is to add color information to the input grayscale image to obtain a color image, which facilitates scene understanding and the next stage of analysis and processing. Traditional image colorization research uses methods such as function mapping, color expansion, and reference examples, which require a lot of manual intervention and are time-consuming and labor-intensive. In recent years, with the development of deep learning, many studies have used deep convolutional neural networks to solve image colorization tasks. Deep learning-based methods have made significant progress in grayscale image colorization tasks by utilizing abundant training data. However, for near-infrared images containing a large amount of noise information, there is a lack of grayscale processing of near-infrared images, and the colorization results will have the problem of overall visual flickering of light and dark.

[0003] Existing image colorization methods propose a colorization model based on memory-enhanced learning. This model uses a query method to obtain color information learned from a training set in a memory network. Based on a threshold triplet loss, the memory network is trained unsupervised without class labels, ultimately completing the image colorization. However, this method has the following problems: First, it does not learn sufficient color information, resulting in a mottled color phenomenon in the colorization result. Second, it lacks grayscale processing for near-infrared images, and the colorization results are significantly affected by noise. Finally, although this method can handle colorization with limited data, it is not efficient and the colorization effect is poor in practical applications. Summary of the Invention

[0004] The purpose of the present invention is to provide a near-infrared image colorization method based on a two-stage conditional generative adversarial network to solve the problem that the existing deep learning-based image colorization method lacks corresponding grayscale processing for near-infrared images containing a large amount of noise information, resulting in the overall visual effect of the colored image being bright and dark flickering.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] The present invention provides a near-infrared image colorization method based on a two-stage conditional generative adversarial network, the near-infrared image colorization method based on the two-stage conditional generative adversarial network comprising:

[0007] S1: Acquire near-infrared grayscale image;

[0008] S2: Constructing grayscale preprocessing module for near-infrared images;

[0009] S3: preprocessing the near-infrared grayscale image using the grayscale preprocessing module of the near-infrared image to obtain a grayscale image;

[0010] S4: Build image colorization module;

[0011] S5: Using the image colorization module, the grayscale image is colorized to obtain a colored generated image, wherein the image colorization module is also used to discriminate the color feature vectors extracted from the colored generated image and the training image, and the discrimination result is used to train the image colorization module.

[0012] Optionally, in step S2, the grayscale preprocessing module of the near-infrared image includes a first generator and a first discriminator, and the first generator and the first discriminator are obtained by training with a first-stage loss function.

[0013] Optionally, the first generator includes a first encoder and a first decoder, the first encoder includes 7 first residual connection blocks, the first to fourth first residual connection blocks constitute a first downsampling code, and the fifth to seventh first residual connection blocks constitute a first intermediate code;

[0014] Each of the first residual connection blocks includes a first 3×3 convolutional layer, a first IN module, a first Relu activation function layer, a second 3×3 convolutional layer, a second Relu activation function layer, a first Dropout module, a first 2×2 convolutional layer, a second IN module and a third Relu activation function layer. The input of the first 3×3 convolutional layer and the first 2×2 convolutional layer is the near-infrared grayscale image and / or the output result of the previous residual connection block. The first 3×3 convolutional layer, the first IN module, the first Relu activation function layer, the second 3×3 convolutional layer and the second Relu activation function layer are connected in sequence. The first 2×2 convolutional layer, the second IN module and the third Relu activation function layer are connected in sequence. The output result of the second Relu activation function layer and the output result of the third Relu activation function layer are added on the feature channel, and the sum is used as the input of the first Dropout module. The output of the first Dropout module is used as the output result of the first residual connection block or the output result of the first encoder.

[0015] Optionally, the first decoder includes 7 first residual connection blocks with a dilated convolution structure, the first to fourth first residual connection blocks with a dilated convolution structure constitute a first intermediate decoding, and the fifth to seventh first residual connection blocks with a dilated convolution structure constitute a first upsampling encoding;

[0016] The first first residual connection block and the first first residual connection block with a dilated convolution structure are connected via a first residual connection path, and the second first residual connection block and the second first residual connection block with a dilated convolution structure are connected via a second residual connection path;

[0017] Each of the first residual connection blocks with a void convolution structure includes a first 3×3 transposed void convolution layer, a third IN module, a fourth Relu activation function layer, a second 3×3 transposed void convolution layer, a fifth Relu activation function layer, a second Dropout module, a first 2×2 transposed convolution layer, a fourth IN module and a sixth Relu activation function layer. The input of the first 3×3 transposed void convolution layer and the first 2×2 transposed convolution layer is the output result of the first encoder or the output result of the previous first residual connection block with a void convolution structure. The first 3×3 transposed void convolution layer, the third IN module, the fourth Relu activation function layer, The u activation function layer, the second 3×3 transposed void convolution layer and the fifth Relu activation function layer are connected in sequence, the first 2×2 transposed convolution layer, the fourth IN module and the sixth Relu activation function layer are connected in sequence, the output result of the fifth Relu activation function layer and the output result of the sixth Relu activation function layer are added on the feature channel, and the sum is used as the input of the second Dropout module, the output of the second Dropout module is used as the output result of the first residual connection block with a void convolution structure or the output result of the first decoder, and the output result of the first decoder is to generate a grayscale image.

[0018] Optionally, in step S2, the first discriminator includes a third 3×3 convolutional layer, a fourth 3×3 convolutional layer, a first BN module, a fifth 3×3 convolutional layer, a sixth 3×3 convolutional layer, a second BN module, a seventh 3×3 convolutional layer, a first LeakRelu activation function layer and an eighth 3×3 convolutional layer connected in sequence, the input of the third 3×3 convolutional layer is used as the input of the first discriminator, and the input content includes a generated grayscale image and a real grayscale image. The output of the eighth 3×3 convolutional layer is used as the output of the first discriminator, and its output result is a grayscale image.

[0019] Optionally, the first stage loss function includes an adversarial loss function and a pixel smoothing L1 loss function, and the first stage loss function L G for:

[0020] L G =λ1*L adv +λ2*L sL1

[0021] Among them, λ1 and λ2 are the weight coefficients of the adversarial loss function and the pixel smoothing L1 loss function, respectively. adv is the adversarial loss function and For the grayscale preprocessing module of the near-infrared image, x is the input near-infrared grayscale image, C is a one-dimensional random noise vector, G(x, C) is the output of the first generator, P data Is a noise vector that follows the x distribution, D is the discriminator’s estimate of the probability that the generated image is a real image, Indicates obedience to x~P data The mathematical expectation of the data distribution, L sL1 is the pixel smoothing L1 loss function and y represents the pixel value of the generated image, Represents the pixel value of the real image.

[0022] Optionally, in step S4, the image colorization module includes a second generator and a second discriminator, and the second generator and the second discriminator are trained based on the second stage loss function.

[0023] Optionally, the second generator includes a color feature vector, a second encoder and a second decoder, the color feature vector is connected to the second encoder, the second encoder includes 7 second residual connection blocks, the first to fourth second residual connection blocks constitute a second downsampling code, and the fifth to seventh second residual connection blocks constitute a second intermediate code;

[0024] Each of the second residual connection blocks includes a ninth 3×3 convolutional layer, a first AdaIN module, a seventh Relu activation function layer, a second 3×3 convolutional layer, an eighth Relu activation function layer, a third Dropout module, a second 2×2 convolutional layer, a second AdaIN module and a ninth Relu activation function layer, wherein the input of the ninth 3×3 convolutional layer and the second 2×2 convolutional layer is the grayscale image or the output result of the previous second residual connection block and / or the output result of the color feature vector, the ninth 3×3 convolutional layer, the first AdaIN module, the seventh Relu activation function layer, the second 3×3 convolutional layer and the eighth Relu activation function layer are connected in sequence, the second 2×2 convolutional layer, the second AdaIN module and the ninth Relu activation function layer are connected in sequence, the output result of the eighth Relu activation function layer and the output result of the ninth Relu activation function layer are added on the feature channel, and the sum is used as the third Dropout module. The input of the pout module is used, and the output of the third Dropout module is used as the output result of the second residual connection block or the output result of the second encoder; the second decoder includes 7 second residual connection blocks with a void convolution structure, the first to fourth second residual connection blocks with a void convolution structure constitute a second intermediate decoding, and the fifth to seventh second residual connection blocks with a void convolution structure constitute a second upsampling encoding; the first second residual connection block and the first second residual connection block with a void convolution structure are connected via a third residual connection path, the second second residual connection block and the second second residual connection block with a void convolution structure are connected via a fourth residual connection path, the third second residual connection block and the third second residual connection block with a void convolution structure are connected via a fifth residual connection path, and the fourth second residual connection block and the fourth second residual connection block with a void convolution structure are connected via a sixth residual connection path;Each of the second residual connection blocks with a void convolution structure includes a third 3×3 transposed void convolution layer, a third AdaIN module, a tenth Relu activation function layer, a fourth 3×3 transposed void convolution layer, an eleventh Relu activation function layer, a fourth Dropout module, a second 2×2 transposed convolution layer, a fourth AdaIN module and a twelfth Relu activation function layer, the input of the third 3×3 transposed void convolution layer and the second 2×2 transposed convolution layer is the output result of the second encoder or the output result of the previous second residual connection block with a void convolution structure, the third 3×3 transposed void convolution layer, the third AdaIN module, the tenth Re The Relu activation function layer, the fourth 3×3 transposed dilated convolution layer, and the eleventh Relu activation function layer are sequentially connected; the second 2×2 transposed convolution layer, the fourth AdaIN module, and the twelfth Relu activation function layer are sequentially connected; the output result of the eleventh Relu activation function layer and the output result of the twelfth Relu activation function layer are added on the feature channel, and the sum is used as the input of the fourth Dropout module; the output of the fourth Dropout module is used as the output result of the second residual connection block with a dilated convolution structure or the output result of the second decoder; the output result of the second decoder is a colored image.

[0025] Optionally, the second discriminator includes an eleventh 3×3 convolutional layer, a twelfth 3×3 convolutional layer, a third BN module, a thirteenth 3×3 convolutional layer, a fourteenth 3×3 convolutional layer, a fourth BN module, a fifteenth 3×3 convolutional layer, a second LeakRelu activation function layer, a sixteenth 3×3 convolutional layer and a Sigmoid activation function layer connected in sequence, the input of the eleventh 3×3 convolutional layer is used as the input of the second discriminator, the input content includes a colored image and a real colored image, the output result of the sixteenth 3×3 convolutional layer is a colored generated image, and the Sigmoid activation function layer is used to discriminate and output the colored generated image and the real colored image.

[0026] Optionally, the second stage loss function includes an adversarial loss function, a pixel smoothing L1 loss function and a style loss function, and the second stage loss function L C for:

[0027] L C =λ3*L adv +λ4*L sL1 +λ5*L s

[0028] Among them, λ3, λ4 and λ5 represent the weight coefficients of the adversarial loss function, pixel smoothing L1 loss function and style loss function respectively, L advis the adversarial loss function and For the image colorization module, x is the input visible light grayscale image, C is the color feature, G(x,C) is the output of the second generator, y is the real color image, P data is a noise vector that follows the x distribution, D is the discriminator’s estimate of the probability that the generated image is a real image, and L sL1 is the pixel smoothing L1 loss function and y represents the pixel value of the generated image, Represents the pixel value of the real image, L s is the style loss function.

[0029] The present invention has the following beneficial effects:

[0030] 1. The present invention constructs a grayscale preprocessing module for near-infrared images, which is based on a conditional generative adversarial network. It uses an improved Bilateral pix2pix as a generator network, inputs a near-infrared grayscale image, and bilaterally transfers the grayscale texture detail features of the near-infrared image through the bilateral residual connection block of the encoder in the generator network, thereby eliminating the noise and blurring information in the near-infrared grayscale image. A purer visible light grayscale image is restored by designing a hollow convolution structure in the decoder. A block discriminator with a Patch structure discriminates the generated result from the real image, thereby continuously improving the quality of the generated grayscale image.

[0031] 2. The present invention designs an image colorization module, which is based on a conditional generative adversarial network. The module feeds the visible light grayscale image obtained above and the color feature vector extracted from the training image into the network generator. The generator is composed of BilateralRes-Unet, which performs bilateral transfer fusion on the color feature information and the visible light grayscale image, and obtains the colored generated image through upsampling decoding recovery. The colored image and the true color image are discriminated using a fully convolutional discriminator, both of which continuously improve the quality of the colored image.

[0032] 3. The present invention constructs a two-stage fusion loss function, uses different loss functions for the near-infrared grayscale preprocessing module and the image colorization module, and determines the weights of different losses to improve the authenticity of the grayscale image generated in the first stage and the color style correctness of the colored image in the second stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flowchart of the near-infrared image colorization method based on a two-stage conditional generative adversarial network of the present invention;

[0034] Figure 2 Schematic diagram of the structure of the grayscale preprocessing module of the near-infrared image of the present invention;

[0035] Figure 3 Schematic diagram of the structure of the first residual connection block;

[0036] Figure 4 Schematic diagram of the structure of the first residual connection block with a dilated convolution structure;

[0037] Figure 5 Schematic diagram of the structure of the first discriminator;

[0038] Figure 6 This is a schematic diagram of the structure of the image coloring module;

[0039] Figure 7 Schematic diagram of the structure of the second residual connection block;

[0040] Figure 8 Schematic diagram of the structure of the second residual connection block with a void convolution structure;

[0041] Figure 9 Schematic diagram of the structure of the second discriminator. DETAILED DESCRIPTION

[0042] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0043] Example

[0044] The present invention provides a near-infrared image colorization method based on a two-stage conditional generative adversarial network. Figure 1 As shown, the near-infrared image colorization method based on the two-stage conditional generative adversarial network includes:

[0045] S1: Acquire near-infrared grayscale image;

[0046] S2: Constructing grayscale preprocessing module for near-infrared images;

[0047] The grayscale preprocessing module provided by the present invention can eliminate noise and blur in near-infrared grayscale images, thereby obtaining a visible light grayscale image that is more suitable for subsequent colorization. Specifically, the grayscale preprocessing module for near-infrared images provided by the present invention includes a first generator and a first discriminator, which are trained using a first-stage loss function.

[0048] Alternatively, refer to Figure 2 As shown, the first generator includes a first encoder and a first decoder, the first encoder includes 7 first residual connection blocks, the 1st to 4th first residual connection blocks constitute a first downsampling code, and the 5th to 7th first residual connection blocks constitute a first intermediate code.

[0049] refer to Figure 3As shown, each of the first residual connection blocks includes a first 3×3 convolutional layer, a first IN module, a first Relu activation function layer, a second 3×3 convolutional layer, a second Relu activation function layer, a first Dropout module, a first 2×2 convolutional layer, a second IN module and a third Relu activation function layer. The input of the first 3×3 convolutional layer and the first 2×2 convolutional layer is the near-infrared grayscale image and / or the output result of the previous residual connection block. The first 3×3 convolutional layer, the first IN module, the first Relu activation function layer, the second 3×3 convolutional layer and the second Relu activation function layer are connected in sequence. The first 2×2 convolutional layer, the second IN module and the third Relu activation function layer are connected in sequence. The output result of the second Relu activation function layer and the output result of the third Relu activation function layer are added on the feature channel, and the sum is used as the input of the first Dropout module. The output of the first Dropout module is used as the output result of the first residual connection block or the output result of the first encoder.

[0050] Here, when the current first residual connection block is the first first residual connection block of the first encoder, the input of the first 3×3 convolutional layer and the first 2×2 convolutional layer is the near-infrared grayscale image; when the current first residual connection block is any one of the second to seventh first residual connection blocks, the input of the first 3×3 convolutional layer and the first 2×2 convolutional layer is the output result of the previous first residual connection block. Correspondingly, when the current first residual connection block is any one of the first to sixth first residual connection blocks of the first encoder, the output of the first Dropout module is the output result of the current first residual connection block; and when the current first residual connection block is the seventh first residual connection block of the first encoder, the output of the first Dropout module is the output result of the first encoder.

[0051] Alternatively, refer to Figure 2 As shown, the first decoder includes 7 first residual connection blocks with a void convolution structure, the 1st to 4th first residual connection blocks with a void convolution structure constitute the first intermediate decoding, and the 5th to 7th first residual connection blocks with a void convolution structure constitute the first upsampling encoding.

[0052] The first first residual connection block and the first first residual connection block with a void convolution structure are connected through a first residual connection path, and the second first residual connection block and the second first residual connection block with a void convolution structure are connected through a second residual connection path.

[0053] refer to Figure 4As shown, each of the first residual connection blocks with a void convolution structure includes a first 3×3 transposed void convolution layer, a third IN module, a fourth Relu activation function layer, a second 3×3 transposed void convolution layer, a fifth Relu activation function layer, a second Dropout module, a first 2×2 transposed convolution layer, a fourth IN module and a sixth Relu activation function layer. The input of the first 3×3 transposed void convolution layer and the first 2×2 transposed convolution layer is the output result of the first encoder or the output result of the previous first residual connection block with a void convolution structure. The first 3×3 transposed void convolution layer, the third IN module, the fourth R The elu activation function layer, the second 3×3 transposed void convolution layer and the fifth Relu activation function layer are connected in sequence, the first 2×2 transposed convolution layer, the fourth IN module and the sixth Relu activation function layer are connected in sequence, the output result of the fifth Relu activation function layer and the output result of the sixth Relu activation function layer are added on the feature channel, and the sum is used as the input of the second Dropout module, the output of the second Dropout module is used as the output result of the first residual connection block with a void convolution structure or the output result of the first decoder, and the output result of the first decoder is to generate a grayscale image.

[0054] Alternatively, refer to Figure 5 As shown, the first discriminator includes a third 3×3 convolutional layer, a fourth 3×3 convolutional layer, a first BN module, a fifth 3×3 convolutional layer, a sixth 3×3 convolutional layer, a second BN module, a seventh 3×3 convolutional layer, a first LeakRelu activation function layer and an eighth 3×3 convolutional layer connected in sequence. The input of the third 3×3 convolutional layer is used as the input of the first discriminator, and the input content includes a generated grayscale image and a real grayscale image. The output of the eighth 3×3 convolutional layer is used as the output of the first discriminator, and its output result is a grayscale image.

[0055] The eighth 3×3 convolutional layer outputs a 4×4 discriminant matrix with a patch discriminant structure, distinguishing between the generated grayscale image and the true grayscale image within the local receptive field. This continuously improves the discriminant performance and generation effect of both the generated grayscale image and the true grayscale image, resulting in higher-quality grayscale images.

[0056] The present invention uses residual connection blocks with the ability to transfer shallow-layer information to deep layers of the network in both the encoding-decoding cascade part and the downsampling part, and uses a dilated convolutional layer in the upsampling decoding part to perform bilateral transfer of the grayscale texture detail features of the near-infrared image, eliminating the bright and dark noise and blurred information in the near-infrared grayscale image, and obtaining a grayscale image closer to real visible light through the multi-scale decoding structure.

[0057] Optionally, when constructing the grayscale preprocessing module of the near-infrared image, it is necessary to consider the adversarial loss and the pixel smoothing L1 loss in order to generate an output as realistic as possible and retain the authenticity of the pixels; therefore, the first-stage loss function of the present invention includes the adversarial loss function and the pixel smoothing L1 loss function, and the first-stage loss function L G for:

[0058] L G =λ1*L adv +λ2*L sL1

[0059] Among them, λ1 and λ2 are the weight coefficients of the adversarial loss function and the pixel smoothing L1 loss function respectively. The first generator is to minimize the target loss. On the contrary, the first discriminator is to maximize the target loss. The two compete with each other to continuously improve the generation effect. The adversarial loss function L adv for For the grayscale preprocessing module of the near-infrared image, x is the input near-infrared grayscale image, C is a one-dimensional random noise vector, G(x, C) is the output of the first generator, P data Is a noise vector that follows the x distribution, D is the discriminator’s estimate of the probability that the generated image is a real image, Indicates obedience to x~P data The mathematical expectation of the data distribution, in order to minimize the content deviation between the generated grayscale image generated by the first generator and the real grayscale image, the distance between the pixel values of the generated grayscale image and the real grayscale image is smoothed L1 constraint, the pixel smoothing L1 loss function L sL1 for y represents the pixel value of the generated image, Represents the pixel value of the real image.

[0060] S3: preprocessing the near-infrared grayscale image using the grayscale preprocessing module of the near-infrared image to obtain a grayscale image;

[0061] S4: Build image colorization module;

[0062] Optionally, the image colorization module includes a second generator and a second discriminator, and the second generator and the second discriminator are trained based on a second-stage loss function.

[0063] Alternatively, refer to Figure 6 As shown, the second generator includes a color feature vector, a second encoder and a second decoder, the color feature vector is connected to the second encoder, the second encoder includes 7 second residual connection blocks, the 1st to 4th second residual connection blocks constitute a second downsampling code, and the 5th to 7th second residual connection blocks constitute a second intermediate code;

[0064] refer to Figure 7 As shown, each of the second residual connection blocks includes a ninth 3×3 convolutional layer, a first AdaIN module, a seventh Relu activation function layer, a second 3×3 convolutional layer, an eighth Relu activation function layer, a third Dropout module, a second 2×2 convolutional layer, a second AdaIN module and a ninth Relu activation function layer, the input of the ninth 3×3 convolutional layer and the second 2×2 convolutional layer is the grayscale image or the output result of the previous second residual connection block and / or the output result of the color feature vector, the ninth 3×3 convolutional layer, the first AdaIN module, the seventh Relu activation function layer, the second 3×3 convolutional layer and the eighth Relu activation function layer are connected in sequence, the second 2×2 convolutional layer, the second AdaIN module and the ninth Relu activation function layer are connected in sequence, the output result of the eighth Relu activation function layer and the output result of the ninth Relu activation function layer are added on the feature channel, and the sum is used as the input of the third Dropout module, and the output of the third Dropout module is used as the output result of the second residual connection block or the output result of the second encoder.

[0065] Here, when the current second residual connection block is the first first residual connection block of the first encoder, the input of the ninth 3×3 convolutional layer and the second 2×2 convolutional layer is the output result of the grayscale image and the color feature vector; when the current second residual connection block is any one of the second to seventh second residual connection blocks, the input of the ninth 3×3 convolutional layer and the second 2×2 convolutional layer is the output result of the previous second residual connection block. Correspondingly, when the current second residual connection block is any one of the first to sixth second residual connection blocks of the second encoder, the output of the third Dropout module is the output result of the current second residual connection block; and when the current second residual connection block is the seventh first residual connection block of the second encoder, the output of the third Dropout module is the output result of the second encoder.

[0066] refer to Figure 8As shown, the second decoder includes 7 second residual connection blocks with a dilated convolution structure, the 1st to 4th second residual connection blocks with a dilated convolution structure constitute a second intermediate decoding, and the 5th to 7th second residual connection blocks with a dilated convolution structure constitute a second upsampling encoding; the first second residual connection block and the first second residual connection block with a dilated convolution structure are connected via a third residual connection path, and the second second residual connection block and the second second residual connection block with a dilated convolution structure are connected via a fourth residual connection path. The third second residual connection block and the third second residual connection block with a void convolution structure are connected through a fifth residual connection path, and the fourth second residual connection block and the fourth second residual connection block with a void convolution structure are connected through a sixth residual connection path; each of the second residual connection blocks with a void convolution structure includes a third 3×3 transposed void convolution layer, a third AdaIN module, a tenth Relu activation function layer, a fourth 3×3 transposed void convolution layer, an eleventh Relu activation function layer, a fourth Dr Dropout module, a second 2×2 transposed convolution layer, a fourth AdaIN module and a twelfth Relu activation function layer, the input of the third 3×3 transposed hole convolution layer and the second 2×2 transposed convolution layer is the output result of the second encoder or the output result of the previous second residual connection block with a hole convolution structure, the third 3×3 transposed hole convolution layer, the third AdaIN module, the tenth Relu activation function layer, the fourth 3×3 transposed hole convolution layer and the eleventh Relu activation function layer are connected in sequence, the second 2×2 transposed convolution layer, the fourth AdaIN module and the twelfth Relu activation function layer are connected in sequence, the output result of the eleventh Relu activation function layer and the output result of the twelfth Relu activation function layer are added on the feature channel, and the sum is used as the input of the fourth Dropout module, the output of the fourth Dropout module is used as the output result of the second residual connection block with a hole convolution structure or the output result of the second decoder, and the output result of the second decoder is a colored image.

[0067] Alternatively, refer to Figure 9As shown, the second discriminator includes an eleventh 3×3 convolutional layer, a twelfth 3×3 convolutional layer, a third BN module, a thirteenth 3×3 convolutional layer, a fourteenth 3×3 convolutional layer, a fourth BN module, a fifteenth 3×3 convolutional layer, a second LeakRelu activation function layer, a sixteenth 3×3 convolutional layer and a Sigmoid activation function layer connected in sequence. The input of the eleventh 3×3 convolutional layer is used as the input of the second discriminator, and the input content includes a colored image and a true colored image. The output result of the sixteenth 3×3 convolutional layer is a colored generated image, and the Sigmoid activation function layer is used to discriminate and output the colored generated image and the true colored image.

[0068] Optionally, the second stage loss function includes an adversarial loss function, a pixel smoothing L1 loss function and a style loss function, and the second stage loss function L C for:

[0069] L C =λ3*L adv +λ4*L sL1 +λ5*L s

[0070] Among them, λ3, λ4 and λ5 represent the weight coefficients of the adversarial loss function, pixel smoothing L1 loss function and style loss function respectively, L adv is the adversarial loss function and For the image colorization module, x is the input visible light grayscale image, C is the color feature, G(x,C) is the output of the second generator, y is the real color image, P data is a noise vector that follows the x distribution, D is the discriminator’s estimate of the probability that the generated image is a real image, and L sL1 is the pixel smoothing L1 loss function and y represents the pixel value of the generated image, Represents the pixel value of the real image, L s is the style loss function.

[0071] In addition, in the design of the image colorization module, in addition to hoping that the colored image has the correct content, the present invention also hopes that the generated image has better texture details, because the above loss does not take into account the subtle texture differences between the two images, and only focuses on the overall structure of the image content. Therefore, the present invention uses style loss to enhance the performance of texture details. Here, only experiments are conducted on the L channel of the real color image and the generated image. The present invention adopts the idea of matching the Gram matrix (representing feature correlation) extracted from certain layers in the pre-trained classification network (VGG19). The Gram matrix The definition is as follows:

[0072]

[0073] Here N l represents the number of feature maps in the network layer l, Represents the output of the i-th filter at position k in layer l. The present invention uses three layers (relu2_2, relu3_2, relu4_2) of the VGG19 network to form the loss.

[0074] S5: Using the image colorization module, the grayscale image is colorized to obtain a colored generated image, wherein the image colorization module is also used to discriminate the color feature vectors extracted from the colored generated image and the training image, and the discrimination result is used to train the image colorization module.

[0075] The present invention discloses a near-infrared image colorization method based on a two-stage conditional generative adversarial network, comprising the following steps: constructing a grayscale preprocessing module for the near-infrared image, which is composed of a conditional generative adversarial network and has a first generator and a first discriminator; the near-infrared grayscale image is input into a generator based on a bilateral pix2pix structure in the network; the first generator uses a first residual connection block with the ability to transfer shallow information to the deep layer of the network in both the encoding-decoding cascade part and the first downsampling part; a hole convolution layer is used in the first upsampling decoding part to bilaterally transfer the grayscale texture detail features of the near-infrared image, eliminate bright and dark noise and fuzzy information in the near-infrared grayscale image, obtain a purer visible light grayscale image through the multi-scale structure of decoding; the first discriminator with a patch structure discriminates between the generated grayscale image and the real grayscale image, and both continuously improve the discrimination performance and generation effect. An image colorization module was designed based on a conditional generative adversarial network. This module extracts primary color information from the training image, obtaining a one-dimensional color feature vector as the colorization condition. This visible light grayscale image and color feature vector are fed into the network's second generator, a Bilateral Res-Unet. Building on the structure of the first generator, the second generator incorporates color vector information and increases the number of symmetrically distributed bilateral residual connection blocks. This color feature information is bilaterally transferred and fused with the visible light grayscale image. A second upsampling decoding process restores the generated colorized image. A fully convolutional discriminator is used to distinguish the colored image from the true color image. The interplay between the generator and discriminator continuously improves the colorized image quality. A two-stage fusion loss function was constructed. The near-infrared grayscale preprocessing module uses an adversarial loss and a pixel-smoothing L1 loss to enhance the realism of the generated grayscale image. The image colorization module uses an adversarial loss, a pixel-smoothing L1 loss, and a style loss to improve the color style accuracy of the colored image.

[0076] In this article, the structures of the first residual connection path and the second residual connection path are the same, and are the same as the structure of the first intermediate code. The existence of the first residual connection path and the second residual connection path can improve the efficiency of bilateral grayscale feature transmission; the structures of the third residual connection path to the sixth residual connection path are the same, and are the same as the structure of the second intermediate code. Since the AdaIN module has dual color information, the existence of the third residual connection path to the sixth residual connection path can improve the quality of bilateral color feature transmission.

[0077] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A near-infrared image colorization method based on a two-stage conditional generative adversarial network, characterized in that: The near-infrared image colorization method based on the two-stage conditional generative adversarial network includes: S1: Acquire near-infrared grayscale image; S2: Constructing grayscale preprocessing module for near-infrared images; In step S2, the grayscale preprocessing module of the near-infrared image includes a first generator and a first discriminator, and the first generator and the first discriminator are obtained by training the first-stage loss function; The first generator includes a first encoder and a first decoder, the first encoder includes 7 first residual connection blocks, the first to fourth first residual connection blocks constitute a first downsampling code, and the fifth to seventh first residual connection blocks constitute a first intermediate code; The first decoder includes 7 first residual connection blocks with a dilated convolution structure, the first to fourth first residual connection blocks with a dilated convolution structure constitute a first intermediate decoding, and the fifth to seventh first residual connection blocks with a dilated convolution structure constitute a first upsampling encoding; S3: preprocessing the near-infrared grayscale image using the grayscale preprocessing module of the near-infrared image to obtain a grayscale image; S4: Build image colorization module; In step S4, the image colorization module includes a second generator and a second discriminator, wherein the second generator and the second discriminator are trained based on the second stage loss function; The second generator includes a color feature vector, a second encoder, and a second decoder, the color feature vector is connected to the second encoder, the second encoder includes 7 second residual connection blocks, the first to fourth second residual connection blocks constitute a second downsampling code, and the fifth to seventh second residual connection blocks constitute a second intermediate code; The second discriminator includes an eleventh 3×3 convolutional layer, a twelfth 3×3 convolutional layer, a third BN module, a thirteenth 3×3 convolutional layer, a fourteenth 3×3 convolutional layer, a fourth BN module, a fifteenth 3×3 convolutional layer, a second LeakRelu activation function layer, a sixteenth 3×3 convolutional layer and a Sigmoid activation function layer connected in sequence, the input of the eleventh 3×3 convolutional layer is used as the input of the second discriminator, the input content includes a colored image and a true colored image, the output result of the sixteenth 3×3 convolutional layer is a colored generated image, and the Sigmoid activation function layer is used to discriminate and output the colored generated image and the true colored image; S5: Using the image colorization module, the grayscale image is colorized to obtain a colored generated image, wherein the image colorization module is also used to discriminate the color feature vectors extracted from the colored generated image and the training image, and the discrimination result is used to train the image colorization module.

2. The near-infrared image colorization method based on a two-stage conditional generative adversarial network according to claim 1 is characterized in that: Each of the first residual connection blocks includes a first 3×3 convolutional layer, a first IN module, a first Relu activation function layer, a second 3×3 convolutional layer, a second Relu activation function layer, a first Dropout module, a first 2×2 convolutional layer, a second IN module, and a third Relu activation function layer. The input of the first 3×3 convolutional layer and the first 2×2 convolutional layer is the near-infrared grayscale image and / or the output result of the previous residual connection block. The first 3×3 convolutional layer, the first IN module, the first Relu activation function layer, the second 3×3 convolutional layer, and the second Relu activation function layer are connected in sequence. The first 2×2 convolutional layer, the second IN module, and the third Relu activation function layer are connected in sequence. The output result of the second Relu activation function layer and the output result of the third Relu activation function layer are added on the feature channel, and the sum is used as the input of the first Dropout module. The output of the first Dropout module is used as the output result of the first residual connection block or the output result of the first encoder.

3. The near-infrared image colorization method based on a two-stage conditional generative adversarial network according to claim 2 is characterized in that: The first first residual connection block and the first first residual connection block with a dilated convolution structure are connected via a first residual connection path, and the second first residual connection block and the second first residual connection block with a dilated convolution structure are connected via a second residual connection path; Each of the first residual connection blocks with a dilated convolution structure includes a first 3×3 transposed dilated convolution layer, a third IN module, a fourth Relu activation function layer, a second 3×3 transposed dilated convolution layer, a fifth Relu activation function layer, a second Dropout module, a first 2×2 transposed convolution layer, a fourth IN module, and a sixth Relu activation function layer. The input of the first 3×3 transposed dilated convolution layer and the first 2×2 transposed convolution layer is the output result of the first encoder or the output result of the previous first residual connection block with a dilated convolution structure. The first 3×3 transposed dilated convolution layer, the third IN module, the fourth Relu activation function layer, the second 3×3 transposed dilated convolution layer, and the fifth Relu activation function layer are connected in sequence. The first 2×2 transposed convolution layer, the fourth IN module, and the sixth Relu activation function layer are connected in sequence. The output result of the fifth Relu activation function layer and the sixth Relu The output results of the activation function layer are added on the feature channel, and the sum is used as the input of the second Dropout module. The output of the second Dropout module is used as the output result of the first residual connection block with a void convolution structure or the output result of the first decoder. The output result of the first decoder is to generate a grayscale image.

4. The near-infrared image colorization method based on a two-stage conditional generative adversarial network according to claim 1 is characterized in that: In step S2, the first discriminator includes a third 3×3 convolutional layer, a fourth 3×3 convolutional layer, a first BN module, a fifth 3×3 convolutional layer, a sixth 3×3 convolutional layer, a second BN module, a seventh 3×3 convolutional layer, a first LeakRelu activation function layer and an eighth 3×3 convolutional layer connected in sequence. The input of the third 3×3 convolutional layer is used as the input of the first discriminator, and the input content includes a generated grayscale image and a real grayscale image. The output of the eighth 3×3 convolutional layer is used as the output of the first discriminator, and its output result is a grayscale image.

5. The near-infrared image colorization method based on a two-stage conditional generative adversarial network according to any one of claims 1 to 4, characterized in that: The first stage loss function includes an adversarial loss function and a pixel smoothing L1 loss function, and the first stage loss function for: in, and are the weight coefficients of the adversarial loss function and the pixel smoothing L1 loss function, respectively. is the adversarial loss function and , for the grayscale preprocessing module of the near-infrared image, is the input near-infrared grayscale image, is a one-dimensional random noise vector, is the output of the first generator, It means obedience x The noise vector of the distribution, D It is the estimate of the probability that the discriminator identifies the generated image as a real image, Express obedience The mathematical expectation of the data distribution, is the pixel smoothing L1 loss function and , y represents the pixel value of the generated image, Represents the pixel value of the real image.

6. The near-infrared image colorization method based on a two-stage conditional generative adversarial network according to claim 1, characterized in that: Each of the second residual connection blocks includes a ninth 3×3 convolutional layer, a first AdaIN module, a seventh Relu activation function layer, a second 3×3 convolutional layer, an eighth Relu activation function layer, a third Dropout module, a second 2×2 convolutional layer, a second AdaIN module and a ninth Relu activation function layer, the input of the ninth 3×3 convolutional layer and the second 2×2 convolutional layer is the grayscale image or the output result of the previous second residual connection block and / or the output result of the color feature vector, the ninth 3×3 convolutional layer, the first AdaIN module, the seventh Relu activation function layer, the second 3×3 convolutional layer and the eighth Relu activation function layer are connected in sequence, the second 2×2 convolutional layer, the second AdaIN module and the ninth Relu activation function layer are connected in sequence, the eighth Relu The output result of the activation function layer and the output result of the ninth Relu activation function layer are added on the feature channel, and the sum is used as the input of the third Dropout module, and the output of the third Dropout module is used as the output result of the second residual connection block or the output result of the second encoder; The second decoder includes 7 second residual connection blocks with a dilated convolution structure, the first to fourth second residual connection blocks with a dilated convolution structure constitute a second intermediate decoding, and the fifth to seventh second residual connection blocks with a dilated convolution structure constitute a second upsampling encoding; The first second residual connection block and the first second residual connection block with a dilated convolution structure are connected via a third residual connection path, the second second residual connection block and the second second residual connection block with a dilated convolution structure are connected via a fourth residual connection path, the third second residual connection block and the third second residual connection block with a dilated convolution structure are connected via a fifth residual connection path, and the fourth second residual connection block and the fourth second residual connection block with a dilated convolution structure are connected via a sixth residual connection path; Each of the second residual connection blocks with a dilated convolution structure includes a third 3×3 transposed dilated convolution layer, a third AdaIN module, a tenth Relu activation function layer, a fourth 3×3 transposed dilated convolution layer, an eleventh Relu activation function layer, a fourth Dropout module, a second 2×2 transposed convolution layer, a fourth AdaIN module and a twelfth Relu activation function layer, the input of the third 3×3 transposed dilated convolution layer and the second 2×2 transposed convolution layer is the output result of the second encoder or the output result of the previous second residual connection block with a dilated convolution structure, the third 3×3 transposed dilated convolution layer, the third AdaIN module, the tenth Relu activation function layer, the fourth 3×3 transposed dilated convolution layer and the eleventh Relu activation function layer are connected in sequence, the second 2×2 transposed convolution layer, the fourth AdaIN module and the twelfth Relu activation function layer are connected in sequence, the output result of the eleventh Relu activation function layer and the twelfth Relu The output results of the activation function layer are added on the feature channel, and the sum is used as the input of the fourth Dropout module. The output of the fourth Dropout module is used as the output result of the second residual connection block with a void convolution structure or the output result of the second decoder. The output result of the second decoder is a colored image.

7. The near-infrared image colorization method based on a two-stage conditional generative adversarial network according to claim 6, characterized in that: The second stage loss function includes an adversarial loss function, a pixel smoothing L1 loss function and a style loss function, and the second stage loss function for: in, 、 and They represent the weight coefficients of the adversarial loss function, pixel smoothing L1 loss function, and style loss function, respectively. is the adversarial loss function and , for the image coloring module, is the input visible light grayscale image, is the color feature, is the output of the second generator, is a true color image, It means obedience x The noise vector of the distribution, D It is the estimate of the probability that the discriminator identifies the generated image as a real image, is the pixel smoothing L1 loss function and , y represents the pixel value of the generated image, represents the pixel value of the real image, is the style loss function.

Images