A supervised generative adversarial network and capsule network infrared image enhancement method

Abstract

The present invention belongs to the technical field of infrared image enhancement methods, and specifically relates to an infrared image enhancement method with a supervised generative adversarial network and a capsule network, comprising the following steps: introducing conditional variables, adopting different convolution kernels to perform variable step-size movement to convolve the image according to the type of image; using the capsule network as the framework of the discriminator in the generative adversarial network; placing the improved Res2net network module in the convolution layer of the generator; and redesigning the reconstruction layer in the generator network. The present invention combines the advantage of the generative adversarial network in generating new data with the vector characteristics of the capsule network, so that the generated image is similar to the real image while having more texture and details. In addition, the present invention redesigns the reconstruction layer in the generator network to solve the problems of parameter redundancy and slow training caused by too many fully connected layers.

Description

Technical Field

[0001] The present invention belongs to the technical field of infrared image enhancement methods, and specifically relates to an infrared image enhancement method using a supervised generative adversarial network and a capsule network. Background Art

[0002] Numerous researchers have developed algorithms for enhancing visible and infrared images, and while visible image enhancement algorithms offer promising results, visible image target detection is subject to environmental constraints. Consequently, infrared imagery emerged as a necessity, offering superior target detection in specialized environments. The increasing use of drones in civilian applications, while providing convenience, also significantly impacts air traffic control, leading to a growing demand for drone countermeasures. Due to their small size and relative distance to infrared equipment, drones may appear smaller within the imaging frame and be significantly affected by infrared noise. Therefore, improving the recognition rate of small, weak infrared targets is a pressing need. Target recognition requires obtaining high-definition infrared images containing these small, weak targets. Summary of the Invention

[0003] In response to the technical problems that the above-mentioned existing drones may appear smaller in the imaging frame and are severely interfered by infrared noise, the present invention provides an infrared image enhancement method using a supervised generative adversarial network and a capsule network, which has high target recognition rate, good texture and detail of the generated image, and fast training speed.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A method for infrared image enhancement using a supervised generative adversarial network and a capsule network comprises the following steps:

[0006] S1. Introduce conditional variables and use different convolution kernels to perform variable step-size movement to convolve the image according to the type of image;

[0007] S2. Using capsule networks as the discriminator framework in generative adversarial networks, combining the advantages of generative adversarial networks in generating new data with the vector nature of capsule networks, we can make the generated images similar to real images while also having more texture and details.

[0008] S3. Place the improved Res2net network module after the convolutional layer in the generator, and use the module's stronger feature extraction capability to effectively extract detailed features in the image;

[0009] S4. Redesign the reconstruction layer in the generator network to solve the problems of parameter redundancy and slow training caused by too many fully connected layers.

[0010] The method of introducing the conditional variable in S1 is: according to the size of the image to be processed and the size of the object contained, the conditional variable sets different convolution kernels and convolution steps, thereby reducing the processing time and enhancing the feature extraction of the region of interest.

[0011] The method of using the capsule network as the framework of the discriminator in the generative adversarial network in S2 is as follows: the discriminator in the generative adversarial network is designed with the capsule network as the framework, a convolution kernel of size M*M is set in the convolution layer of the discriminator, and the step size is set to a1; the values ​​of a1,M are set according to the actual situation of the image in a supervised situation, and the convolution layer is followed by the PrimaryCaps layer, the DigitCaps layer, and the fully connected layer.

[0012] The method for extracting detailed features in the image in S3 is as follows: a Res2Net network module is introduced after the convolution layer in the capsule network framework, and after the convolution layer preprocesses the image, multi-scale features are further extracted from the target area without increasing the computational load;

[0013] The method for extracting multi-scale features from the target area is:

[0014] The feature map after 1*1 convolution is divided into N modules, defined as X i , (i=1,2,…N), each feature block has the same scale size. Except for X1, other submodules have 3*3 convolution kernels, defined as K i (), its output is Y i , submodule X i and K i-1 () add, then input to K i (), the Y i for:

[0015]

[0016] The method for redesigning the reconstruction layer in the generator network in S4 is as follows: reconstructing the fully connected layer using a deconvolution network, processing the vector output by the digital capsule layer, retaining only the vector corresponding to a correctly output capsule, passing it through a fully connected layer, and then using the BatchNormalization optimization method. After the ReLU activation function, a feature map is obtained, and finally a deconvolution operation is performed using a fixed-size convolution kernel to obtain a reconstructed generated image.

[0017] The method for obtaining the reconstructed generated image is as follows: processing the vector output by the digital capsule layer to retain only the vector corresponding to the correctly output capsule; passing it through a fully connected layer, and then using the BatchNormalization optimization method and the ReLU activation function to obtain a feature map; finally, using a fixed-size convolution kernel to perform a deconvolution operation to obtain the reconstructed generated image.

[0018] The BatchNormalization optimization method is to keep the input of the middle layer of the neural network with a standard normal distribution with a mean of 0 and a variance of 1. The calculation formula is as follows:

[0019]

[0020]

[0021]

[0022]

[0023] The μ B represents the batch training mean, x i is the i-th input data, m is the number of batch training samples, represents the variance, The input data is normalized, and the parameters γ and β are trainable parameters to maintain the original distribution of the input data.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention enhances infrared images containing small and weak targets to improve image clarity. The present invention introduces conditional variables, so that the network can supervise and enhance the input image area while shortening the processing time; introduces the Res2Net module after the convolution layer to extract multi-scale features of the image; reconstructs the capsule network decoder, uses a deconvolution network instead of a multi-layer fully connected network, reduces redundant parameters to reduce the amount of network calculation; and finally obtains the generated enhanced image. The present invention combines the advantage of generative adversarial networks in generating new data and the vector characteristics of capsule networks, so that the generated image is similar to the real image while having more texture and details. In addition, the present invention redesigns the reconstruction layer in the generator network to solve the problems of parameter redundancy and slow training caused by too many fully connected layers. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can, without inventive effort, derive other implementation drawings based on the provided drawings.

[0027] The structures, proportions, sizes, etc. illustrated in this specification are intended solely to complement the contents disclosed herein and to facilitate understanding and reading by persons skilled in the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall remain within the scope of the technical contents disclosed herein.

[0028] Figure 1 It is a schematic diagram of the present invention used for enhancing images containing infrared dim small targets.

[0029] Figure 2 This is the res2net network application location map of the present invention.

[0030] Figure 3 This is a schematic diagram of the reconstruction of the fully connected layer of the capsule network of the present invention.

[0031] Figure 4 Schematic diagram of image enhancement results of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of this application, not all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following embodiments are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0034] In this embodiment, if Figure 1 As shown in FIG, a supervised generative adversarial network combined with a capsule network is used to enhance infrared images containing small objects, including:

[0035] An infrared image enhancement method based on generative adversarial network is constructed. Its general framework consists of a generator and a discriminator, and a conditional variable is introduced. Different convolution kernels are used to perform variable step-size convolution on the image according to the type of image, which can effectively shorten the processing time and train a better generator.

[0036] The discriminator is constructed based on the capsule network. The introduction of the capsule network has the advantage of good vector properties, which enables the generator to generate images with higher resolution based on the original image.

[0037] like Figure 2 As shown in the figure, the Res2net network is inserted in the middle of the generator convolutional layer to utilize its excellent feature extraction ability to enhance the feature extraction range of Real images.

[0038] The original fully connected layer is reconstructed to solve the problems of parameter redundancy and slow training caused by too many fully connected layers.

[0039] Based on the same inventive concept, Figure 3 As shown, in this embodiment, a supervised generative adversarial network combined with a capsule network is designed to enhance infrared images containing small and weak targets. The specific steps are as follows:

[0040] Step 101: Data preparation: According to the target requirements, this design uses the public infrared weak target data set on the Internet for processing.

[0041] First, the images are cropped into a uniform size of 256*256, and the images in different scenes are classified. The number of categories is set according to the specific image type.

[0042] Step 102: Introduce the conditional variables. Different convolution kernel sizes are set based on the location of small objects in the specific image content. In this example, the convolution kernels are set to 3*3, 5*5, and 9*9, respectively, depending on the image type. The conditional vector is one-hot encoded, and the same conditional variables are set for the same type of image.

[0043] Step 103: Random noise generation is introduced. The torch.randn() in the pytorch framework is used to generate a random noise vector. The generation method is: randomly sample 100 times in a uniform distribution between the set interval (-1, 1) to generate a vector of (1, 1, 100).

[0044] Step 104: The entire network structure consists of two parts: the generator and the discriminator; the generator has a fully connected layer, 2 BN layers, 2 ReLU layers, two deconvolution layers, and finally a Tanh activation function.

[0045] Step 104-1: Input 100-dimensional random noise into the fully connected layer to obtain an output matrix, which is then reshaped into a matrix of a specific size.

[0046] Step 104-2: The matrix of a specific size is passed through a BN layer, a ReLU layer, a first deconvolution layer, a BN layer, a ReLU layer, a second deconvolution layer, a BN layer, a ReLU layer, and finally a tanh activation function to output an image matrix of (256, 256, 3), which is a generated sample.

[0047] Step 105: The entire network structure consists of two parts: the generator and the discriminator. Therefore, the discriminator model is constructed as follows: convolutional layer, Res2net module, leaky ReLU layer, PrimaryCaps layer, DiscriCaps layer, leaky ReLU layer, improved fully connected layer, activation layer. In this example, the specific details are as follows:

[0048] Step 105-1: Take an image matrix (256, 256, 3) and a conditional vector C as input to the discriminator:

[0049] Step 105-2: After preprocessing the input data, different conditional vectors C can be set to select different convolution kernel sizes and strides values, and train targeted network models based on the image content.

[0050] Step 105-3: The input image matrix is ​​input into the Res2net module through the convolutional layer, and the image features are further extracted and output to the PrimaryCap layer.

[0051] Step 105-4: In the PrimaryCap layer, we use 32 capsules, each of which applies 8 9*9*256 convolution kernels to the input variable to obtain the corresponding output tensor.

[0052] Step 105-5: In the PrimaryCap layer, each capsule will accept the output tensor as input, with a total of 10 digital capsules; inside the capsule, each input vector passes through the 8*16 weight matrix to map the 8-dimensional input space to the 16-dimensional output space.

[0053] Step 105-6: The decoder consists of a fully connected layer that accepts the 16-dimensional vector from DigitCap, learns it, and reconstructs a 256*256*3 pixel image. This example uses an improved fully connected layer to filter out a correct vector from the DigitCap layer, passes it through a fully connected layer, and processes it in a batch optimization manner. Then, it passes through an activation layer to obtain a feature map. Finally, it undergoes a deconvolution operation using a convolution kernel of a specific size to reconstruct a 256*256*3 pixel image.

[0054] Step 106: In this example, a publicly available online dataset is used and a custom database is constructed based on content classification. The Adam optimizer is selected: the Adam learning rate is set to 0.0002, the first-order moment estimate exponential decay rate is 0.5, and the second-order moment estimate exponential decay rate is 0.999; the batch size is 4; the number of epochs is 400; and the DiscriCaps layer parameters are updated using the capsule neural network's dynamic routing algorithm.

[0055] The simulation experiment of the present invention is implemented in Python based on the pytorch framework. The computer system configuration is: win10 64-bit processor, CoreTM i5-7th CPU, and the experimental IDE uses Pycharm editor.

[0056] The experimental data samples come from the public infrared dim target image dataset on the Internet. 10,000 256*256 size images are randomly input into the network as the dataset, and 300 images are randomly selected from the remaining samples as the test set. During the entire training and testing process, 100*100 size noise is randomly generated as input, and high-resolution images are generated through generative adversarial methods.

[0057] The three commonly used methods of esrgan, srgan and Ccapsul_gan of the present invention were selected to compare the infrared image enhancement effects. Figure 4 As shown, image quality can be evaluated from two perspectives: 1. Human subjective evaluation, such as whether small targets in infrared images can be clearly presented, whether the contrast of the target area is improved compared to the original image, and whether the edge area processing can be clearly presented; and objective data evaluation. This embodiment uses two evaluation indicators, Peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM), for comparative analysis. The PSNR is evaluated based on the error between corresponding pixels in the image, and the formula is shown below:

[0058]

[0059] N represents the number of pixels, I represents the original image, and J represents the reconstructed image. The mean square error between images is:

[0060]

[0061] m and n represent the length and width of the image. PSNR is strongly correlated with MES. The smaller the MSE, the larger the PSNR. The larger the PSNR, the better the image quality.

[0062] Structural Similarity (SSIM) is shown below:

[0063]

[0064] where μ I , μ J are the mean values ​​of images I and J respectively, are the variances of images I and J, σ IJ is the covariance of I and J. SSIM compares images based on three different factors: brightness, contrast, and structure. Its value range is [0, 1]. Higher values ​​indicate higher image similarity.

[0065] In this example, the PSNR value of the generated image is compared with the original image, and the SSIM value is 17.91 and 0.77, which is a stronger enhancement than other algorithms.

[0066] This example has been experimentally verified to have a significant enhancement effect compared to the contrast algorithm, both in terms of subjective impressions and objective data, with significant improvements in resolution and contrast.

[0067] The above only describes in detail the preferred embodiments of the present invention, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the scope of protection of the present invention.

Claims

1. A method for infrared image enhancement using a supervised generative adversarial network and a capsule network, characterized by: The following steps are involved: S1. Introduce conditional variables and use different convolution kernels to perform variable step-size convolution on the image according to the type of image; S2. Using capsule networks as the discriminator framework in generative adversarial networks, combining the advantages of generative adversarial networks in generating new data with the vector nature of capsule networks, we can make the generated images similar to real images while also having more texture and details. S3. The improved Res2Net network module is placed after the convolution layer in the generator, and its stronger feature extraction capability is utilized to effectively extract detailed features in the image. The Res2Net network module is introduced after the convolution layer in the capsule network framework. After the convolution layer preprocesses the image, multi-scale feature extraction is further performed on the target area without increasing the computational load. S4. Redesign the reconstruction layer in the generator network to solve the problems of parameter redundancy and slow training caused by too many fully connected layers; reconstruct the fully connected layer using a deconvolution network, process the vector output by the digital capsule layer, and only retain the vector corresponding to the correct output capsule. Pass it through a fully connected layer, and then use the BatchNormalization optimization method. After the ReLU activation function, the feature map is obtained, and finally a fixed-size convolution kernel is used to perform deconvolution operation to obtain the reconstructed generated image.

2. The infrared image enhancement method using a supervised generative adversarial network and a capsule network according to claim 1, characterized in that: The method of introducing the conditional variable in S1 is: according to the size of the image to be processed and the size of the object contained, the conditional variable sets different convolution kernels and convolution steps, thereby reducing the processing time and enhancing the feature extraction of the region of interest.

3. The infrared image enhancement method using a supervised generative adversarial network and a capsule network according to claim 1, characterized in that: The method of using the capsule network as the framework of the discriminator in the generative adversarial network in S2 is as follows: the discriminator in the generative adversarial network is designed with the capsule network as the framework, a convolution kernel of size M*M is set in the convolution layer of the discriminator, and the step size is set to a1; the value of a1,M is set according to the actual situation of the image in a supervised situation, and the convolution layer is followed by a PrimaryCaps layer, a DigitCaps layer, and a fully connected layer.

4. The infrared image enhancement method using a supervised generative adversarial network and a capsule network according to claim 1, characterized in that: The method for extracting multi-scale features from the target area is: The feature map after 1*1 convolution is divided into N modules, defined as X i ,(i=1,2,…N), each feature block has the same scale size. Except for X1, other submodules have 3*3 convolution kernels, defined as K i (), its output is Y i , submodule X i and K i-1 () add, then input to K i (), the Y i for: .

5. The infrared image enhancement method using a supervised generative adversarial network and a capsule network according to claim 1, characterized in that: The method for obtaining the reconstructed generated image is as follows: processing the vector output by the digital capsule layer and retaining only the vector corresponding to the correctly output capsule; It passes through a fully connected layer, and then adopts the BatchNormalization optimization method and the ReLU activation function to obtain the feature map; finally, a fixed-size convolution kernel is used for deconvolution operation to obtain the reconstructed image.

6. The infrared image enhancement method using a supervised generative adversarial network and a capsule network according to claim 5, characterized in that: The BatchNormalization optimization method is to keep the input of the middle layer of the neural network with a standard normal distribution with a mean of 0 and a variance of 1. The calculation formula is as follows: described represents the batch training mean, is the i-th input data, is the number of batch training samples, represents the variance, Normalize the input data, parameters and parameters is a trainable parameter that maintains the original distribution of the input data.

Images