An infrared image super-resolution method based on a generative adversarial network

Abstract

The application discloses an infrared image super-resolution method based on a generative adversarial network, which comprises the following steps: combining a dense residual module and a super-resolution reconstruction method of multi-scale feature extraction, effectively improving the super-resolution reconstruction quality of the infrared image by combining with an interpolation up-sampling technology; meanwhile, the receptive field of the convolution kernel is increased by expanding the feature extraction capability of the residual network, so that the key features can be captured more effectively; in addition, the mean square error loss is introduced in the loss function, aiming at balancing the clear visual effect and the high PSNR value, so as to achieve better image reconstruction effect. The super-resolution performance is obviously improved on the objective evaluation index, and better effects are also shown in processing the noise, blur and artifacts of the image. The application can be deployed on an embedded platform and has wide practical value.

Description

Technical Field

[0001] This invention relates to the field of super-resolution in computer vision, and more specifically, to a method for infrared image super-resolution using generative adversarial networks. Background Technology

[0002] Infrared imaging technology is an advanced imaging method that captures infrared radiation signals that are not directly perceptible to the human eye. These signals are emitted by objects and are usually related to the object's temperature. Using special sensors, these infrared radiation signals are converted into photoelectric signals that can be processed by electronic devices. Subsequently, these signals are processed by complex algorithms and converted into images that can be recognized by the naked eye. The core advantage of this technology lies in its excellent anti-interference capability, enabling it to operate in harsh lighting conditions and even provide clear images in complete darkness. Furthermore, infrared imaging technology has the ability to penetrate certain materials, such as smoke and fog, and to detect targets at considerable distances.

[0003] In the military field, infrared imaging technology is used in night vision equipment, thermal imaging reconnaissance, missile guidance, and target tracking. It helps soldiers conduct combat and reconnaissance at night or in low-visibility conditions, improving battlefield situational awareness. In the civilian sector, this technology is also widely used in security monitoring, industrial inspection, medical imaging, and environmental monitoring. For example, in industrial inspection, infrared imaging can help detect equipment overheating problems and prevent malfunctions and accidents; in the medical field, it can be used to observe human body temperature distribution, aiding in the diagnosis of certain diseases.

[0004] Despite significant advancements in infrared imaging technology, it still faces several challenges. The most significant issue is that the spatial resolution of infrared images is generally lower than that of visible light imaging systems. This means that infrared images are less detailed than visible light images. Furthermore, high-precision infrared imaging equipment is often expensive, limiting its application in certain fields.

[0005] Therefore, improving infrared image resolution and developing infrared image super-resolution reconstruction technology has become particularly urgent. The goal of this technology is to increase the resolution of infrared images, enabling them to provide more detailed information. Through algorithm optimization and hardware improvements, infrared image super-resolution reconstruction technology is expected to significantly improve the performance of infrared imaging, reduce costs, and expand its application range in various fields. This is of great significance for enhancing the application value of infrared imaging technology in both military and civilian fields, and also opens up new possibilities for future technological development. Summary of the Invention

[0006] To address the above problems, this invention proposes a generative adversarial network-based infrared image super-resolution method. By deeply mining the original data in low-resolution infrared images, optimizing the residual module, and combining it with bicubic interpolation technology, the method preserves low-frequency information in infrared images, achieving a balance between clear visual effects and obtaining a high peak signal-to-noise ratio (PSNR).

[0007] Technical Solution: To achieve the above objectives, this invention proposes an infrared image super-resolution method based on generative adversarial networks, which includes the following steps:

[0008] S1. Input the low-resolution infrared image into the model and use bicubic interpolation to upsample the input image by four times; the low-resolution image has a pixel size of 480×270.

[0009] S2. The input image enters the shallow feature extraction module of the generator network, and the low-resolution image features are initially extracted by expanding the number of channels;

[0010] S3. Input the extracted shallow image features into the deep feature extraction module of the model, and output the deep image features;

[0011] S4. The extracted shallow image features and deep image features are added pixel by pixel to perform global residual learning;

[0012] S5. The finally learned features are fed into the upsampling module to enlarge and reduce the feature resolution to four channels, thus obtaining the output image of the generated model.

[0013] S6. Add the image features obtained in S5 to the image obtained by upsampling in S1 pixel by pixel to obtain the final output result of the generator network;

[0014] S7. Input the original high-resolution image and the output image of the generative model into the discriminant network to determine whether the two images are consistent. If they are inconsistent, continue to adjust the parameters of the generative model and continue the iteration. If they are consistent, the model ends the iteration and the final output is the output image of the generative model. The high-resolution image is 1920×1080.

[0015] Furthermore, the implementation steps of S3 include:

[0016] The deep feature extraction module consists of 16 residual dense modules, each with the same structure and consisting of three residual modules.

[0017] The input feature x1 first enters the first residual module, which performs multi-scale feature extraction on the input. The image features extracted at different scales are added pixel by pixel. After multiple feature extractions, the feature is sent to the channel attention mechanism module to obtain the output x2. Finally, x2 is weighted and added to x2 to form the output rb1 of the first residual module.

[0018] rb1 is input into the second residual module, and deep features are extracted by local feature fusion and local residual learning. By fusing residuals at different levels, better super-resolution results are obtained, and the training process can be made more stable. Finally, the same weighted superposition method as the first residual module is used to obtain the output rb2 of the second residual module.

[0019] Input rb2 into the third residual module to obtain the output rb3. The structure of this residual module is the same as that of the second residual module.

[0020] The weighted rb3 is then superimposed with x1 to obtain the final output rbn1 of the residual dense module;

[0021] Following the above method, rbn1 is input into the second residual dense module to obtain rbn2, and so on to obtain rbn3 to rbn16. rbn1 to rbn16 are then merged according to the second dimension to obtain the output of the deep feature extraction module.

[0022] Furthermore, the discriminant network implementation steps involved in S5 include:

[0023] The image feature channels after global residual learning are 64. Using the sub-pixel convolution function PixelShuffle provided by PyTorch, a four-fold upsampling is performed to reduce the number of channels to 4. PixelShuffle reassembles low-resolution feature maps into high-resolution feature maps through convolution and multi-channel reconstruction. The number of channels in the input feature map is the square of the upsampling factor. Its core operation is to divide the channels of the input feature map into multiple groups, each containing r... 2 There are 3 channels, where r is the upsampling factor. The channels within each group are rearranged into a high-resolution feature map, where each pixel is derived from the original r... 2 It consists of 1 channel, and the input feature map size is H×W×(r). 2 The output feature map size is (H×r)×(W×r)×C, where H and W are the height and width of the feature map, respectively, and C is the number of channels in the feature map.

[0024] Furthermore, the implementation steps of S6 include:

[0025] The image features output from S5 are input into a convolutional layer with a kernel size of 3×3. The number of feature channels is adjusted to 3, making it the same as the number of channels in the super-resolution image obtained from S1. Then, the features are stacked pixel by pixel to obtain the final output of the generator network.

[0026] It can improve the PSNR value of super-resolution images compared to real high-resolution images, compensate for low-frequency information in the image, and obtain better visual effects.

[0027] Furthermore, the discriminant network implementation steps involved in S7 include:

[0028] After the image is input into the discrimination network, it first passes through a 3x3 convolutional layer with 64 channels. Then, the image passes through a LeakyReLU activation layer, which prevents max pooling. After that, it passes through seven convolutional layers in sequence, with the number of filters in each layer being 64, 128, 128, 256, 256, 512, and 512 respectively. After processing by these eight convolutional layers, 512 feature maps are extracted from the image. The classification probability is then output through the sigmoid function.

[0029] Furthermore, multi-scale feature extraction in the residual dense module is defined as follows:

[0030] The input features are fed into two convolutional layers with kernel sizes of 3×3 and 5×5, respectively, and then activated by LeakyReLU layers. The features are then added pixel by pixel to obtain the required multi-scale features, which are used in subsequent residual convolutional layers.

[0031] Furthermore, the channel attention mechanism in the residual dense module is defined as follows:

[0032] The input feature x is subjected to average pooling to obtain a feature map with a resolution of 1×1, and then compressed to the original number of channels by a convolutional layer. After the ReLU activation layer, the number of feature channels is expanded to the initial number of channels. This gives us the weights y of different channels of x learned by the channel attention mechanism. Multiplying x and y gives us the feature x with the added channel attention weights. ′ .

[0033] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) The infrared image super-resolution algorithm of the present invention achieves a good balance in terms of texture details and edge structure restoration, and demonstrates excellent infrared image reconstruction effect;

[0035] (2) By increasing the receptive field of the convolution kernel, the present invention can better capture feature information. When dealing with the problem of infrared image super-resolution, it surpasses other super-resolution techniques in terms of average peak signal-to-noise ratio (PSNR), and the generated image has richer texture and contains more information.

[0036] (3) The network part of this invention combines convolutional neural network with traditional bicubic interpolation algorithm, makes full use of the information in the low frequency region of infrared image, and adds mean square error loss to achieve a balance between clear visual effect and obtaining higher objective evaluation index. Attached Figure Description

[0037] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments are briefly described below.

[0038] Figure 1 This is a schematic diagram of the generative network model according to an embodiment of the present invention.

[0039] Figure 2 A schematic diagram for determining the network model.

[0040] Figure 3 This is a schematic diagram of a dense residual module. Detailed Implementation

[0041] To more clearly illustrate the technical solution of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0042] Infrared imaging technology can convert infrared radiation signals emitted by objects invisible to the human eye into photoelectric signals, and further process them into images visible to the naked eye. This technology possesses strong anti-interference capabilities, penetration power, and a long operating range, and has been widely used in both military and civilian fields. Despite the rapid progress in infrared imaging technology, its spatial resolution is generally still inferior to visible light imaging systems, and high-precision infrared imaging equipment is expensive. Therefore, improving infrared image resolution and developing infrared image super-resolution reconstruction technology has become particularly urgent.

[0043] In the field of visible light image enhancement, numerous super-resolution (SR) techniques have been developed. Early traditional techniques included bilinear interpolation and bicubic interpolation, among others. In recent years, with the rise of deep learning, neural networks have received considerable attention in image processing due to their superior data fitting capabilities. Generative Adversarial Networks (GANs) achieve relatively excellent reconstruction results through adversarial learning between the generator and discriminator. Compared to visible light, infrared images have more blurred edges, less defined texture features, and more noticeable noise.

[0044] Current super-resolution methods suffer from performance deficiencies and poor reconstruction capabilities. This paper proposes an improved infrared super-resolution method based on generative adversarial networks (GANs). Based on this idea, this invention provides an infrared image super-resolution method using a dense residual network to deepen the network structure and introduces a multi-scale feature extraction module, combined with interpolation upsampling methods, to achieve super-resolution reconstruction of infrared images. It significantly improves the super-resolution performance in objective evaluation metrics and also demonstrates good performance in handling image noise, blur, and artifacts.

[0045] Please see Figure 1 and Figure 2 , Figure 1 A flowchart of the generator network portion of an infrared image super-resolution method based on generative adversarial networks provided by this invention. Figure 2 This is a flowchart of the discriminant network portion of an infrared image super-resolution method based on generative adversarial networks provided by the present invention. Figure 1 and Figure 2 As shown, an infrared image super-resolution method based on generative adversarial networks includes the following steps:

[0046] Step 1: Input the low-resolution infrared image into the model and perform a four-fold magnification using bicubic interpolation.

[0047] Step 2: Then, the low-resolution infrared image is fed into the shallow module of the generator network to initially extract image features by increasing the number of channels;

[0048] Step 3: Perform deep feature extraction, and send the features extracted from the shallow layer into the deep layer module to further extract deep image features;

[0049] The deep feature extraction module consists of 16 structurally identical residual dense modules, each containing three residual modules. In the first residual module, the initial feature x1 is processed through multi-scale feature extraction and channel attention mechanisms to obtain x2, which is then weighted and summed to form the output rb1. The second residual module receives rb1, extracts deep features through local feature fusion and residual learning, optimizes the super-resolution effect, and stabilizes the training process, finally obtaining the output rb2 using the same weighted summation method. The third residual module receives rb2, outputs rb3, which is weighted and summed with x1 to obtain the final output rbn1 of the residual dense module. rbn1 is then input into the next residual dense module, sequentially obtaining rbn2 to rbn16. Finally, rbn1 to rbn16 are merged along the second dimension to form the final output of the deep feature extraction module. The multi-scale feature extraction module processes input features through two convolutional layers of different sizes (3×3 and 5×5), followed by a Leaky ReLU activation layer. The feature map obtained by pixel-wise summation provides a rich feature base for subsequent residual convolutional layers. The channel attention mechanism reduces the number of channels to 1 / 16 through average pooling and convolutional layers, then expands it back to the original number through a ReLU activation layer to calculate the weight y of each channel. Finally, the original feature x is multiplied by the weight y to obtain the weighted feature x that incorporates channel attention. ′ .

[0050] Step 4: Add the shallow and deep features pixel by pixel to achieve global residual learning;

[0051] Step 5: Upsample the features. The learned features are amplified by the upsampling module and converted into four channels to generate a super-resolution image.

[0052] After global residual learning, the number of channels in the image features is expanded to 64. Then, PyTorch's PixelShuffle function is used for a 4x upsampling, reducing the number of channels to 4. PixelShuffle transforms low-resolution feature maps into high-resolution feature maps through convolution and channel rearrangement techniques, where the number of channels in the input feature map is the square of the upsampling factor. Specifically, it divides the channels of the input feature map into several groups, each containing the square of the upsampling factor r, and then rearranges these channels to form a high-resolution feature map, where each pixel is reduced from the original r... 2 It consists of several channels. For example, if the size of the input feature map is H×W×(r 2 If the size of the output feature map is (H×r)×(W×r)×C, then the size of the output feature map will become (H×r)×(W×r)×C.

[0053] Step 6: Perform image feature fusion by adding the upsampled image features to the previously upsampled image pixel by pixel to obtain the final output of the generator network.

[0054] The image features output from step 5 are first passed through a 3×3 convolutional layer to adjust the number of channels to 3 to match the number of channels of the super-resolution image obtained in step 1. Then, the adjusted features are added pixel by pixel to the image from step 1. The purpose is to improve the PSNR value of the super-resolution image and supplement the low-frequency information of the image to obtain a better visual effect.

[0055] Step 7: Input the original high-resolution image and the output image of the generative model into the discriminant network to determine whether the two images are consistent. If they are inconsistent, continue to adjust the parameters of the generative model and continue the iteration. If they are consistent, the model ends the iteration and the final output is the output image of the generative model. The high-resolution image is 1920×1080.

[0056] The image is first preprocessed using a 3x3 convolutional layer with 64 channels. Then, a LeakyReLU activation layer is applied to prevent gradient vanishing. The entire model consists of eight convolutional layers, with the number of filters increasing progressively from 64 to 512: 64, 64, 128, 128, 256, 256, 512, and 512. After processing by these layers, the image is decomposed into 512 feature maps. Finally, the classification probability of the image is output using a sigmoid function.

[0057] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. An infrared image super-resolution method based on a generative adversarial network, characterized in that, The method comprises the following steps: S1, inputting an infrared low-resolution image into a model, using bicubic interpolation method to perform four times up-sampling on the input image; the low-resolution image has a pixel of 480*270; S2, inputting the image into a shallow feature extraction module of a generation network, and preliminarily extracting low-resolution image features by expanding the number of channels; S3, inputting the extracted shallow image features into a deep feature extraction module of the model, and outputting deep image features; S4, adding the extracted shallow image features and the deep image features pixel by pixel to perform global residual learning; S5, inputting the finally learned features into an up-sampling module, enlarging and reducing the feature resolution to four channels to obtain an output image of the generation model; S6, adding the image features obtained in S5 and the image obtained by up-sampling in S1 pixel by pixel to obtain the final output result of the generation network; S7, inputting the original high-resolution image and the output image of the generation model into a discrimination network to determine whether the two images are consistent, if not, continue to adjust the parameters of the generation model and continue iteration; if yes, the iteration of the model is ended, and the final output is the output image of the generation model, and the high-resolution image has a pixel of 1920*1080; The implementation steps of S3 comprise: The deep feature extraction module is composed of 16 residual dense modules, each residual dense module has the same structure and is composed of three residual modules; The input feature x1 first enters the first residual module, performs multi-scale feature extraction on the input, adds the image features extracted in different scales pixel by pixel, and then inputs the features extracted for multiple times into a channel attention mechanism module to obtain an output x2, finally adds x2 after weighting to x2 to form the output rb1 of the first residual module; The rb1 is input into the second residual module to extract deep features by using local feature fusion and local residual learning, the different levels of residuals are fused, the same weighting and stacking method as the first residual module is adopted, and the output rb2 of the second residual module is obtained; The rb2 is input into the third residual module to obtain an output rb3, the residual module has the same structure as the second residual module; the rb3 is added to x1 after weighting to obtain the final output rbn1 of the residual dense module; According to the above method, the rbn1 is input into the second residual dense module to obtain rbn2, and rbn3 to rbn16 are obtained in the same way, the rbn1 to rbn16 are merged according to the second dimension to obtain the output of the deep feature extraction module.

2. The infrared image super-resolution method based on a generative adversarial network according to claim 1, characterized in that, The implementation steps of the discrimination network involved in S5 comprise: The image feature channel number after global residual learning is 64, four times up-sampling is performed using the sub-pixel convolution function PixelShuffle provided by pytorch, and the channel number is reduced to 4. PixelShuffle is to obtain a high-resolution feature map by convolution and multi-channel reorganization from a low-resolution feature map. The channel number of the input feature map is the square of the up-sampling factor. The input feature map is divided into r 2 groups, each group contains r 2 channels, and the channels in each group are rearranged into a high-resolution feature map, where each pixel point is composed of r 2 original channels. The input feature map size is HxWx(r 2 xC), and the output feature map size is (Hxr)x(Wxr)x C, where H and W are the height and width of the feature map respectively, and C is the channel number of the feature.

3. The infrared image super-resolution method based on a generative adversarial network according to claim 1, characterized in that, The implementation steps of S6 comprise: The image features output by S5 are input into a convolution layer with a convolution kernel size of 3*3, the feature channel number is adjusted to 3 to be the same as the channel number of the super-resolution image obtained in S1, and then the pixel is added to obtain the final output result of the generation network.

4. The infrared image super-resolution method based on a generative adversarial network according to claim 1, characterized in that, The implementation steps of the discrimination network involved in S7 comprise: After the image input discrimination network, first through a size of 3x3 convolution layer, the layer has 64 channels, then, the image passes through a LeakyReLU activation layer; then in turn through seven convolution layers, the number of filters of each layer is: 64, 128, 128, 256, 256, 512, 512, after the above 8 convolution layers, the image is extracted 512 feature maps; through the sigmoid function output classification probability.

5. The infrared image super-resolution method based on a generative adversarial network according to claim 1, characterized in that, The multi-scale feature extraction in the residual dense module is defined as: The input features are respectively input into two convolution layers with convolution kernel sizes of 3x3 and 5x5, and are respectively passed through LeakyReLU activation layers, and then pixel-by-pixel addition is performed to obtain the required multi-scale features for the subsequent residual convolution layer.

6. The infrared image super-resolution method based on a generative adversarial network according to claim 1, characterized in that, The channel attention mechanism in the residual dense module is defined as: The input feature x is average-pooled to obtain a feature map with a resolution of 1x1, and then a convolution layer is used to compress the channel number to the original channel number After the ReLU activation layer, the feature channel number is expanded to the initial channel number, and thus the weight y of different channels of x learned by the channel attention mechanism is obtained. x is multiplied by y to obtain the feature x with the channel attention weight added ′ .

Images