A large field of view infrared image super-resolution reconstruction method

Abstract

The application discloses a large-view-field infrared image super-resolution reconstruction method, comprising a first image preprocessing module, a first residual convolutional coding network, a second image preprocessing module, a second residual convolutional coding network, an output module and an evaluation module. The application can realize large-view-field infrared image super-resolution reconstruction.

Description

Technical Field

[0001] This invention relates to the field of super-resolution reconstruction of large field-of-view infrared images, and in particular to a method for super-resolution reconstruction of large field-of-view infrared images. Background Technology

[0002] Image super-resolution reconstruction refers to the process of restoring a set of low-quality, low-resolution images (or motion sequences) into corresponding high-resolution images using specific algorithms. Image super-resolution reconstruction has significant application prospects in military, medical, public safety, and computer vision fields.

[0003] Super-resolution reconstruction algorithms in the spatial domain can be mainly divided into three categories: 1. Interpolation-based reconstruction methods; 2. Constraint-based reconstruction methods; and 3. Learning-based reconstruction methods. With the successful application of convolutional neural networks in image classification and recognition, learning-based reconstruction methods are now widely used, such as SRCNN, FSRCNN, ESPCN, SRGAN, and EDSR. Learning-based methods, to better train the network, generally involve downsampling the original high-resolution image to obtain a low-resolution image as the input image. This low-resolution image is then compared with the original high-resolution image H, and the differences are used to adjust the model parameters. Through iterative training, the differences are minimized.

[0004] However, this method is not suitable for training wide field-of-view infrared images because wide field-of-view infrared images image a large range of objects onto a limited detection surface. This results in wide field-of-view infrared images having lower resolution and less high-frequency detail information compared to visible light images and narrow field-of-view images, making it difficult to obtain high-resolution and suitable wide field-of-view infrared images as training sample libraries. Therefore, how to effectively perform super-resolution reconstruction of wide field-of-view infrared images is of great significance. Figure 2 The images are existing wide field-of-view images and their corresponding high-frequency information images. Image (a) is a wide field-of-view image, and image (b) is its corresponding high-frequency information. It can be seen from the images that the high-frequency detail information of the wide field-of-view infrared image is very limited and the resolution is lower. Summary of the Invention

[0005] The purpose of this invention is to provide a method for super-resolution reconstruction of large field-of-view infrared images, aiming to solve the problem of super-resolution reconstruction of large field-of-view infrared images.

[0006] This invention provides a method for super-resolution reconstruction of large field-of-view infrared images, comprising:

[0007] S1. The input large field-of-view infrared image is adjusted to a uniform size by the first image preprocessing module to obtain a first high-resolution image, and the first high-resolution image is sampled to obtain a first low-resolution image.

[0008] S2. Obtain the first low-resolution image through the first residual convolutional coding network, convert the first low-resolution image into a first convolutional high-resolution image for output, and calculate the optimal parameters of the first residual convolutional coding network.

[0009] S3. The second image preprocessing module performs edge detection on the input large field-of-view infrared image to obtain the edge image of the input image. The edge image is adjusted to a uniform size to obtain the second high-resolution image. The second high-resolution image is sampled to obtain the second low-resolution image.

[0010] S4. Obtain the second low-resolution image through the second residual convolutional coding network, convert the second low-resolution image into a second convolutional high-resolution image for output, and calculate the optimal parameters of the second residual convolutional coding network.

[0011] S5. The output module calls the first image preprocessing module, the first residual convolutional coding network, the second image preprocessing module, and the second residual convolutional coding network to obtain the first convolutional high-resolution image and the second convolutional high-resolution image. The first convolutional high-resolution image and the second convolutional high-resolution image are then fused to obtain the reconstructed super-resolution image.

[0012] S6. Evaluate the reconstructed super-resolution image through the evaluation module.

[0013] By employing the embodiments of the present invention, high-frequency detail information of images is effectively increased, thereby improving the super-resolution reconstruction effect of large field-of-view infrared images.

[0014] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the large field-of-view infrared image super-resolution reconstruction method according to an embodiment of the present invention;

[0017] Figure 2 It consists of existing large field-of-view images and their corresponding high-frequency information images;

[0018] Figure 3This is a schematic diagram of the basic residual module structure of the ResNet network in the large field-of-view infrared image super-resolution reconstruction method according to an embodiment of the present invention;

[0019] Figure 4 This is a schematic diagram of the residual convolutional autoencoder network in the large field-of-view infrared image super-resolution reconstruction method according to an embodiment of the present invention;

[0020] Figure 5 This is a schematic diagram of the residual convolutional autoencoder network in the large field-of-view infrared image super-resolution reconstruction method according to an embodiment of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Method Implementation Examples

[0023] According to embodiments of the present invention, a method for super-resolution reconstruction of large field-of-view infrared images is provided. Figure 1 This is a schematic diagram of the large field-of-view infrared image super-resolution reconstruction method according to an embodiment of the present invention, as shown below. Figure 1 As shown, it specifically includes:

[0024] S1. The input large field-of-view infrared image is adjusted to a uniform size by the first image preprocessing module to obtain a first high-resolution image, and the first high-resolution image is sampled to obtain a first low-resolution image.

[0025] S1 specifically includes: adjusting the input large field-of-view infrared image to a uniform size to obtain a first high-resolution image, and then downsampling and upsampling the first high-resolution image to obtain a first low-resolution image.

[0026] S2. Obtain the first low-resolution image through the first residual convolutional coding network, convert the first low-resolution image into a first convolutional high-resolution image for output, and calculate the optimal parameters of the first residual convolutional coding network.

[0027] S2 specifically includes: acquiring the first low-resolution image, converting the first low-resolution image into a first convolutional high-resolution image for output, and obtaining the optimal parameters of the first residual convolutional coding network by minimizing the mean square error between the first high-resolution image and the first low-resolution image.

[0028] S3. The second image preprocessing module performs edge detection on the input large field-of-view infrared image to obtain the edge image of the input image. The edge image is adjusted to a uniform size to obtain the second high-resolution image. The second high-resolution image is sampled to obtain the second low-resolution image.

[0029] S3 specifically includes: adjusting the input large field-of-view infrared image to a uniform size to obtain a second high-resolution image, and then downsampling and upsampling the second high-resolution image to obtain a second low-resolution image.

[0030] S4. Obtain the second low-resolution image through the second residual convolutional coding network, convert the second low-resolution image into a second convolutional high-resolution image for output, and calculate the optimal parameters of the second residual convolutional coding network.

[0031] S4 specifically includes: acquiring the second low-resolution image, converting the second low-resolution image into a second convolutional high-resolution image for output, and obtaining the optimal parameters of the second residual convolutional coding network by minimizing the mean square error between the second high-resolution image and the second low-resolution image.

[0032] S5. The output module calls the first image preprocessing module, the first residual convolutional coding network, the second image preprocessing module, and the second residual convolutional coding network to obtain the first convolutional high-resolution image and the second convolutional high-resolution image. The first convolutional high-resolution image and the second convolutional high-resolution image are then fused to obtain the reconstructed super-resolution image.

[0033] S6. Evaluate the reconstructed super-resolution image through the evaluation module.

[0034] Based on an autoencoder, this invention proposes an edge-enhanced dual residual convolutional autoencoder network for super-resolution reconstruction of large field-of-view infrared images, specifically targeting the characteristics of large field-of-view infrared images. Experimental results show that this method can effectively increase the high-frequency detail information of the image and improve the super-resolution reconstruction results of large field-of-view infrared images.

[0035] An autoencoder is a neural network model trained using unsupervised training. It utilizes backpropagation to estimate the network's parameters, aiming to make the output equal to the input. In other words, an autoencoder learns an identity function. It works by compressing the input into a latent-space representation and then using the output that reconstructs this representation. Autoencoders are simple to construct, have clear optimization objectives, and can extract features in an unsupervised manner, achieving good results in image resolution reconstruction.

[0036] (1) Convolutional autoencoder

[0037] An autoencoder is a neural network designed to replicate its input to its output. The network consists of two parts: an encoder and a decoder. The encoder compresses the input into a latent space representation, while the decoder reconstructs the input from the latent space representation. The goal of the network is to reconstruct its input so that its hidden layers learn a good representation of that input.

[0038] CAE (Convolutional Auto-Encoder) follows the same principle as Auto-Encoder: encoding followed by decoding. The decoded data is compared to the original data for training, resulting in relatively stable parameters. In CAE, the network consists of convolutional layers and pooling layers. Convolution acts as a filter, while pooling extracts invariant features.

[0039] (2) Residual Network

[0040] Because CNNs can extract low / mid / high-level features, the more layers a network has, the richer the features it can extract at different levels. Furthermore, deeper networks extract more abstract features with more semantic information. However, simply increasing the depth can lead to gradient vanishing or gradient explosion.

[0041] Figure 3 This is a schematic diagram of the basic residual module structure of the ResNet network in a large field-of-view infrared image super-resolution reconstruction system according to an embodiment of the present invention.

[0042] To address the vanishing gradient problem, the ResNet model introduces skip connections that skip one or more network layers. Its basic unit (residual module) model structure is as follows: Figure 3 As shown, this basic unit contains two convolutional layers. Figure 3 The rightmost arrow represents a skip connection. Let x represent the input, and assume the desired network mapping obtained through learning is H(x), which can be used as the input to the bottom activation function.

[0043] In the figure, the portion within the dotted box needs to learn the residual mapping function F(x) = H(x) - x. The residual unit of the ResNet network contains two 3×3 convolutional layers with the same number of output channels. Each convolutional layer is followed by a batch normalization layer (BN layer) and a ReLU activation function. With skip connections in the residual module, the input can propagate faster through the network layers. Therefore, the structure of residual learning is introduced into the algorithm model of this invention to achieve better image reconstruction results.

[0044] To effectively enhance edge detail in large field-of-view infrared images, this invention constructs a dual residual convolutional auto-encoder network based on edge enhancement. Figure 4 This is a schematic diagram of a residual convolutional autoencoder network for a large field-of-view infrared image super-resolution reconstruction system according to an embodiment of the present invention, as shown below. Figure 4 As shown.

[0045] 1 is a convolutional layer, 2 is a downsampling layer, and 3 is an upsampling layer;

[0046] The network consists of two main parts: an encoding part and a decoding part, as shown in the wireframe diagram.

[0047] The encoding layer consists of 5 convolutional layers and 2 max-pooling layers, symmetrically. The decoding layer consists of 2 upsampling layers and 5 convolutional layers. The first and second layers are convolutional layers, the third is a max-pooling layer, the fourth and fifth are convolutional layers, the sixth is a max-pooling layer, and the seventh is a convolutional layer. Correspondingly, the eighth layer is an upsampling layer, the ninth and tenth layers are convolutional layers, the eleventh layer is an upsampling layer, and the twelfth, thirteenth, and fourteenth layers are convolutional layers.

[0048] The convolutional layers all have 3×3 kernels and use the ReLU activation function. Since convolution reduces the size of the feature maps, zero-padding is performed on the feature maps during each convolution to ensure that the size of each feature map remains consistent throughout the network model, facilitating subsequent residual addition. After each downsampling, the image size is halved, but the number of channels doubles. In the decoding part, the upsampled image is correspondingly doubled in size, while the number of channels is halved. To fully utilize the contextual information of larger image regions, the residual learning structure from the ResNet model is introduced, fusing different convolutional layers and extracted feature information. This allows the network model to obtain multiple image features extracted by convolutional kernels at different scales, improving the detail quality of the reconstructed image. The network used in this invention employs a symmetrical addition method between the encoding and decoding layers, with a total of four layers. The outputs of the first and thirteenth convolutional layers are added together; the outputs of the second and twelfth convolutional layers are added together; the outputs of the fourth and tenth convolutional layers are added together; and the outputs of the fifth and ninth convolutional layers are added together. This connection method combines local and global residual learning, enabling multi-channel information transmission and summing different extracted image features to obtain richer image feature information.

[0049] The edge-enhanced dual residual convolutional autoencoder network-based large field-of-view infrared image super-resolution reconstruction algorithm uses the aforementioned residual convolutional autoencoder network as the base network, and adds an input image preprocessing module and a subsequent output fusion module.

[0050] Training phase:

[0051] The edge-enhanced dual residual convolutional autoencoder network is trained using two identical networks (network (1) and network (2)), the difference being in the image preprocessing module and the output.

[0052] The training process is as follows: When an image is input into the network, it first enters the image preprocessing module. Image preprocessing module 1 first adjusts the image to a uniform size, which is used as a high-resolution image, i.e., output 1. The high-resolution image is then downsampled and upsampled to obtain a low-resolution image of the original image size, which is used as the input to the residual convolutional autoencoder network.

[0053] Therefore, the first residual convolutional autoencoder takes a low-resolution image as input and outputs a high-resolution image. By minimizing the mean squared error between the high-resolution and low-resolution images, the optimal parameters of the network are obtained.

[0054] The difference between image preprocessing module 2 and preprocessing module 1 is the addition of an edge extraction module. The input image enters preprocessing module 2, where edge detection is performed to obtain the edge images of the input image. These edge images are then resized to a uniform size and used as the high-resolution image, which is output 2. The high-resolution image is then downsampled and upsampled to obtain a low-resolution image of the original image size, which is used as the input to the residual convolutional autoencoder (RCA). Therefore, the second RCA takes the low-resolution edge image as input and outputs the high-resolution edge image. Similarly, by minimizing the mean squared error between the high-resolution and low-resolution images, the optimal parameters of the network are obtained.

[0055] The parameters of the two residual networks are obtained after training.

[0056] Figure 5 This is a schematic diagram of the residual convolutional autoencoder network of the large field-of-view infrared image super-resolution reconstruction system according to an embodiment of the present invention;

[0057] Testing phase (test):

[0058] After preprocessing, the input images are fed into a dual residual convolutional autoencoder network to obtain a reconstructed high-resolution image and a high-resolution edge image. The two outputs are then fused (experiments have shown that subtracting the edge image from the high-resolution image achieves the best results) to obtain the final output.

[0059] In super-resolution reconstruction, commonly used objective evaluation metrics include SSIM, PSNR, and MSE, which are evaluation methods based on complete reference images.

[0060] These three evaluation metrics primarily assess the reconstruction effect based on the similarity between the reconstructed image and the original image. However, for large-field-of-view infrared image super-resolution reconstruction using edge enhancement-based dual residual convolutional autoencoders, high-frequency details are enhanced, so the reconstruction result cannot be evaluated based on similarity to the original image. Therefore, this invention employs a no-reference image method to evaluate the reconstruction effect. No-reference image evaluation methods are generally based on image statistical characteristics, such as mean, standard deviation, image entropy, and average gradient.

[0061] This invention primarily uses entropy, average gradient, and variance to evaluate the reconstruction effect.

[0062] 1. Image entropy:

[0063] Image entropy refers to the average information content of an image. From an information theory perspective, it measures the amount of information in an image; the higher the entropy, the more information the image contains. Assuming the gray values ​​of each pixel in the image are independent, the gray-level distribution of the image is p = {p1, p2, ..., pi, ..., pn}, where pi represents the ratio of the number of pixels with gray value i to the total number of pixels in the image, and n is the total number of gray levels. The formula for calculating this entropy is:

[0064]

[0065] Where P(i) is the probability of a certain pixel value i appearing in the image, and n is the gray value range (usually 0 to 255).

[0066] The larger the information entropy E value of an image, the larger the gray-level area that deviates from the peak of the image histogram, the more equal the probability of all gray-level values ​​appearing, and the greater the amount of information carried by the image, the richer the information.

[0067] 2. Average gradient:

[0068] The average gradient sensitively reflects an image's ability to express subtle details and contrasts. In an image, a large rate of change in gray levels along a certain direction results in a large gradient. The expression for the average gradient is defined as:

[0069]

[0070] Where m and n are the width and height of the image, respectively, and f(i,j) is the grayscale value of pixel (i,j). The average gradient value is used to measure image sharpness, reflecting subtle detail contrasts and texture transformations within the image. Generally, the larger the value, the more image layers there are, and the sharper the image appears. It can be used to evaluate the detail representation capability of the fused image.

[0071] 3. Standard deviation

[0072] Standard deviation refers to the degree of dispersion of image pixel gray values ​​relative to the mean. The larger the standard deviation, the more dispersed the gray levels in the image, and the better the image quality.

[0073]

[0074]

[0075] The evaluation metrics show that the super-resolution reconstruction algorithm for large field-of-view infrared images based on edge enhancement dual residual convolutional autoencoder network achieves good image quality.

[0076] By employing the embodiments of the present invention, high-frequency detail information of images is effectively increased, thereby improving the super-resolution reconstruction effect of large field-of-view infrared images.

[0077] System Implementation Examples

[0078] According to an embodiment of the present invention, a large field-of-view infrared image super-resolution reconstruction system is provided, specifically including:

[0079] First image preprocessing module: used to adjust the input large field-of-view infrared image to a uniform size to obtain a first high-resolution image, and to sample the first high-resolution image to obtain a first low-resolution image;

[0080] The first image preprocessing module is specifically used to adjust the input large field-of-view infrared image to a uniform size to obtain a first high-resolution image, and to obtain a first low-resolution image by downsampling and upsampling the first high-resolution image respectively.

[0081] First residual convolutional coding network: used to acquire the first low-resolution image, convert the first low-resolution image into a first convolutional high-resolution image for output, and calculate the optimal parameters of the first residual convolutional coding network as the final parameters of the first residual convolutional coding network.

[0082] The first residual convolutional coding network is specifically used to: acquire the first low-resolution image, convert the first low-resolution image into a first convolutional high-resolution image for output, and obtain the optimal parameters of the first residual convolutional coding network by minimizing the mean square error between the first convolutional high-resolution image and the first low-resolution image.

[0083] The second image preprocessing module is used to perform edge detection on the input large field-of-view infrared image to obtain the edge image of the input image, adjust the edge image to a uniform size to obtain the second high-resolution image, and sample the second high-resolution image to obtain the second low-resolution image.

[0084] The second image preprocessing module is specifically used to adjust the input large field-of-view infrared image to a uniform size to obtain a second high-resolution image, and to obtain a second low-resolution image by downsampling and upsampling the second high-resolution image respectively.

[0085] The second residual convolutional coding network is used to acquire the second low-resolution image, convert the second low-resolution image into a second convolutional high-resolution image for output, and calculate the optimal parameters of the second residual convolutional coding network as the final parameters of the second residual convolutional coding network.

[0086] The second residual convolutional coding network is specifically used to: acquire the second low-resolution image, convert the second low-resolution image into a second convolutional high-resolution image for output, and obtain the optimal parameters of the second residual convolutional coding network by minimizing the mean square error between the second convolutional high-resolution image and the second low-resolution image.

[0087] Output module: used to call the first image preprocessing module, the first residual convolutional coding network, the second image preprocessing module and the second residual convolutional coding network to obtain the first convolutional high-resolution image and the second convolutional high-resolution image, and to fuse the first convolutional high-resolution image and the second convolutional high-resolution image to obtain the reconstructed super-resolution image;

[0088] Evaluation module: Used to evaluate the reconstructed super-resolution image.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions to the technical solutions of the embodiments of the present invention do not cause the essence of the corresponding technical solutions to deviate from the scope of the present solution.

Claims

1. A method for super-resolution reconstruction of large field-of-view infrared images, characterized in that, include, S1. The input large field-of-view infrared image is adjusted to a uniform size by the first image preprocessing module to obtain a first high-resolution image, and the first high-resolution image is sampled to obtain a first low-resolution image. S2. Obtain the first low-resolution image through the first residual convolutional coding network, convert the first low-resolution image into a first convolutional high-resolution image for output, and calculate the optimal parameters of the first residual convolutional coding network. S3. The second image preprocessing module performs edge detection on the input large field-of-view infrared image to obtain the edge image of the input image. The edge image is adjusted to a uniform size to obtain the second high-resolution image. The second high-resolution image is sampled to obtain the second low-resolution image. S4. Obtain the second low-resolution image through the second residual convolutional coding network, convert the second low-resolution image into a second convolutional high-resolution image for output, and calculate the optimal parameters of the second residual convolutional coding network. S5. The output module calls the first image preprocessing module, the first residual convolutional coding network, the second image preprocessing module, and the second residual convolutional coding network to obtain the first convolutional high-resolution image and the second convolutional high-resolution image. The first convolutional high-resolution image and the second convolutional high-resolution image are then fused to obtain the reconstructed super-resolution image. S6. Evaluate the reconstructed super-resolution image through the evaluation module; The difference between the second image preprocessing module and the first image preprocessing module is that the second image preprocessing module adds an edge detection step before resizing.

2. The method according to claim 1, characterized in that, S1 specifically includes: adjusting the input large field-of-view infrared image to a uniform size to obtain a first high-resolution image, and then downsampling and upsampling the first high-resolution image to obtain a first low-resolution image.

3. The method according to claim 2, characterized in that, S2 specifically includes: acquiring the first low-resolution image, converting the first low-resolution image into a first convolutional high-resolution image for output, and obtaining the optimal parameters of the first residual convolutional coding network by minimizing the mean square error between the first high-resolution image and the first low-resolution image.

4. The method according to claim 3, characterized in that, S3 specifically includes: adjusting the input large field-of-view infrared image to a uniform size to obtain a second high-resolution image, and then downsampling and upsampling the second high-resolution image to obtain a second low-resolution image.

5. The method according to claim 4, characterized in that, S4 specifically includes: acquiring the second low-resolution image, converting the second low-resolution image into a second convolutional high-resolution image for output, and obtaining the optimal parameters of the second residual convolutional coding network by minimizing the mean square error between the second high-resolution image and the second low-resolution image.

Images