A multi-scale infrared image super-resolution reconstruction method, system and storage medium

Abstract

The present invention relates to the technical field of infrared image super-resolution reconstruction, and specifically relates to a multi-scale infrared image super-resolution reconstruction method, system and storage medium; the multi-scale infrared image super-resolution reconstruction method, system and storage medium comprise the following steps: photographing a natural scene with an infrared camera to obtain an infrared image FLS data set, and dividing the data set into a training set and a test set; a super-resolution reconstruction network is composed of three parts: a shallow feature extraction block, a deep feature extraction block and a reconstruction block; the network is first pre-trained with an ImageNet data set, then trained with an infrared image data set FLS, and finally a composite loss function is used to constrain the network training so that a high-resolution infrared image with clear texture and edge is reconstructed; the present invention enables infrared image super-resolution reconstruction to have a smaller amount of calculation, a smaller amount of parameters and a faster reconstruction speed while maintaining the super-resolution accuracy, which is more in line with practical applications.

Description

Technical Field

[0001] The present invention relates to the field of super-resolution reconstruction, and in particular to a multi-scale infrared image super-resolution reconstruction method. Background Art

[0002] Image super-resolution reconstruction technology is a method that uses computer vision and deep learning technology to improve image resolution. In the field of infrared imaging, due to limitations of hardware equipment and cost, it is usually difficult and expensive to obtain high-resolution infrared images. Therefore, the emergence of super-resolution reconstruction technology fills this technical gap. Although there have been many studies dedicated to improving the effect of super-resolution reconstruction of infrared images, it still faces a series of challenges, including limitations such as data scarcity, complexity of infrared image features, and slow model operation speed.

[0003] The Chinese patent publication number is "CN115100039B", and its name is "A lightweight image super-resolution reconstruction method based on deep learning"; this method extracts rich multi-scale features by constructing a multi-scale feature selection module, enhances the network's expression ability, and improves the image super-resolution reconstruction performance; however, although this method is effective in improving resolution, its parameter volume and computational complexity are too large, and its ability to recover high-frequency information is limited; the reconstructed high-resolution image is blurred in details such as edges and textures, and the network runs slowly, which cannot meet the actual hardware requirements; therefore, how to increase the speed of the super-resolution reconstruction network, while improving the clarity of the reconstructed high-resolution image and solving the problem of blurred edge texture details has become a challenge that researchers in this field need to solve urgently. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] The main purpose of the embodiment of the present invention is to propose a multi-scale infrared image super-resolution reconstruction method, aiming to design a super-resolution method that can generate an infrared high-resolution image based on an infrared low-resolution image.

[0006] (II) Technical solution

[0007] The technical solution of the present invention to solve the above problem is to provide a multi-scale infrared image super-resolution reconstruction method, comprising the following steps:

[0008] 1) Get the dataset:

[0009] Use infrared cameras to capture image data of various scenes to form an infrared image FLS dataset;

[0010] 2) Data preprocessing:

[0011] Perform bicubic downsampling on the dataset to obtain low-resolution images;

[0012] Perform data augmentation methods such as rotating and cropping low-resolution images to expand the data set;

[0013] The data set is carefully divided into training set, test set and validation set to facilitate comprehensive training and evaluation of the model;

[0014] 3) Build a network model:

[0015] The network uses a convolutional neural network as the basic architecture, which includes a shallow feature extraction block, a deep feature extraction block and a reconstruction block. The low-resolution infrared image is reconstructed through the super-resolution network and finally outputs a high-resolution infrared image.

[0016] 4) Train the network model and fine-tune it:

[0017] First, pre-train the network model with the ImageNet dataset, save the weights, and then assign the weights to the network model;

[0018] The entire network is trained using the prepared infrared image FLS dataset until the number of training times reaches the initial set threshold or the value of the loss function reaches the preset range, and the network model parameters are saved;

[0019] Use the validation set to adjust the network model and optimize the network model parameters;

[0020] 5) Save the model:

[0021] After fine-tuning the network model, the model parameters are solidified and stored in a computer-readable storage medium. If a super-resolution reconstruction task is required later, the infrared low-resolution image can be directly input into the network model to obtain an infrared high-resolution image.

[0022] Furthermore, the super-resolution reconstruction network in step 3) includes:

[0023] The shallow feature extraction block consists of convolutional layers and is used to extract shallow feature information of the image;

[0024] The deep feature extraction block is composed of multiple stacked blocks with the same structure, where the stacked block consists of a multi-scale feature extraction block MSF and a global feature extraction block GFE, which is used to extract high-frequency information such as texture and edge of the image;

[0025] The reconstruction block, consisting of a convolutional layer and a sub-pixel upsampling layer, is used to reconstruct high-resolution images.

[0026] Furthermore, the training method of the super-resolution reconstruction network in step 4) comprises:

[0027] Network model pre-training: Use the ImageNet dataset to pre-train the network model and save the weights;

[0028] Network model training: Secondly, the infrared image FLS dataset is used to train and adjust the network;

[0029] This method can accelerate the convergence speed of the network and enhance its generalization ability. At the same time, it can utilize the common features between the two data sets to help the network generate clearer high-resolution images more effectively.

[0030] Furthermore, in the step 4), the network is trained with the training set until the number of training times reaches a set threshold or the value of the loss function reaches a preset range. In the step of saving the network model parameters, the loss function adopts a composite loss function, including L1 pixel loss, frequency domain loss and high-frequency loss, wherein the high-frequency loss and frequency domain loss are used to reconstruct high-frequency information such as texture and details of the infrared image; the total loss is calculated by a weighted combination of the above three losses.

[0031] In order to solve the above technical problems, the present invention further proposes a multi-scale infrared image super-resolution reconstruction system, which is applied to the above super-resolution reconstruction method, comprising:

[0032] Data acquisition module, used to collect infrared images of multiple scenes;

[0033] A data preprocessing module is used to preprocess the data from the data acquisition module and divide the data into a training set and a test set;

[0034] A super-resolution reconstruction module is used to perform super-resolution reconstruction on the data from the data pre-processing module, thereby outputting a high-resolution infrared image;

[0035] The data storage module is used to store the parameters of the super-resolution reconstruction module.

[0036] To solve the above technical problem, the present invention also proposes a computer-readable storage medium storing a computer program and parameters of a super-resolution reconstruction network. When the computer program is executed by a processor, the processor executes the steps of the super-resolution reconstruction method as described above.

[0037] (III) Beneficial effects

[0038] Compared with the prior art, the present invention provides a multi-scale infrared image super-resolution reconstruction method, system and storage medium, which have the following beneficial effects:

[0039] 1. This paper designs a lightweight super-resolution reconstruction network model. It performs convolution and other operations on images that are downsampled by 2 times and 4 times, and trains it into an attention mechanism to maximize the use of image pixel information and improve computational efficiency. At the same time, it performs deep feature extraction on the original image to improve the super-resolution reconstruction speed as much as possible while ensuring the accuracy of infrared image reconstruction. The network model proposed in this experiment has only 0.26M parameters and a computing power of 13.61G.

[0040] 2. The present invention proposes a new attention mechanism to extract the edge, channel and spatial information of infrared images, so that the reconstructed model shows good results on the FLS dataset. Under the premise of ensuring a small increase in the number of parameters and calculations, in the experiment with a reconstruction multiple of 2, the structural similarity is improved by 0.0366 and the peak signal-to-noise ratio is improved by 0.81. In the experiments with reconstruction multiples of 3 and 4, the evaluation indicators are also improved.

[0041] 3. The present invention proposes a new composite loss that integrates L1 loss, frequency domain loss and high-frequency loss to constrain the training of the network model so that the reconstructed high-resolution image has clear edge texture and other high-frequency information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 A flowchart of a super-resolution reconstruction method according to the present invention;

[0044] Figure 2 A network block diagram of a multi-scale infrared image super-resolution reconstruction method according to the present invention;

[0045] Figure 3 This is a network structure diagram of the multi-scale feature extraction block MSF of the present invention;

[0046] Figure 4 A network structure diagram of mixed attention in the global feature extraction block of the present invention;

[0047] Figure 5 This is a network structure diagram of the global feature extraction block GFE described in the present invention;

[0048] Figure 6 is a schematic diagram of the system of the present invention;

[0049] Figure 7It is a schematic diagram for comparing corresponding evaluation indicators of the prior art and the method proposed in the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] In view of the problems existing in the prior art, the present invention provides a multi-scale infrared image super-resolution reconstruction method, system and storage medium, which are described in detail below in conjunction with the accompanying drawings.

[0052] A multi-scale infrared image super-resolution reconstruction method, system and storage medium, the following are specific embodiments.

[0053] Embodiment 1:

[0054] like Figure 1 As shown, a multi-scale infrared image super-resolution reconstruction method specifically includes the following steps:

[0055] Step 1, obtain the data set: use an infrared camera to shoot infrared image data of various scenes to form the infrared data set FLS, and download the ImageNet data set at the same time.

[0056] Step 2, data preprocessing: downsample the dataset by 2, 3, or 4 times, and perform data enhancement methods such as cropping and rotation to expand the dataset; then divide the preprocessed dataset into a training set and a test set.

[0057] Step 3: Build a network model: Figure 2 As shown in the figure, the network model includes a shallow feature extraction block, a deep feature extraction block and a reconstruction block; the shallow feature extraction block consists of a convolution layer with a convolution kernel of 3×3; the deep feature extraction block consists of three stacked blocks with the same structure and multiple jump connections, where the stacked block consists of a multi-scale feature extraction block MSF and a global feature extraction block GFE; the reconstruction block consists of a convolution layer with a convolution kernel of 3×3 and a sub-pixel convolution layer;

[0058] Shallow feature extraction block: The input of the shallow feature extraction block is an infrared low-resolution image. It consists of a convolution layer with a convolution kernel of 3×3, which is used to extract shallow features of the image and increase the dimension of the feature map.

[0059] The deep feature extraction module consists of three stacked blocks. The output of each stacked block is directly connected to the output of the deep feature extraction module through a skip connection. These feature maps are spliced ​​through channels to fuse more deep features. Finally, the spliced ​​feature maps are used as the input of the reconstruction module.

[0060] The reconstruction block consists of a convolution layer with a convolution kernel of 3×3 and a sub-pixel convolution layer. The convolution layer adjusts the number of channels of the input feature map, and the sub-pixel convolution layer reconstructs the feature map and outputs the final infrared high-resolution image.

[0061] Step 4: Train the network model and fine-tune it: When training the network, the loss function uses a composite loss function, including L1 loss, frequency domain loss, and high-frequency loss. The L1 pixel loss is used to measure the difference between the generated image and the real image. It calculates the loss value by comparing the pixel values ​​of the generated image and the real image. The specific mathematical expression is shown in the following formula:

[0062] L1=||I SR -I HR ||1

[0063] Among them, I SR is by putting I LR As the network input, I HR is the corresponding ground-truth, that is, the original high-resolution image.

[0064] The frequency domain loss is used to measure the difference between the generated image and the real image in the frequency domain. It calculates the loss value by comparing the representation of the generated image and the real image in the frequency domain. Its specific mathematical expression is shown in the following formula:

[0065] L f =||F SR -I HR ||1

[0066] Where F stands for Fast Fourier Transform.

[0067] After the high-frequency loss obtains the low-frequency image through a low-pass filter, it is subtracted from the original high-resolution image to obtain the high-frequency image of the infrared image. This loss emphasizes the learning of high-frequency details; its specific mathematical expression is shown in the following formula:

[0068] First, calculate the high-frequency details I H , specifically:

[0069] I H =I HR -F1(I HR )

[0070] Among them, F1 represents a low-pass filter.

[0071] Then, I HR Filter the smaller pixel values ​​and assign different weights to get H t , specifically:

[0072]

[0073] Among them, f represents the Sigmoid operation, a is the preset threshold, and i represents the position of the pixel in the image.

[0074] Finally, define the high frequency loss L H , specifically:

[0075] L H =H t ⊙(||I SR -I HR ||)

[0076] The total loss is composed of the losses of the above three parts, which can be calculated by weighted combination. The specific calculation formula of the total loss L is as follows:

[0077] L=λ1L1+λ2L f +λ3L H

[0078] Among them, λ1, λ2 and λ3 represent the proportion coefficients of different loss functions in the total loss.

[0079] Start network training, set the number of training times to 1000, and set the learning rate during training to 0.001 to ensure the stability of the network during training. A smaller learning rate can help the network adjust parameters more carefully in the early stage of training to avoid missing the local optimal solution. In order to optimize the network model parameters, select the Adam optimizer as an alternative to the stochastic gradient descent algorithm. The function value threshold of the loss function is set at around 0.005. When the value of the loss function is less than 0.005, it can be considered that the training of the entire network has been basically completed. After the training is completed, save the network parameters for future use.

[0080] The present invention, before using the FLS infrared image dataset for network training, first uses the ImageNet dataset to pre-train the network, using the common features between the two datasets to help the network more effectively generate clearer high-resolution images;

[0081] Fine-tune the network model: Use the validation set to adjust the network model, optimize the network model parameters, and further improve the super-resolution reconstruction network performance to obtain high-resolution infrared images with clear edge textures.

[0082] Step 5, save the model, complete the network model adjustment, fix the network parameters, and determine the final multi-scale super-resolution reconstruction network model.

[0083] Embodiment 2:

[0084] The present invention proposes a multi-scale feature extraction block and a global feature extraction block, which are as follows:

[0085] Multi-scale feature extraction block MSF, the structure is as follows Figure 3 As shown, the feature map is divided into three parts after the channel segmentation operation: a, b, c, where the number of feature maps contained in the three branches a, b, and c is the same;

[0086] After the a branch is downsampled by 2 times, it is extracted by convolution block 1, and finally the feature map is restored to its original size by upsampling; the b branch is downsampled by 4 times, and the feature is extracted by convolution block 2, and then the feature map is restored to its original size by upsampling; the c branch extracts the feature map information through the attention mechanism; the outputs of the a and b branches and the output of the c branch are spliced ​​through the channels, and the dimension of the feature map is adjusted through convolution block 3, and the pixel addition operation is performed with the c branch; at the same time, the output of the convolution block 3 is subjected to the Sigmoid operation to obtain the attention information, and the dot product operation is performed with the result of the splicing of the a and b channels; the output after the dot product operation and the output after the pixel addition of the c branch are spliced ​​through the channel, and then passed through the convolution block 4, the output channel is modified and connected to the next stacking block as the final output;

[0087] Among them, the downsampling method of branches a and b adopts adaptive maximum pooling, and the upsampling method adopts nearest neighbor interpolation; convolution block 1 and convolution block 2 are depth-separable convolutions with a convolution kernel of 3×3; convolution kernel 3 and convolution block 4 are convolution layers with a convolution kernel of 1×1, which are used to adjust the dimension of the feature map for subsequent operations;

[0088] Among them, the attention mechanism of branch c adopts a hybrid attention structure, the structure is as follows Figure 4 As shown in the figure, it includes channel attention, spatial attention and corner attention. These attention mechanisms extract information of different scales respectively to fully capture the importance of feature maps; channel attention is used to emphasize the correlation between different channels, spatial attention focuses on the spatial distribution of feature maps, and corner attention focuses on detecting corner information in images; this comprehensive attention mechanism can effectively capture key information of different scales in feature maps, thereby improving the network's ability to represent complex scenes;

[0089] The global feature extraction block GFE, such as Figure 5As shown in the figure, it consists of convolution block five, convolution block six, GLUE activation function, jump connection and layer normalization. Convolution block five encodes the spatial local context and doubles the number of channels of the input features used for mixed channels. Convolution block six reduces the number of channels to the original input dimension, thereby retaining important feature information; GLUE activation function is used to introduce nonlinear mapping, which helps the model learn more complex feature representations, and jump connection helps to alleviate the gradient disappearance problem and improve the efficiency of information transmission; finally, layer normalization is used to normalize the feature map, which helps the network to train stably and converge faster. These components work together to enable the GFE module to effectively extract global features, providing important support for improving the performance of the model;

[0090] Among them, convolution block 5 is a depth-separable convolution with a convolution kernel of 3×3, and convolution block 6 is a convolution layer with a convolution kernel of 1×1;

[0091] The corresponding evaluation indicators of the method proposed in this invention are compared with the existing technical indicators, such as Figure 7 As shown in the figure, it can be seen that the method proposed in the present invention has an increase in computational complexity and parameters compared with the existing method, but the structural similarity and peak signal-to-noise ratio are greatly improved; these indicators show that the present invention can achieve reconstruction of more accurate high-resolution images while ensuring the network operation speed.

[0092] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A multi-scale infrared image super-resolution reconstruction method, characterized in that: The following steps are involved: 1) Get the dataset: Use infrared cameras to capture a variety of scene image data to form an infrared image dataset FLS; 2) Data preprocessing: Perform bicubic downsampling on the dataset to obtain low-resolution images; Rotate and crop low-resolution images to augment the data set; The data set is carefully divided into training set, test set and validation set to facilitate comprehensive training and evaluation of the model; 3) Build a network model: The entire network includes a shallow feature extraction module, a deep feature extraction module, and a reconstruction module; The shallow feature extraction module consists of a convolution layer with a convolution kernel of 3×3, which is used to extract the shallow features of the infrared image and perform dimensionality increase on the feature map; The deep feature extraction module consists of multiple stacked blocks, each of which consists of a multi-scale feature extraction module (MSF) and a global feature extraction module (GFE); the output of each stacked block is directly connected to the output of the deep feature extraction module through a skip connection; then, these feature maps are channel-joined so that the reconstructed image can retain more high-frequency features; finally, the joined feature maps are used as the input of the reconstruction module; The reconstruction module consists of a convolutional layer and a sub-pixel convolutional layer. The convolutional layer adjusts the number of channels of the input feature map, and the sub-pixel convolutional layer reconstructs the feature map and outputs the final infrared high-resolution image. The network of step 3) includes: The shallow feature extraction block consists of a convolution layer with a convolution kernel of 3×3, which is used to extract the shallow feature information of the image; The deep feature extraction block contains three stacked blocks, and the stacked blocks have the same structure, which are all composed of MSF and GFE. MSF is a multi-scale feature extraction module, which is composed of feature extraction branches at three scales, and GFE is a global feature extraction module. The reconstruction module consists of a convolution layer with a convolution kernel of 3×3 and a sub-pixel convolution layer; The multi-scale feature extraction module MSF consists of three parallel branches. The feature map is divided into three parts after channel segmentation: a, b, and c. Each part contains the same number of feature maps. a, b, and c are used as inputs of branches a1, b1, and c1 respectively, and finally input into branches a1, b1, and c1. The a1 branch and the b1 branch are downsampled by 2 and 4 times respectively, and then the feature information is extracted by convolution blocks 1 and 2 respectively. Then, the original size of the feature map is restored by upsampling. The a1 and b1 branches are first spliced ​​through the channel and then fused through convolution block 3. Then, the attention information is obtained through the Sigmoid operation, and the dot multiplication operation is performed with the result of splicing the a and b channels. The c1 branch does not perform downsampling, so the c1 branch directly extracts feature map information through a hybrid attention mechanism. The structure includes channel attention, spatial attention, and corner attention. At the same time, the output of the convolution block 3 is pixel-added with the c1 branch, and then the output obtained is channel-joined with the output obtained after the Sigmoid operation and dot multiplication operation of branches a1 and b1. After the information of the three branches a1, b1, and c1 is fused, it passes through the convolution block 4, modifies the feature dimension, and is connected to the next stacking block as the final output; Among them, the downsampling of a1 and b1 branches adopts adaptive maximum pooling, and the upsampling adopts nearest neighbor interpolation; convolution blocks 1 and 2 are depth-separable convolutions with a convolution kernel of 3×3, and convolution blocks 3 and 4 are convolution layers with a convolution kernel of 1×1, which are used to adjust the dimension of the feature map; The global feature extraction module GFE consists of convolution block five, convolution block six, GLUE activation function, jump connection and layer normalization; convolution block five encodes the spatial local context and modifies the number of channels to twice the original number, and convolution block six reduces the number of channels to the original input dimension to retain important feature information; GLUE activation function introduces nonlinear mapping to help the model learn more complex feature representations; jump connection alleviates the gradient vanishing problem and improves the efficiency of information transmission; layer normalization normalizes the feature map to promote stable network training and faster convergence; these components work together to enable the GFE module to effectively extract global features and improve model performance; The convolution block 5 is a depth-separable convolution with a convolution kernel of 3×3, and the convolution block 6 is a convolution layer with a convolution kernel of 1×1, which is used to adjust the dimension of the feature map. 4) Train the network model and fine-tune it: First, pre-train the network model with the ImageNet dataset, save the weights, and then assign the weights to the network model; The entire network is trained using the prepared infrared image dataset FLS until the number of training times reaches the initial set threshold or the value of the loss function reaches the preset range, and the network model parameters are saved; Use the validation set to adjust the network model and optimize the network model parameters; The loss function in step 4) adopts a composite weighted loss function, including L1 loss, frequency domain loss and high frequency loss function; The L1 pixel loss is used to measure the difference between the generated image and the real image. It calculates the loss value by comparing the pixel values ​​of the generated image and the real image. Its specific mathematical expression is shown in the following formula: L1=||I SR -I HR ||1 Among them, I SR is by putting I LR As the network input, I HR is the corresponding ground-truth, i.e. the original high-resolution image, I LR It is a low-resolution image; The frequency domain loss is used to measure the difference between the generated image and the real image in the frequency domain. It calculates the loss value by comparing the representation of the generated image and the real image in the frequency domain. Its specific mathematical expression is shown in the following formula: L f =||F SR -I HR ||1 Where F represents fast Fourier transform; After the high-frequency loss obtains the low-frequency image through a low-pass filter, it is subtracted from the original high-resolution image to obtain the high-frequency image of the infrared image. This loss emphasizes the learning of high-frequency details; its specific mathematical expression is shown in the following formula: First, calculate the high-frequency details I H , specifically: I H =I HR -F1(I HR ) Among them, F1 represents a low-pass filter; Then, I HR Filter the smaller pixel values ​​and assign different weights to get H t , specifically: Among them, H t is a weight matrix, f represents the Sigmoid operation, a is the preset threshold, and i represents the position of the pixel in the image; Finally, define the high frequency loss L H , specifically: L H =H t ⊙(||I SR -I HR ||) The total loss is composed of the losses of the above three parts, which can be calculated by weighted combination. The specific calculation formula of the total loss L is as follows: L=λ1L1+λ2L f +λ3L H Among them, λ1, λ2 and λ3 represent the proportion coefficients of different loss functions in the total loss respectively. 5) Save the model: After fine-tuning the network model, the model parameters are solidified and stored in a computer-readable storage medium. If a super-resolution reconstruction task is required later, the infrared low-resolution image can be directly input into the network model to obtain an infrared high-resolution image.

2. The multi-scale infrared super-resolution reconstruction method according to claim 1, characterized in that: The infrared image datasets FLS in step 1) are all taken by individuals using infrared cameras in multiple scenes.

3. The multi-scale infrared super-resolution reconstruction method according to claim 1, characterized in that: The model training in step 4) is first pre-trained using the ImageNet dataset, and then trained using the infrared image dataset FLS.

4. A super-resolution reconstruction system, applied to the super-resolution reconstruction method according to any one of claims 1 to 3, characterized in that: include: Data acquisition module, used to collect infrared images of multiple scenes; A data preprocessing module is used to preprocess the data from the data acquisition module and divide the data into a training set and a test set; A super-resolution reconstruction module is used to perform super-resolution reconstruction on the data from the data pre-processing module, thereby outputting a high-resolution infrared image; The data storage module is used to store the parameters and codes of the super-resolution reconstruction module.

5. A computer-readable storage medium, which stores a computer program and parameters of a super-resolution reconstruction network. When the computer program is executed by a processor, the processor executes the steps of the super-resolution reconstruction method as described in any one of claims 1 to 3.

Images