Infrared image super-resolution reconstruction method combining dual-path convolution and self-attention

Abstract

The application discloses an infrared image super-resolution reconstruction method combining two-path convolution and self-attention, and comprises the following steps: 1. dividing training data and test data of obtained open-source infrared image data sets and pre-processing the data; 2. constructing a network model of fusion features, wherein the network model of fusion features comprises a shallow feature extraction network, a deep feature extraction network and an image reconstruction extraction network; 3. training the network model of fusion features based on the constructed infrared image training data set; and 4. taking the constructed infrared image test data set as input and performing infrared image super-resolution reconstruction through the trained network model of fusion features. The application combines CNN and self-attention, utilizes different sizes of receptive fields, models local features and global features of an image, and adds a bidirectional information interaction module, thereby enhancing the modeling capability of channel dimensions and spatial dimensions and better restoring details of a reconstructed image.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and provides a method for super-resolution reconstruction of infrared images that combines dual-path convolution with self-attention. Background Technology

[0002] Infrared imaging systems obtain infrared images by receiving the infrared radiation from target objects using detectors. They can provide image information that visible light cameras cannot provide in complex environments with low light, low visibility, and difficult observation. Furthermore, they can operate day and night and in various weather conditions, exhibiting excellent anti-interference capabilities, high penetration, high sensitivity, and high stealth, thus being widely used in aerospace, medical, and remote sensing fields. However, infrared imaging systems are susceptible to environmental factors such as smoke, dust, and rain, and are limited by lenses, sensors, processors, and imaging technologies. The captured infrared images typically have low resolution, often exhibiting low contrast, blurred edges, and other problems. These low-resolution, low-quality infrared images pose challenges to downstream tasks such as target detection and segmentation, necessitating improvements in the resolution and quality of infrared images for better utilization.

[0003] However, overcoming the manufacturing process and physical limitations of infrared imaging system hardware is quite difficult, and improving image quality through hardware is time-consuming and labor-intensive. Image super-resolution, as a research direction in the field of computer vision, uses algorithms to reconstruct clear, high-resolution images from low-quality input images, playing a crucial role in tasks such as face recognition and small object detection. Therefore, it can also be considered a reliable and effective method to improve the resolution and quality of infrared images.

[0004] In past research, many methods have been proposed in the field of image super-resolution, which can be mainly divided into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods.

[0005] Interpolation-based methods obtain high-resolution images by establishing mapping relationships and estimating the pixels to be interpolated. These methods are simple in principle and fast in computation, but they introduce relatively little information from the image, making it difficult to recover detailed features effectively. Reconstruction-based methods model and analyze the image imaging process and extract prior information from the fused images to reconstruct high-resolution images. Due to the constraints of prior information, the obtained images can recover more detailed features. However, these methods require a lot of prior knowledge, have low algorithm efficiency, and are sensitive to scale factors. In recent years, deep learning has significantly improved performance in various computer vision tasks compared to traditional methods, and learning-based methods have been widely studied. Learning-based methods construct neural network models, use pairs of low-resolution and high-resolution images for end-to-end training, and finally use the trained network model for super-resolution image reconstruction. Summary of the Invention

[0006] This invention addresses the shortcomings of existing technologies by providing a method for super-resolution reconstruction of infrared images that combines dual-path convolution with self-attention. This method, based on convolution operations, self-attention, feature fusion, and information interaction, jointly learns local and global features of the image to perform super-resolution reconstruction of low-resolution infrared images.

[0007] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0008] Step 1: First, the obtained open-source infrared image dataset is divided into training and testing data, and the data is preprocessed, including image cropping, image downsampling, and image enhancement. Through preprocessing, low-resolution images corresponding one-to-one with the original high-resolution images are obtained. This pair of low-resolution and high-resolution images constitutes the training and testing datasets for the network model.

[0009] Step 2: Construct a fusion feature network model, which includes a shallow feature extraction network, a deep feature extraction network, and an image reconstruction extraction network.

[0010] Step 3: Train the fusion feature network model based on the constructed infrared image training dataset.

[0011] The present invention also provides the following steps for verification:

[0012] Step 4: Following the process in Step 3, use the same training method to train several mainstream super-resolution reconstruction network models proposed in recent years.

[0013] Step 5: Test each trained model with test data, calculate and compare the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the reconstructed images, and use the metrics to verify the performance of the model.

[0014] Step 6: Compare the visual effects of the images reconstructed by each model to intuitively verify the model performance.

[0015] Step 7: Use Local Attribution Map (LAM) analysis to show the key regions that the model focuses on when reconstructing the image.

[0016] This invention makes the following contributions to research and innovation compared to existing technologies:

[0017] 1. This invention combines convolution operations and self-attention, utilizes receptive fields of different sizes to model local and global features of images, and adds a two-way information interaction module to enhance the modeling capabilities of channel and spatial dimensions, thus completing super-resolution reconstruction of infrared images and filling the research gap in infrared image super-resolution reconstruction.

[0018] 2. Through the design of a dual-branch network model, and the deployment of a dual-path convolutional structure, a multi-head attention mechanism unit, and a bidirectional information interaction module, this invention has achieved an advanced level in comparative experiments with cutting-edge methods in the field of image super-resolution reconstruction.

[0019] 3. By analyzing the reconstructed images using LAM, the key areas that the model focuses on during image reconstruction can be revealed, thus better explaining the model's image reconstruction process. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0021] Figure 1 This is a flowchart illustrating the specific operational process of this invention. It includes the relationships between the various modules of the invention, as well as the implementation process of the invention.

[0022] Figure 2 This is a diagram of the overall architecture of the model of this invention. It includes a shallow feature extraction stage, a deep feature extraction stage, and an image reconstruction stage.

[0023] Figure 3 This is a structural diagram of the deep feature extraction stage of the present invention. It includes several stacked dual-branch feature fusion modules and a convolutional layer with a kernel size of 3×3.

[0024] Figure 4 This is a structural diagram of the dual-branch feature fusion module. It mainly consists of two branches: a convolutional branch and a self-attention branch, as well as a bidirectional information interaction module between the branches.

[0025] Figure 5 This is a sample image from the dataset used in this invention. The infrared images used for training and testing in this invention come from five datasets, including different backgrounds, scenes, and numerous objects both outdoors and indoors.

[0026] Figure 6 This is a comparison of the visual effects of images reconstructed by this invention and other mainstream models. The advantages of this invention can be demonstrated by observing the quality of the reconstructed images.

[0027] Figure 7 This invention compares LAM analysis with other mainstream models. The size of the red area indicates how many useful pixels the model utilizes. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings.

[0029] This implementation example Figure 1-7 As shown in Tables 1-4, the design flow of an infrared image super-resolution reconstruction method combining dual-path convolution and self-attention is illustrated. The specific steps are as follows:

[0030] Step 1: First, the obtained open-source infrared image dataset is divided into training and testing data, and the data is preprocessed, including image cropping, image downsampling, and image enhancement. Through preprocessing, low-resolution images corresponding one-to-one with the original high-resolution images are obtained. These pairs of images constitute the training and testing datasets for the network model.

[0031] Step 2: Construct a fusion feature network model, which includes a shallow feature extraction network, a deep feature extraction network, and an image reconstruction extraction network.

[0032] Specifically, such as Figure 2 As shown, the network model contains three sub-networks: a shallow feature extraction network, a deep feature extraction network, and an image reconstruction extraction network.

[0033] The shallow feature extraction network consists of a 3×3 convolutional layer, and the deep feature extraction network consists of several residual feature fusion modules stacked together with a 3×3 convolutional layer added at the end. The image reconstruction extraction network includes two 3×3 convolutional layers and a Pixel Shuffle, which is connected between the two convolutional layers.

[0034] The residual feature fusion module is as follows Figure 3 As shown, a convolutional layer with a kernel size of 3×3 is constructed by several stacked dual-branch feature fusion modules. The dual-branch feature fusion modules are as follows: Figure 4 As shown, it mainly includes convolutional branches and self-attention branches, as well as a bidirectional information interaction module between convolutional branches and self-attention branches.

[0035] The convolutional branch uses convolutional layers with kernel sizes of 3×3 and 5×5 as two paths for feature extraction, and a convolutional layer with a kernel size of 1×1 for dimensionality reduction.

[0036] The convolutional layers selected in the convolutional branch with kernel sizes of 3×3 and 5×5 are depth-wise convolutional layers.

[0037] The two-way interaction module includes a channel interaction module and a spatial interaction module.

[0038] The channel interaction module includes a globally adaptive average pooling layer for information compression, two batch normalization layers, two convolutional layers with a kernel size of 1×1, and a GeLU activation function. Finally, a sigmoid function is used to generate the channel attention map C. am ∈R 1×1×c ;

[0039] The spatial interaction module includes two batch normalization layers, two 1×1 convolutional layers for dimensionality reduction, and a GeLU activation function. Finally, a spatial attention map S is generated using the Sigmoid function. am ∈R h×w×1 .

[0040] Step 3: Based on the constructed infrared image training dataset, the feature fusion network model is trained to obtain the feature fusion model. The experimental environment of this invention is based on the PyTorch framework, running on Ubuntu 20.04, and uses two Nvidia GeForce 3090Ti graphics cards for parallel acceleration. After preprocessing, 600 pairs of low-resolution infrared images from the IR700 dataset are used. LR and high-resolution infrared images I HR Model training involves inputting a low-resolution image into the model and outputting a super-resolution image I. SR The L1 loss is calculated using the output image and the original high-resolution image. The network model parameters are optimized by minimizing the loss and using the Adam optimizer. Training is completed after 300,000 iterations. The calculation process of the L1 loss can be expressed as follows:

[0041] L1=||I SR -IHR ||1.

[0042] Step 4: Using the constructed infrared image test dataset as input, perform infrared image super-resolution reconstruction through the trained fusion feature network model. The specific reconstruction process is as follows:

[0043] A shallow feature extraction network uses a 3×3 convolutional layer to obtain a shallow feature map, mapping the input from a simple low-dimensional space to a more abstract high-dimensional space, thus enabling the early extraction of low-frequency information in the image. For a given low-resolution input image... Where h, w and c in These represent the height, width, and number of channels of the input image, respectively. A shallow feature map F can be obtained through convolutional layer processing. SF ∈R h×w×c Where c is the number of channels in the intermediate feature map, which can be expressed by the formula:

[0044] F SF =Conv 3×3 (I LR ).

[0045] The deep feature extraction stage uses several stacked residual feature fusion modules and a convolutional layer with a kernel size of 3×3 to extract deep feature maps from the image. Each residual feature fusion module consists of several stacked bi-branch feature fusion modules and a convolutional layer with a kernel size of 3×3. As the modules are stacked, the network deepens, enabling more comprehensive feature extraction and learning. The final convolutional layer further aggregates and enhances the features. Therefore, from F... SF Extracting deep feature map F DF ∈R h×w×c It can be expressed by the formula:

[0046]

[0047]

[0048] Where RMFB represents the residual feature fusion module, MFB represents the dual-branch feature fusion module, and N and M are the number of these two modules, respectively. In the experiment, both were set to 6.

[0049] The structure of the dual-branch feature fusion module, such as Figure 4 As shown, it mainly consists of two branches: a convolutional branch and a self-attention branch, as well as a bidirectional information interaction module between the branches. After concatenating the different features extracted from the two branches, the channel dimension is reduced through linear mapping to eliminate redundant information and maintain consistency with the channel dimension of the input feature map, thereby performing residual connections. Then, layer normalization and multilayer perceptron are used for nonlinear transformation and further feature representation.

[0050]

[0051]

[0052] Table 1

[0053] Table 1 shows the effectiveness of combining convolution and self-attention. Using both branches outperforms using only convolution or only self-attention, especially on the IR700 test set, where it improves PSNR by 0.17 and 0.23 respectively, and SSIM by 0.0021 and 0.0035 respectively. From a theoretical perspective, convolution focuses on modeling local features, while self-attention focuses on modeling global features. The experimental results demonstrate that combining these two operations can simultaneously address both image details and the overall picture, resulting in better feature modeling and maximizing model performance.

[0054] The convolutional branch models features within regions and local areas. This branch employs a dual-path convolutional structure, using convolutional layers with kernel sizes of 3×3 and 5×5 as the two paths for feature extraction. Depth-wise convolution is used instead of traditional convolution, significantly reducing the number of network parameters and computational cost. After feature extraction, dimensionality reduction is performed using convolutional layers with kernel sizes of 1×1, filtering out unnecessary information. The features from these two paths are then concatenated to maintain consistency with the input dimension for subsequent feature fusion. Specifically, for a given intermediate feature map I∈R... h×w×c After passing through depth-wise convolutional layers with kernel sizes of 3×3 and 5×5 respectively, the h, w, and c of the two output feature maps remain unchanged. Then, they are each passed through a 1×1 convolutional layer for dimensionality reduction, reducing the number of channels c of the output feature maps to c / 2. Finally, they are concatenated along the channel dimension to restore the number of channels to c. Therefore, the feature F extracted within the convolutional branch is... Conv ∈R h×w×c The extraction process can be represented as:

[0055] F Conv =[C0nv 1×1 (DConv 3×3 (I)), Conv 1×1 (DConv 5×5 (I))],

[0056] Where Conv 1×1 DConv represents a convolutional layer with a kernel size of 1×1. 3×3 and DConv5×5 represents depth-wise convolutional layers with kernel sizes of 3×3 and 5×5, respectively, and [] represents the concatenation of channel dimensions.

[0057] It should be noted that the intermediate feature mapping I∈R mentioned in this embodiment h×w×c The feature maps generated during the entire infrared image super-resolution reconstruction process are collectively referred to as intermediate feature maps.

[0058]

[0059] Table 2

[0060] Table 2 shows the effectiveness of the dual-path convolutional structure. Using convolutional layers with kernel sizes of 3×3 and 5×5 simultaneously outperforms using only a 3×3 kernel layer, only a 5×5 kernel layer, and simultaneously using convolutional layers with kernel sizes of 3×3, 5×5, and 7×7. Particularly on the IR700 test set, it improved PSNR by 0.07, 0.04, and 0.11 respectively, and SSIM by 0.0007, 0.0005, and 0.0017 respectively. Adding a 7×7 kernel layer resulted in the greatest performance degradation. From a theoretical perspective, since dimensionality reduction is required at the end, inserting too many convolutional layers may compress useful information extracted by other convolutional layers and extract unnecessary information, ultimately reducing performance.

[0061] The self-attention branch excels at capturing global dependencies between contexts and modeling global features. This branch employs a multi-head self-attention mechanism for computation. Since the original self-attention computation requires a two-dimensional matrix as input, similarly, for a given intermediate feature map I∈R... h×w×c First, the intermediate feature map is transformed into I′∈R hw× c, then uses three trainable mapping matrices to map it to Q, K, and V respectively, where Q, K, and V are intermediate variables. The mapping process can be expressed as:

[0062] Q = I'M Q K = I′M K V = I′M V ,

[0063] Where M Q M K and M V ∈R c×c Let V be a trainable projection matrix, designed to improve the network's fitting ability. Then, weights are assigned to V by calculating the correlation between Q and K. Therefore, the self-attention calculation can be expressed as:

[0064]

[0065] QK T Correlation is obtained by calculating the dot product of two vectors. The obtained similarity measure is scaled and standardized, and softmax is used to normalize the result so that the elements in the self-attention matrix are between 0 and 1.

[0066] At this point, according to the formula for the computational complexity of matrix operations, the computational complexity of completing the above operations is 4hwc. 2 +2(hw) 2 c. As the image length and width (i.e., h and w) increase, the computational cost increases quadratically. To improve computational speed and reduce complexity, I′∈R hw×c Divide into n groups of size m 2 For non-overlapping windows, calculate the self-attention within each window, where n = hw / m 2 And m 2 << hw, therefore we get Q, K and After window partitioning, the self-attention calculation described above will be completed within a local window, with a computational cost of 4hwc. 2 +2hw(m) 2 c. In addition, to establish connections between windows, feature displacement is performed by moving window partitions and the displacement size is set to m / 2.

[0067] To enable the network model to collectively focus on information from different representation subspaces at different locations, these subspaces are divided into several groups, i.e., multiple heads. Specifically, Q, K, and V are divided into t groups using a linear mapping. Each group computes its own result, and finally, the results from multiple groups are concatenated to obtain the final result. The computation process can be represented as follows:

[0068]

[0069] MHSA(Q,K,V)=[H1,H2,...,H t ],

[0070] in It is a trainable projection matrix. By dividing the entire self-attention operation into t heads, richer feature information can be extracted from multiple dimensions.

[0071] The bidirectional information exchange module between branches includes a channel interaction module and a spatial interaction module. Considering the different weight sharing mechanisms of convolution and self-attention (where convolution shares weights in the spatial dimension while self-attention shares weights in the channel dimension), the features initially obtained from the convolution branch are passed to the self-attention branch through the channel interaction module, and then the features calculated by the self-attention branch are passed back to the convolution branch through the spatial interaction module to update the feature map, thereby realizing complementary information exchange between the two branches.

[0072] For channel interaction, the goal is to pass channel-dimensional information to the self-attention branch to enhance features along the channel dimension. The channel interaction module includes a globally adaptive average pooling layer for information compression, two batch normalization layers, two 1×1 convolutional layers, and a GeLU activation function. Finally, a sigmoid function is used to generate the channel attention map. Specifically, the F-value obtained in the convolutional branch... Conv ∈R h×w×c After passing through the channel interaction module, the channel attention map C is obtained. am ∈R 1×1×c And multiply it by V along the channel dimension to obtain it. Therefore, using the updated V * The calculation is performed in the self-attention branch. The calculation process is as follows:

[0073]

[0074] F Attn =MHSA(Q, K, V) * ),

[0075] in This indicates that through a broadcast mechanism, based on the channel attention map C... am Weights are assigned to each channel in V. At this point, the self-attention calculation is complete. This is then transformed using window restoration and deformation. Convert to For further processing.

[0076] In terms of spatial interaction, since the weights in the convolution kernel are fixed when scanning the entire feature of two-dimensional space, it is necessary to... Information from the data is passed to the convolutional branch to improve features in the spatial dimension. The spatial interaction module consists of two batch normalization layers, two 1×1 convolutional layers for dimensionality reduction, and a GeLU activation function. Finally, the spatial attention map is generated using the sigmoid function. Specifically, the previously obtained... After passing through the spatial interaction module, the spatial attention map S is obtained. am ∈R h×w×1In the spatial dimension, it is related to the initially extracted convolutional features F Conv Multiply to get the updated result. The calculation process is as follows:

[0077]

[0078] Here, ⊙ represents multiplying corresponding elements of two matrices via a broadcast mechanism. It can be seen that in channel interaction, the weights at different spatial locations remain constant but change across different channels, while the opposite occurs in spatial interaction.

[0079]

[0080] Table 3

[0081] Table 3 shows the effectiveness of the bidirectional information interaction module. Using both the channel interaction module and the spatial interaction module performs better than not using the information interaction module, using only the channel interaction module, or using only the spatial interaction module. Particularly when tested on the IR700 test set, removing the bidirectional interaction reduced the model's PSNR and SSIM by 0.11 and 0.0015, respectively. From a theoretical perspective, the features extracted from the two branches undergo beneficial bidirectional interaction, complementing each other's information and thus improving performance.

[0082] The design employs a dual-branch and bidirectional interaction approach, enhancing the modeling capabilities of convolution and self-attention. However, concatenating the features from these two branches along the channel dimension doubles the number of channels, making residual connections with the intermediate feature map I impossible. Therefore, a linear mapping is needed to reduce the dimensionality of the fused features. The calculation process for dimensionality reduction and residual connections can be represented as follows:

[0083]

[0084] Where M Mix It is a trainable projection matrix. Finally, a normalization layer and a multilayer perceptron are added for nonlinear transformation and further feature representation, and residual connections are used. The computation process can be represented as:

[0085]

[0086] LN stands for Layer Normalization Layer, and MLP stands for Multilayer Perceptron.

[0087] The shallow feature map F extracted during the image reconstruction stage SF With deep feature map F DF The low-resolution feature maps are fused and then upscaled by a specified factor using Pixel Shuffle. Finally, a 3×3 convolutional layer is used to obtain the final super-resolution image. Where s is the super-resolution magnification, the calculation process can be expressed as:

[0088] I SR =Conv 3×3 (Pixelshuffle(Conv 3×3 (F SF +F DF ))).

[0089] Finally, to verify the effectiveness of this embodiment, the following verification method is provided:

[0090] Following the procedure in step 3, the same training method was used to train several mainstream image super-resolution reconstruction network models proposed in recent years. The selected mainstream network models include RRDB, RCAN, NLSN, SwinIR, and HAT, all of which are methods published in top conferences or journals in recent years.

[0091] Each trained model was tested using test data. The Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) of the reconstructed images were calculated and compared to validate model performance. Test data included 100 images from the IR700 dataset (excluding training images), 100 images from the DLS-NUC-100 dataset, 100 images from the IR100 dataset, 22 images from the results-A dataset, and 50 images from the Flir dataset. Figure 5 As shown, the infrared images in these datasets include different backgrounds, scenes, and numerous objects both outdoors and indoors, which can better test the model's performance and generalization ability.

[0092]

[0093]

[0094] Table 4

[0095] Table 4 shows the results illustrating the average PSNR and SSIM calculated for each model on each test dataset at different super-resolution magnifications. It can be seen that the present invention achieves the best performance on all test datasets at all super-resolution magnifications, especially on the IR700 test set. At 2x, 3x, and 4x super-resolution magnifications, PSNR is improved by 0.21–0.344, 0.09–0.33, and 0.12–0.31, respectively, while SSIM is improved by 0.001–0.0014, 0.0007–0.0021, and 0.0019–0.0051, respectively. Comparative experiments were also designed to demonstrate the effectiveness of the combination of convolution and self-attention, the use of a dual-path convolutional structure, and the bidirectional information interaction module in the network model design of the present invention. All models used in the comparative experiments were trained using the same training parameters and at a 4x super-resolution magnification, following the procedure in step 3.

[0096] By comparing the visual effects of the images reconstructed by each model, the model performance can be intuitively verified. For example... Figure 6 As shown, five sets of original high-resolution images and super-resolution images reconstructed using the trained model are presented and compared. HR represents a magnified portion of the image, and the corresponding PSNR and SSIM are calculated at the bottom of each set of images for illustrative purposes. It can be seen that, in the IR700 dataset, the gaps between railings in "18.png" are clearer and more distinct; the iron window in "135.png" is reconstructed more straight and regular, especially at the bottom; in "137.png," the shape and boundaries of the building surface are clearer; in the DLS-NUC-100 dataset, the layout of the exterior wall tiles in "75.png" is reconstructed more accurately; and in the IR100 dataset, the railing structure in "022379.png" is clearer. These visual comparisons demonstrate the effectiveness of this invention.

[0097] Local Attribution Map (LAM) analysis is used to show the key regions that the model focuses on during image reconstruction. For example... Figure 7As shown, LAM analysis was performed on SwinIR, HAT, and the model of this invention. LAM is an analysis method designed for image super-resolution reconstruction. In LAM, each pixel is assigned an importance weight, reflecting the degree of contribution of that pixel to the final image reconstruction. The red-marked areas are information pixels that contribute to the reconstruction; the darker the color, the higher the contribution. The larger the red area, the more useful pixels are utilized, and the more comprehensive the information obtained. The calculated diffusion index (DI) reflects the range of pixels involved in image reconstruction; the higher the DI, the more pixels are used, and the wider the range. It can be seen that the model of this invention utilizes a larger range of pixels around the target region for image reconstruction. Compared with SwinIR and HAT, the DI is improved by an average of 6.334 and 5.986 in the five test images, explaining the model's image reconstruction process.

Claims

1. A method for super-resolution reconstruction of infrared images combining dual-path convolution and self-attention, characterized in that... Includes the following steps: Step 1: Divide the obtained open-source infrared image dataset into training and testing datasets, and preprocess the data. Step 2: Construct a fusion feature network model, which includes a shallow feature extraction network, a deep feature extraction network, and an image reconstruction extraction network. The shallow feature extraction network consists of convolutional layers with a kernel size of 3×3. The deep feature extraction network is composed of several residual feature fusion modules stacked together, with a convolutional layer with a kernel size of 3×3 added at the end. The image reconstruction extraction network includes two convolutional layers with a kernel size of 3×3 and a pixel shuffle, with the pixel shuffle connected between the two convolutional layers. Step 3: Train the fusion feature network model based on the constructed infrared image training dataset; Step 4: Using the constructed infrared image test dataset as input, perform infrared image super-resolution reconstruction by training the fusion feature network model. Step 4-1, Shallow feature extraction stage: Use a convolutional layer with a kernel size of 3×3 to obtain a shallow feature map; Step 4-2, Deep Feature Extraction Stage: Several stacked residual feature fusion modules and a convolutional layer with a kernel size of 3×3 are used to extract the deep feature map of the image. The residual feature fusion module consists of several stacked dual-branch feature fusion modules and a convolutional layer with a kernel size of 3×3. The dual-branch feature fusion module includes a convolutional branch and a self-attention branch, as well as a bidirectional information interaction module between the convolutional branch and the self-attention branch. The two-way information interaction module includes a channel interaction module and a spatial interaction module. The channel interaction module includes a globally adaptive average pooling layer for information compression, two batch normalization layers, two convolutional layers with a kernel size of 1×1, and a GeLU activation function. Finally, a sigmoid function is used to generate the channel attention map. ; The spatial interaction module includes two batch normalization layers, two 1×1 convolutional layers for dimensionality reduction, and a GeLU activation function. Finally, a sigmoid function is used to generate a spatial attention map. ; The method for deep feature extraction is as follows: The convolutional branches extract features through two paths: depth-wise convolutional layers with kernel sizes of 3×3 and 5×5, respectively. Then, a convolutional layer with a kernel size of 1×1 is used for dimensionality reduction. This concatenates the features of the two convolutional branches to keep the feature dimensions consistent with the input dimensions for subsequent feature fusion. The self-attention branch employs a multi-head self-attention mechanism for computation, mapping the input intermediate feature map. Mapped to the same shape , and Through calculation and The correlation is Assign appropriate weights, and use a window partitioning strategy to reduce computational cost. , and Grouping allows network models to collectively focus on information from different representation subspaces at different locations; Step 4-3, Image Reconstruction and Extraction Stage: The extracted shallow feature map is fused with the deep feature map, and the low-resolution feature map is magnified by a specified factor using Pixel Shuffle. At the same time, a convolutional layer with a kernel size of 3×3 is used to obtain the final super-resolution image.

2. The infrared image super-resolution reconstruction method combining dual-path convolution and self-attention as described in claim 1, characterized in that, The preprocessing methods in step 1 include image cropping, image downsampling, and image enhancement.

3. The infrared image super-resolution reconstruction method combining dual-path convolution and self-attention as described in claim 1, characterized in that, Both the training and testing datasets contain the original high-resolution images and low-resolution images that correspond one-to-one with the original high-resolution images.

4. The infrared image super-resolution reconstruction method combining dual-path convolution and self-attention as described in claim 1, characterized in that, The convolutional branch uses convolutional layers with kernel sizes of 3×3 and 5×5 as two paths for feature extraction, and a convolutional layer with a kernel size of 1×1 for dimensionality reduction.

5. The infrared image super-resolution reconstruction method combining dual-path convolution and self-attention as described in claim 4, characterized in that, The convolutional layers selected in the convolutional branch with kernel sizes of 3×3 and 5×5 are depth-wise convolutional layers.

6. The infrared image super-resolution reconstruction method combining dual-path convolution and self-attention according to claim 3, characterized in that, In step 3, the training method for the fusion feature network model is as follows: based on the PyTorch framework, running on Ubuntu 20.04, using two Nvidia GeForce 3090Ti graphics cards for parallel acceleration computation, and using 600 pairs of low-resolution infrared images from the training dataset. and high-resolution infrared images Model training involves inputting low-resolution images into the model and outputting super-resolution images. The L1 loss is calculated using the output image and the original high-resolution image. The network model parameters are optimized by minimizing the loss and using the Adam optimizer. Training is completed after 300,000 iterations. The calculation process of the L1 loss can be expressed as follows: 。 7. The infrared image super-resolution reconstruction method combining dual-path convolution and self-attention according to claim 1, characterized in that, The bidirectional information interaction module is implemented as follows: the features initially obtained by the convolution branch are transmitted to the self-attention branch through the channel interaction module, and then the features calculated by the self-attention branch are transmitted back to the convolution branch through the spatial interaction module to update the feature map, thereby realizing the complementary information exchange between the two branches.

Images