Multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer

Abstract

The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. By cleverly combining the advantages of CNN and Transformer, in the reconstruction task of infrared image super-resolution images, clearer detail information and edge structure can be reconstructed, and to a certain extent, contrast and clarity are improved, unnecessary artifacts and noise are reduced, and a clearer appearance is maintained while restoring high-precision texture and structure, with good visual effects.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution reconstruction, and in particular relates to a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. Background Art

[0002] Image super-resolution reconstruction technology aims to reconstruct high-quality super-resolution images with rich detail and sharp edges from low-resolution images. Infrared thermal imaging systems offer advantages such as strong anti-interference capabilities, long working distances, strong penetration capabilities, and all-weather operation. However, due to the effects of infrared optical diffraction, the captured images suffer from low resolution, insufficient detail, and blurred edges. Infrared image super-resolution reconstruction technology aims to restore the high-frequency information lost during the imaging process, thereby reconstructing high-quality images with rich detail and fine texture. It has been widely used in security monitoring, military reconnaissance, target detection, remote sensing, and other fields.

[0003] Currently, using deep learning methods to perform super-resolution image reconstruction is a key area of ​​research. Based on network architecture, deep learning-based super-resolution reconstruction algorithms are primarily based on convolutional neural networks (CNNs) and implemented through specifically designed network architectures, such as densely connected networks, generative adversarial networks, and autoencoders. In 2019, Zhang et al. proposed an RNAN based on a residual structure for high-quality image super-resolution reconstruction. The reconstructed images have low noise and artifacts, but the algorithm has a large number of parameters. Because excessive convolutions and parameters typically consume high computational costs and memory, Tian et al. proposed a lightweight LESRCNN in 2020. This network achieves a good balance between reconstruction performance and operational efficiency, but the reconstruction of image texture details requires further improvement. To address the issue of information loss, in 2021, the same team proposed CFSRCNN by cascading several types of modules to combine low-resolution and high-resolution features. Experimental results show that this network can significantly improve the visual quality of reconstructed images, but the algorithm consumes a lot of memory. To prevent the loss of original information, Tian et al. proposed HGSRCNN in 2022 based on a symmetric structure, improving the expressiveness of images. In 2023, Zhang et al. proposed PCS, which expanded the effective receptive field of the convolutional network without adding any additional parameters or FLOPs. This network can reconstruct images with rich details and high clarity, but the algorithm has a high number of parameters.

[0004] In recent years, Transformer has achieved remarkable success in SR. Its powerful self-attention mechanism can learn the global information of the image, thereby improving model performance and has been widely used in computer vision tasks. In 2022, Yang et al. introduced Transformer into SR and proposed CAT. This network uses the attention mechanism to mine the deep features of the image to accurately reconstruct the HR image, but the high computational complexity limits its practical application. To address this problem, in the same year, Xia et al. proposed a DAT based on a deformable self-attention module. Experiments show that the model can focus on more relevant parts. In 2024, Zhang et al. proposed a new Transformer network RGT. The network can model global spatial information while maintaining low computational cost, but the detailed texture effect is not good.

[0005] The success of deep learning in visible light super-resolution (SR) has prompted researchers to apply it to infrared super-resolution, but due to insufficient datasets and low image resolution, infrared super-resolution is still in its developmental stages. In 2016, Choi et al. first applied CNN to infrared image super-resolution and proposed the TEN model. This model used only three layers of CNN to represent the mapping between LR and HR images, but the network was trained using visible light data to enhance the resolution of infrared images. However, due to the differences between the visible and infrared spectra, the improvement in reconstruction accuracy was limited. In 2023, Wu et al. proposed MFDRN based on information distillation, achieving network lightweighting, but performance needs to be further improved.

[0006] While CNN-based algorithms have achieved promising results, they are limited by the inherent weaknesses of the convolution operator, limiting them to local feature extraction and making it difficult to capture global dependencies. Transformers have achieved significant success in SR, but existing self-attention mechanisms often neglect the incorporation of contextual features, which impacts network performance. Infrared images often suffer from low resolution, insufficient detail, and blurred edges. Therefore, fully leveraging both local and global features of infrared images and efficiently aggregating these rich features is a crucial issue. Summary of the Invention

[0007] To solve the above problems, the present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, which can fully utilize the texture and structure information contained in low-resolution infrared images, reconstruct super-resolution images close to the real scene, and improve image reconstruction performance.

[0008] A multi-input hierarchical infrared image super-resolution reconstruction method based on a collaborative CNN and Transformer is proposed. The method inputs a low-resolution infrared image to be reconstructed into a trained infrared image super-resolution network model, which then outputs a reconstructed super-resolution infrared image. The infrared image super-resolution network model includes a multi-input encoder, a fusion unit, a decoder, and a skip connection unit.

[0009] The multi-input encoder part is used to extract different scale features of the low-resolution infrared image to obtain feature maps of different scales;

[0010] The fusion part is used to fuse feature maps of different scales from different scales to obtain aggregated feature maps of different scales;

[0011] The decoder part is used to reconstruct features of aggregated feature maps of different scales to obtain reconstructed feature maps;

[0012] The jump connection part is used to fuse the reconstructed feature map and the low-resolution infrared image to obtain a super-resolution infrared image.

[0013] Furthermore, the multi-input encoder part includes a first Bicubic unit to a third Bicubic unit, a first encoder EB1 to a third encoder EB3, a first SCM unit, and a second SCM unit;

[0014] The first Bicubic unit is used to adjust the size of the low-resolution infrared image to the size required by the final output super-resolution image to obtain a feature map F0;

[0015] The second Bicubic unit is used to downsample the feature map F0 by a factor of two to obtain a feature map F1;

[0016] The third Bicubic unit is used to downsample the feature map F1 by a factor of two to obtain a feature map F2;

[0017] The first encoder EB1 is used to extract features from the feature map F0 to obtain a first scale feature map;

[0018] The first SCM unit is used to perform rough feature extraction on the feature map F1 to obtain a first rough extracted feature map;

[0019] The second encoder EB2 is used to perform joint feature extraction on the first scale feature map and the first coarse extraction feature map to obtain a second scale feature map;

[0020] The second SCM unit is used to perform rough feature extraction on the feature map F2 to obtain a second rough extracted feature map;

[0021] The third encoder EB3 is used to perform joint feature extraction on the second scale feature map and the second coarse extraction feature map to obtain a third scale feature map.

[0022] Furthermore, the first encoder EB1 is composed of a cascade of convolution unit I and a local-global feature perception unit I; the second encoder EB2 is composed of a cascade of convolution unit II, FAM unit I and a local-global feature perception unit II; the third encoder EB3 is composed of a cascade of convolution unit III, FAM unit II and a local-global feature perception unit III;

[0023] The convolution unit I in the first encoder EB1 is used to extract shallow features of the feature map F0 to obtain the initial shallow features F0'; the local-global feature perception unit I is used to extract the features of the initial shallow features F0' to obtain the first-scale feature map;

[0024] The convolution unit II in the second encoder EB2 is used to extract shallow features of the downsampled first-scale feature map to obtain a first shallow feature map; the FAM unit I is used to fuse the first shallow feature map with the first coarse extracted feature map output by the first SCM unit to obtain a first fused feature map; the local-global feature perception unit II is used to extract features from the first fused feature map to obtain a second-scale feature map;

[0025] The convolution unit III in the third encoder EB3 is used to extract the shallow features of the downsampled second-scale feature map to obtain a second shallow feature map; the FAM unit II is used to fuse the second shallow feature map with the second coarse extracted feature map output by the second SCM unit to obtain a second fused feature map; the local-global feature perception unit III is used to extract the features of the second fused feature map to obtain a third-scale feature map.

[0026] Furthermore, any local-global feature perception unit is composed of a cascade of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM;

[0027] The local texture attention subunit LTAM is composed of a center difference convolution combination, four ordinary convolution combinations and a 1×1 convolution layer cascade; wherein the center difference convolution combination is composed of a center difference convolution layer CD Conv and an activation layer LReLU cascade, and the ordinary convolution combination is composed of a 3×3 ordinary convolution layer Conv and an activation layer LReLU cascade. The local texture feature map output by the local texture attention subunit LTAM is obtained as follows:

[0028] F LTAM =Conv((Cat(F5,F4,F3,F2,F1,F in ))) 1×1 +Fin

[0029] F1=Lrelu(CDConv(F in ))+F in

[0030] F m =F 3 (Cat(F in ,F1,...,F m-1 )),m=2,3,4,5

[0031] Among them, F LTAM is the local texture feature map finally output by the current local texture attention subunit LTAM, F in is the feature map of the central difference convolution layer CD Conv of the current local texture attention subunit LTAM, F1~F5 are the outputs of the central difference convolution combination and the four ordinary convolution combinations respectively, Conv(·) 1×1 is the operation of the 1×1 convolution layer; CDConv(·) is the operation of the center difference convolution layer CD Conv, Lrelu is the activation function of the activation layer LReLU, Cat(·) represents the splicing operation; m=1,2,3,4,5 are the numbers of the center difference convolution combination and the four ordinary convolution combinations respectively; F 3 (·)=Lrelu(Conv(·) 3×3 );Conv(·) 3×3 The operation of the 3×3 convolution layer;

[0032] The global Transformer attention subunit GTAM consists of three cascaded residual structures, where the first residual structure contains a LayerNorm layer and a moving window self-attention mechanism SW-MSA; the second residual structure contains a LayerNorm layer and a channel attention mechanism CA; the third residual structure contains a LayerNorm layer and an affine multi-layer perceptron AffMLP. The scale feature map output by the global Transformer attention subunit GTAM is obtained as follows:

[0033] F res1 =SW-MAS(LN(F LTAM ))+F LTAM

[0034] F res2 =CA(LN(F res1 ))+F res1

[0035] F GTAM =AffMLP(LN(F res2 ))+Fres2

[0036] Among them, F res1 is the output feature map of the first residual structure, SW-MAS(·) is the operation of the moving window self-attention mechanism SW-MSA, LN(·) represents the operation of the LayerNorm layer, F res2 is the output feature map of the second residual structure, CA(·) is the operation of the channel attention mechanism CA, AffMLP(·) is the operation of the affine multi-layer perceptron AffMLP, F GTAM is the output feature map of the third residual structure, and F GTAM The scale feature map is the final output of the current local-global feature perception unit.

[0037] Furthermore, the fusion part includes a first MFF unit, a second MFF unit and a third MFF unit; wherein any MFF unit is sequentially cascaded by a 1×1 convolution layer, a 3×3 convolution layer and a 3×3 depth-separable convolution layer, and any MFF unit obtains an aggregate feature map F MFF The method is as follows:

[0038]

[0039] in, is the output of the first encoder EB1, is the output of the second encoder EB2, is the output of the third encoder EB3, Conv(·) 1×1 The operation of the 1×1 convolution layer; Conv(·) 3×3 The operation of the 3×3 convolution layer; DSConv(·) 3×3 is the operation of a 3×3 depthwise separable convolutional layer; Cat(·) represents the concatenation operation; R(·) represents the scaling operation using bicubic interpolation.

[0040] Furthermore, the decoder part includes a first decoder DB1, a second decoder DB2 and a third decoder DB3, wherein the first decoder DB1 is sequentially composed of a 3×3 convolution unit I and a local-global feature perception unit I in cascade; the second decoder DB2 is sequentially composed of a 3×3 convolution unit II and a local-global feature perception unit II in cascade; the third decoder DB3 is sequentially composed of a feature refinement unit FRM, a 3×3 convolution unit III and a local-global feature perception unit III in cascade;

[0041] The feature refinement unit FRM of the third decoder DB3 is used to enhance the aggregated feature map output by the third MFF unit to obtain an enhanced aggregated feature map; the convolution unit III is used to extract shallow features of the enhanced aggregated feature map to obtain a third decoded shallow feature map; the local-global feature perception unit III is used to extract local features of the third decoded shallow feature map to obtain a third decoded feature map;

[0042] The convolution unit II of the second decoder DB2 is used to extract shallow features of the upsampled third decoding feature map and the aggregated feature map output by the second MFF unit to obtain a second decoding shallow feature map; the local-global feature perception unit II is used to extract local features of the second decoding shallow feature map to obtain a second decoding feature map;

[0043] The convolution unit I of the first decoder DB1 is used to extract shallow features of the upsampled second decoding feature map and the aggregated feature map output by the first MFF unit to obtain a first decoding shallow feature map; the local-global feature perception unit I is used to extract local features of the first decoding shallow feature map to obtain a first decoding feature map;

[0044] The first decoded feature map is used as the reconstructed feature map finally output by the decoder part, and the method for obtaining the reconstructed feature map finally output by the decoder part is as follows:

[0045]

[0046] Among them, k = 1, 2, 3 represents the decoder number, Cat(·) is the feature concatenation operation, Up(·) represents the upsampling operation, Conv(·) 3×3 is the operation of a 3×3 convolutional unit; LGFPM(·) is the operation of a local-global feature perception unit; FRM(·) is the operation of a feature refinement unit FRM, F1”~F3” represent the inputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3 respectively, and F1”’~F3”’ represent the final outputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3 respectively.

[0047] Furthermore, the feature refinement unit FRM includes two 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, a global average pooling layer, and an attention mechanism AR;

[0048] The aggregated feature map output by the third MFF unit is divided into five branches, each of which enters a 1×1 convolution layer, three 3×3 dilated convolution layers with dilation coefficients of 4, 8, and 12, and a global average pooling layer. The feature maps obtained by the five branches are superimposed to obtain a high-level feature map; another 1×1 convolution layer is used to perform feature aggregation on the high-level feature map to obtain a high-level aggregated feature map; the attention mechanism AR is used to perform feature enhancement on the high-level aggregated feature map to obtain a high-level enhanced feature map;

[0049] The high-level aggregate feature map is element-wise multiplied with the high-level enhanced feature map to obtain the final enhanced aggregate feature map.

[0050] Furthermore, the jump connection part includes a 3×3 convolutional layer;

[0051] After the 3×3 convolution layer extracts features from the reconstructed feature map, the extracted feature map is combined with the low-resolution infrared image to obtain a super-resolution infrared image.

[0052] Furthermore, the hybrid loss function L is used when training the infrared image super-resolution network model. total as follows:

[0053] L total =L pix +λ1L str +λ2L tex

[0054] Among them, L pix is the pixel loss, L str is the structural loss, L tex is the texture loss, λ1 represents the structural loss L str The weight of λ2 represents the texture loss L tex The weight of .

[0055] Furthermore, L pix , L str , L tex The expressions are:

[0056]

[0057]

[0058]

[0059] Where N is the number of low-resolution infrared images in the training sample, is the pixel matrix of the predicted super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample output by the infrared image super-resolution network model, is the pixel matrix of the actual super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample, 1 is the L1 norm, represents the fast Fourier transform, and SSIM(·) is the structural similarity.

[0060] Beneficial effects:

[0061] 1. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. By cleverly combining the advantages of CNN and Transformer, in the reconstruction task of infrared image super-resolution images, clearer detail information and edge structure can be reconstructed, and to a certain extent, contrast and clarity are improved, unnecessary artifacts and noise are reduced, and a clearer appearance is maintained while restoring high-precision texture and structure, with good visual effects.

[0062] 2. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. The local-global feature perception unit is composed of a carefully designed local texture attention subunit and a global Transformer attention subunit. It can simultaneously capture local texture details and global structures from infrared images of different scales, and can reconstruct high-quality images with rich information, fine texture, high contrast, and strong robustness.

[0063] 3. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. The feature refinement unit can effectively extract high-level context features and local details by combining different convolutional receptive fields, realize mutual complementation of features, and enhance the expression ability of image details.

[0064] 4. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. The fusion part fuses features from different scales and depths, and can obtain rich feature representations to better reconstruct high-quality images. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flow chart of the design method of the infrared image super-resolution network model provided by the present invention;

[0066] Figure 2 The principle block diagram of the infrared image super-resolution network model provided by the present invention;

[0067] Figure 3 This is a block diagram of the principle of the local-global feature perception unit provided by the present invention;

[0068] Figure 4A block diagram of the principle of the local texture attention subunit in the local-global feature perception unit provided by the present invention;

[0069] Figure 5 A block diagram of the principle of the global Transformer attention subunit in the local-global feature perception unit provided by the present invention;

[0070] Figure 6 A principle block diagram of the feature refinement unit provided by the present invention;

[0071] Figure 7 The principle block diagram of the MFF unit provided by the present invention;

[0072] Figure 8 A comparison chart of the reconstruction results of different algorithms provided by the present invention, including, from top to bottom: HR, RNAN, LESRCNN, CFSRCNN, HGSRCNN, CAT, DAT, PCS, RGT, TEN, MFDRN and the present invention. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0074] To fully utilize the texture and structural information contained in low-resolution infrared images and reconstruct images close to the real scene, the present invention proposes a multi-input hierarchical infrared image super-resolution reconstruction method based on a collaborative CNN and Transformer. The low-resolution infrared image to be reconstructed is input into a trained infrared image super-resolution network model, which outputs a reconstructed super-resolution infrared image, thereby improving image reconstruction performance. The infrared image super-resolution network model includes a multi-input encoder part, a fusion part, a decoder part, and a skip connection part.

[0075] The multi-input encoder part is used to extract different scale features of the low-resolution infrared image to obtain feature maps of different scales;

[0076] The fusion part is used to fuse feature maps of different scales from different scales to obtain aggregated feature maps of different scales;

[0077] The decoder part is used to reconstruct features of aggregated feature maps of different scales to obtain reconstructed feature maps;

[0078] The jump connection part is used to fuse the reconstructed feature map and the low-resolution infrared image to obtain a super-resolution infrared image.

[0079] like Figure 1As shown, the design method of the infrared image super-resolution network model provided by the present invention includes the following steps:

[0080] Step 1: Use the multi-input encoder-decoder structure as the framework to build an infrared image super-resolution network model. The model is divided into four parts: the multi-input encoder part, the fusion part, the decoder part and the skip connection part.

[0081] Step 2: Construct a local-global feature perception unit LGFPM consisting of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM to simultaneously capture local texture details and global structures from infrared images of different scales.

[0082] Step 3: Design the feature refinement module FRM. By combining different convolutional receptive fields, it effectively extracts high-level contextual features and local details, achieves mutual complementation of features, and enhances the expressiveness of image details.

[0083] Step 4: Design a multi-level feature fusion module MFF to fuse features from different scales and depths to obtain rich feature representations to better reconstruct high-quality images.

[0084] Step 5: Construct a hybrid loss function consisting of pixel loss, structure loss, and texture loss to guide network optimization in a supervised learning manner.

[0085] Step 6: Use a multi-input encoder-decoder structure as a framework to build a network model, use the training set of the super-resolution network to train the network model, determine the optimal values ​​of the parameters in the network model through parameter optimization, and obtain an infrared image super-resolution network model.

[0086] like Figure 2 As shown, the multi-input encoder part includes the first Bicubic unit to the third Bicubic unit, the first encoder EB1 to the third encoder EB3, the first SCM unit, and the second SCM unit;

[0087] The first Bicubic unit is used to adjust the size of the low-resolution infrared image to the size required by the final output super-resolution image to obtain a feature map F0; the second Bicubic unit is used to downsample the feature map F0 by a factor of two to obtain a feature map F1; the third Bicubic unit is used to downsample the feature map F1 by a factor of two to obtain a feature map F2; the first encoder EB1 is used to extract features from the feature map F0 to obtain a first-scale feature map; the first SCM unit is used to perform coarse feature extraction on the feature map F1 to obtain a first coarsely extracted feature map; the second encoder EB2 is used to perform joint feature extraction on the first-scale feature map and the first coarsely extracted feature map to obtain a second-scale feature map; the second SCM unit is used to perform coarse feature extraction on the feature map F2 to obtain a second coarsely extracted feature map; the third encoder EB3 is used to perform joint feature extraction on the second-scale feature map and the second coarsely extracted feature map to obtain a third-scale feature map.

[0088] Furthermore, the first encoder EB1 is composed of a cascade of convolution unit I and a local-global feature perception unit I; the second encoder EB2 is composed of a cascade of convolution unit II, FAM unit I and a local-global feature perception unit II; the third encoder EB3 is composed of a cascade of convolution unit III, FAM unit II and a local-global feature perception unit III;

[0089] The convolution unit I in the first encoder EB1 is used to extract shallow features of the feature map F0 to obtain the initial shallow features F0'; the local-global feature perception unit I is used to extract local features of the initial shallow features F0' to obtain the first-scale feature map;

[0090] The convolution unit II in the second encoder EB2 is used to extract shallow features of the downsampled first-scale feature map to obtain a first shallow feature map; the FAM unit I is used to fuse the first shallow feature map with the first coarse extracted feature map output by the first SCM unit to obtain a first fused feature map; the local-global feature perception unit II is used to extract local features of the first fused feature map to obtain a second-scale feature map;

[0091] The convolution unit III in the third encoder EB3 is used to extract the shallow features of the downsampled second-scale feature map to obtain a second shallow feature map; the FAM unit I is used to fuse the second shallow feature map with the second coarse extracted feature map output by the second SCM unit to obtain a second fused feature map; the local-global feature perception unit II is used to extract the local features of the second fused feature map to obtain a third-scale feature map.

[0092] Specifically, the multi-input encoder extracts features at different scales from the input image. The fusion stage fuses the encoded features from multiple stages to achieve sufficient feature aggregation. The decoder then reconstructs these aggregated features. Finally, the texture details of the input image are transferred to the output image via skip connections, improving image reconstruction capabilities.

[0093] Specifically, for the input LR image I LR ∈R H×W×C First, adjust the image size through Bicubic to get the same output SR image I SR ∈R tH×tW×C Feature maps of the same size F0∈R tH×tW×C :

[0094] F0=Bicubic ↑t (I LR )

[0095] Where H, W and C in are the height, width and number of channels of the input low-resolution image, and t is the scale factor.

[0096] Next, feature extraction is performed on F0 through the multi-input encoder section, which consists of three encoder blocks (EBs): EB1, EB2, and EB3. EB1 extracts information from the original scale, while EB2 and EB3 extract information from the features downsampled in the previous encoding stage. Each EB uses a 3×3 convolutional layer to extract shallow features and a local-global feature perception module (LGFPM) to extract local image features and build long-range dependencies. To address the information loss caused by the downsampling operation, F0 is continuously downsampled and features are extracted using the SCM. These features are then combined with the shallow features from the EB2 and EB3 encoding stages using the FAM and the LGFPM operation. The encoding process can be expressed as follows:

[0097]

[0098] Where k is the index of EB, F′ k EB k Encoded feature map. Conv(·) 3×3 Represents a 3×3 convolutional layer. f1 = Bicubic ↓2 (F0), f2 = Bicubic ↓2 (f1). Down represents a downsampling block, which consists of a 3×3 convolutional layer with a stride of 2 and a LeakyReLU. After downsampling, the size of the output feature map is halved and the number of channels is doubled.

[0099] Further, if Figure 3As shown in Figure 2, any local-global feature perception unit LGFPM is composed of a cascade of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM, which are used for local and global feature extraction respectively;

[0100] like Figure 4 As shown in the figure, the local texture attention subunit LTAM is composed of a center difference convolution combination, four ordinary convolution combinations and a 1×1 convolution layer cascade; wherein, the center difference convolution combination is composed of a center difference convolution layer CD Conv and an activation layer LReLU cascade, and the ordinary convolution combination is composed of a 3×3 ordinary convolution layer Conv and an activation layer LReLU cascade. The local texture feature map output by the local texture attention subunit LTAM is obtained as follows:

[0101] F LTAM =Conv((Cat(F5,F4,F3,F2,F1,F in ))) 1×1 +F in

[0102] F1=Lrelu(CDConv(F in ))+F in

[0103] F m =F 3 (Cat(F in ,F1,...,F m-1 )),m=2,3,4,5

[0104] Among them, F LTAM is the local texture feature map finally output by the current local texture attention subunit LTAM, F in is the feature map of the central difference convolution layer CD Conv of the current local texture attention subunit LTAM, F1~F5 are the outputs of the central difference convolution combination and the four ordinary convolution combinations respectively, Conv(·) 1×1 is the operation of the 1×1 convolution layer; CDConv(·) is the operation of the center difference convolution layer CD Conv, Lrelu is the activation function of the activation layer LReLU, Cat(·) represents the splicing operation; m=1,2,3,4,5 are the numbers of the center difference convolution combination and the four ordinary convolution combinations respectively; F 3 (·)=Lrelu(Conv(·) 3×3 );Conv(·) 3×3 The operation of the 3×3 convolution layer;

[0105] Specifically, the Local Texture Attention Module (LTAM) consists of a centered differential convolution (CD Conv) and traditional convolution. First, a centered differential convolution (CD Conv) and four 3×3 convolutional layers are used to extract deep features from the input. LeakyReLU layers are used after each layer to increase nonlinear representation capabilities. To enhance detail extraction, features extracted from previous layers are reused in a DenseNet-like manner. Next, the hierarchical features are concatenated and the number of feature map channels is adjusted through a 1×1 convolutional layer. Finally, skip connections are used to aggregate input details with the deep feature map.

[0106] like Figure 5 As shown, the global Transformer attention subunit GTAM is composed of three residual structures in cascade, where the first residual structure contains a LayerNorm layer and a moving window self-attention mechanism SW-MSA; the second residual structure contains a LayerNorm layer and a channel attention mechanism CA; the third residual structure contains a LayerNorm layer and an affine multi-layer perceptron AffMLP. The method for obtaining the scale feature map output by the global Transformer attention subunit GTAM is as follows:

[0107] F res1 =SW-MAS(LN(F LTAM ))+F LTAM

[0108] F res2 =CA(LN(F res1 ))+F res1

[0109] F GTAM =AffMLP(LN(F res2 ))+F res2

[0110] Among them, F res1 is the output feature map of the first residual structure, SW-MAS(·) is the operation of the moving window self-attention mechanism SW-MSA, LN(·) represents the operation of the LayerNorm layer, F res2 is the output feature map of the second residual structure, CA(·) is the operation of the channel attention mechanism CA, AffMLP(·) is the operation of the affine multi-layer perceptron AffMLP, F GTAM is the output feature map of the third residual structure, and F GTAM The scale feature map is the final output of the current local-global feature perception unit.

[0111] It should be noted that, in order to solve the problem that the moving window self-attention mechanism lacks attention to channel information, the channel attention mechanism CA(·) is used to improve the mutual dependence between channels. This module does not perform window partitioning, but uses linear mapping to project the input into Q, K and V matrices. By comparing Q T and K, get the attention feature map, and compare it with V T Multiply them together, and the final result is reshaped into a feature map of the same size as the input of the structure. The corresponding formula is as follows:

[0112]

[0113] Where, F CA is the output of the channel attention mechanism, Q = XP Q 、K=XP K 、V=XP V The X is the input of the channel attention mechanism, P Q 、P K and P V is the linear transformation matrix. d is the dimension of Q or K, softmax is the Softmax activation function, and R is the reshaping of the dimension.

[0114] The affine multilayer perceptron is used to improve the robustness of the model. First, an affine transformation layer is used to make the network more stable without increasing training costs. Then, a linear transformation layer is used to map the features to a high-dimensional space, and ReLU is used to increase nonlinear expression capabilities. Next, a linear transformation layer is used to restore the original dimensions. Finally, an affine transformation layer is introduced again to increase the flexibility of the network. The corresponding formula is as follows:

[0115] F AffMLP =Aff(linear(relu(linear(Aff(f in )))))

[0116] Where, f in is the input of the affine multilayer perceptron, linear is the linear transformation layer, relu is the ReLU activation function, and Aff is the affine transformation layer.

[0117] Further, if Figure 2 As shown, the multi-input fusion part includes a first MFF unit, a second MFF unit and a third MFF unit; wherein, as Figure 7 As shown, any MFF unit is composed of a 1×1 convolution layer, a 3×3 convolution layer, and a 3×3 depth-separable convolution layer cascaded in sequence, so any MFF unit obtains the aggregated feature map F MFF The method is as follows:

[0118]

[0119] in, is the output of the first encoder EB1, is the output of the second encoder EB2, is the output of the third encoder EB3, Conv(·) 1×1 The operation of the 1×1 convolution layer; Conv(·) 3×3 The operation of the 3×3 convolution layer; DSConv(·) 3×3 is the operation of a 3×3 depthwise separable convolutional layer; Cat(·) represents the concatenation operation; R(·) represents the scaling operation using bicubic interpolation.

[0120] That is to say, in order to make full use of the multi-scale features extracted in the encoding stage, the fusion part simultaneously fuses the features extracted from each EB through the multi-level feature fusion (MFF) unit. The results of different levels can be expressed as:

[0121]

[0122] Where, F k ” is the fusion result of the k-th level. (·) ↑2 Bicubic double upsampling, (·) ↓4 、(·) ↓2 Bicubic four times and two times downsampling respectively.

[0123] As can be seen, the fusion part of the present invention takes the output of the encoder block as input and maps the fusion result to the decoder block. First, bicubic interpolation is used to scale the inconsistently sized encoded features to unify the size of the feature maps. Next, the scaled features are concatenated along the channel dimension and passed through a 1×1 convolutional layer to generate a preliminary fusion result. A 3×3 convolutional layer is then used to extract shallow features. Finally, a 3×3 depthwise separable convolutional layer is used to better focus on the key details of the features.

[0124] Further, if Figure 2 As shown, the decoder part includes a first decoder DB1, a second decoder DB2 and a third decoder DB3, wherein the first decoder DB1 is sequentially composed of a 3×3 convolution unit I and a local-global feature perception unit I in cascade; the second decoder DB2 is sequentially composed of a 3×3 convolution unit II and a local-global feature perception unit II in cascade; the third decoder DB3 is sequentially composed of a feature refinement unit FRM, a 3×3 convolution unit III and a local-global feature perception unit III in cascade;

[0125] The feature refinement unit FRM of the third decoder DB3 is used to enhance the aggregated feature map output by the third MFF unit to obtain an enhanced aggregated feature map; the convolution unit III is used to extract shallow features of the enhanced aggregated feature map to obtain a third decoded shallow feature map; the local-global feature perception unit III is used to extract local features of the third decoded shallow feature map to obtain a third decoded feature map;

[0126] The convolution unit II of the second decoder DB2 is used to extract shallow features of the upsampled third decoding feature map and the aggregated feature map output by the second MFF unit to obtain a second decoding shallow feature map; the local-global feature perception unit II is used to extract local features of the second decoding shallow feature map to obtain a second decoding feature map;

[0127] The convolution unit I of the first decoder DB1 is used to extract shallow features of the upsampled second decoding feature map and the aggregated feature map output by the first MFF unit to obtain a first decoding shallow feature map; the local-global feature perception unit I is used to extract local features of the first decoding shallow feature map to obtain a first decoding feature map;

[0128] The first decoded feature map is used as the reconstructed feature map finally output by the decoder part, and the method for obtaining the reconstructed feature map finally output by the decoder part is as follows:

[0129]

[0130] Where k = 1, 2, 3 represents the decoder number, Cat(·) is the feature concatenation operation, Up(·) represents the upsampling operation, which consists of a 3×3 transposed convolution layer with a stride of 2 and a LeakyReLU. After upsampling, the number of channels in the output feature map is halved and the size is doubled; Conv(·) 3×3 is the operation of a 3×3 convolutional unit; LGFPM(·) is the operation of a local-global feature perception unit; FRM(·) is the operation of a feature refinement unit FRM, F1”~F3” represent the inputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3 respectively, and F1”’~F3”’ represent the final outputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3 respectively.

[0131] like Figure 6 As shown, the feature refinement unit FRM includes two 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, a global average pooling layer, and an attention mechanism AR;

[0132] The aggregated feature map output by the third MFF unit is divided into five branches, which respectively enter one of the 1×1 convolution layers, three 3×3 dilated convolution layers with dilation coefficients of 4, 8, and 12, and a global average pooling layer. The feature maps obtained by the five branches are superimposed to obtain a high-level feature map; another 1×1 convolution layer is used to perform feature aggregation on the high-level feature map to obtain a high-level aggregated feature map; the attention mechanism AR is used to perform feature enhancement on the high-level aggregated feature map to obtain a high-level enhanced feature map; the high-level aggregated feature map is element-wise multiplied with the high-level enhanced feature map to obtain the final enhanced aggregated feature map.

[0133] That is to say, the feature refinement unit FRM samples the input tensor in parallel through a 1×1 convolution, three 3×3 dilated convolutions with dilation coefficients of 4, 8, and 12, and a global average pooling layer. The three dilated convolution branches use a layer-by-layer stacking method to enable the network to extract richer high-level features; the feature maps obtained by the five branches are superimposed to increase the information interaction of the feature maps; then, the extracted features are aggregated through a 1×1 convolution; finally, the attention mechanism AR is designed to multiply the AR processed results by the aggregated features element-by-element to obtain refined features. The AR structure first extracts features and performs nonlinear mapping on the feature map through a 3×3 convolution layer and ReLU; then, a second 3×3 convolution layer is used to further extract features and increase the network's expressive power. Finally, Sigmoid is used to generate attention weights to enhance attention to input features. The corresponding formula is as follows:

[0134]

[0135]

[0136]

[0137] Where M is the input of the feature refinement module, Cat is the feature concatenation operation, is a 3×3 convolutional layer with an expansion rate of n, AP is a global average pooling layer, For element-by-element multiplication, AR is δ(Conv(relu(Conv(Conv(·) 1×1 ) 3×3 )) 3×3 ), δ is Sigmoid.

[0138] As can be seen, the decoder consists of three decoder blocks (DBs): DB1, DB2, and DB3. These blocks are designed to progressively recover and refine feature maps, thereby reconstructing high-quality infrared images. DB1 and DB2 use the concatenation of the upsampled features from the previous decoding stage and the MFF results from the current level as input for reconstruction, while DB3 reconstructs only the features fused by the MFF from the current level. The structure of each DB is identical to that of the EB. Since all encoded features converge before DB3, a feature refinement module (FRM) is designed to further refine and enhance the encoded features to better utilize deep features. The resulting features are then introduced into DB3.

[0139] Furthermore, the jump connection part includes a 3×3 convolution layer; after the 3×3 convolution layer extracts features from the reconstructed feature map, the extracted feature map is combined with the low-resolution infrared image to enhance the expression ability of the network and obtain a super-resolution infrared image I SR as follows:

[0140] I SR =F0+Conv(F1”) 3×3

[0141] Furthermore, the hybrid loss function L is used when training the infrared image super-resolution network model. total as follows:

[0142] L total =L pix +λ1L str +λ2L tex

[0143] Among them, L pix is the pixel loss, L str is the structural loss, L tex is the texture loss, λ1 represents the structural loss L str The weight of λ2 represents the texture loss L tex The weight of .

[0144] L pix , L str , L tex The expressions are:

[0145]

[0146]

[0147]

[0148] Where N is the number of low-resolution infrared images in the training sample, is the pixel matrix of the predicted super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample output by the infrared image super-resolution network model, is the pixel matrix of the actual super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample, 1 is the L1 norm, represents the fast Fourier transform, SSIM(·).

[0149] It should be noted that during the training of the infrared image super-resolution network model, the training and test datasets were generated. The training set used the publicly available FLIR infrared image dataset, containing 8,662 pairs of LR-HR images. The LR images were obtained by downsampling the HR images by a factor of two using Bicubic. Testing was performed on the FLIR, Thermal700, and Thermal950 datasets, with 210, 80, and 80 images, respectively, selected as the test sets. Bicubic was used to construct LR infrared images with a downsampling factor of 2.

[0150] Set the training parameters, including: using the Python framework to implement the algorithm, training the network on an NVIDIA RTX 4070TiSuper GPU, setting the batch size to 4, the maximum number of iterations (epochs) to 1000, and the learning rate to 0.00001×10 -6 , and it drops by half every 200 epochs.

[0151] The entire network is trained, with λ1 and λ2 set to 0.1 in the loss function, and the Adam optimizer is used to find the optimal solution based on the gradient descent algorithm and backpropagate to update the parameters of the entire model.

[0152] The test set is used to test the multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, and peak signal-to-noise ratio (PSNR), structural similarity (SSIM), image quality factor (UQI), image sharpness (IS), and multivariate normal distribution distance (FID) are used as evaluation indicators of the model.

[0153] First, in order to visually observe the performance differences of different methods and verify the effectiveness and generalization ability of the multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, this section selects two images from each of the FLIR, Thermal700 and Thermal950 datasets for subjective evaluation. The results are shown in the figure. Figure 8As shown, it can be seen that the reconstructed images of RNAN, CFSRCNN and RGT have similar visual appearance, and have problems such as over-smoothing and artifacts. LESRCNN, CAT and DAT find it difficult to completely restore high-frequency texture details, and the edge contours are blurred. HGSRCNN and PCS eliminate the blurring phenomenon to a certain extent, but the edge contours and details are still not clear enough, and the loss of dark details leads to a decrease in the local contrast of the image. The SR images generated by TEN and MFDRN have a mosaic phenomenon, which gives people a visually cluttered feeling and the overall visual quality is poor. The present invention can reconstruct clearer detail information and edge structures, improve the contrast and clarity to a certain extent, reduce unnecessary artifacts and noise, and have good visual effects.

[0154] By comparing the magnified areas, we can find that the method of the present invention has certain advantages in recovering high-frequency details. Figure 8 As can be seen in the first column, the contrast method fails to clearly restore the outline of the fence, exhibiting distortion. However, the image reconstructed using the present invention has sharper edges and produces more accurate results. This phenomenon is also evident in the reconstruction of the wooden stakes in the second column and the steps in the fifth column. Furthermore, the images reconstructed using the present invention have minimal noise and are more similar to the original image, as shown in the window in the third column.

[0155] In summary, the method of the present invention performs stably in different types of image SR, reflecting the robustness of the algorithm. The reconstruction results maintain a clearer appearance while restoring high-precision texture and structure, improving image contrast and visibility, and laying the foundation for subsequent image processing.

[0156] Furthermore, 210, 80, and 80 images were selected from the FLIR, Thermal700, and Thermal950 cameras, respectively, for quantitative analysis. The average values ​​of the objective metrics are shown in Table 1. The best result is indicated in bold, the second-best result is indicated by a solid underline, and the third-best result is indicated by a dashed underline. As can be seen, the proposed method demonstrates excellent performance on all three datasets, achieving optimal values ​​for all five evaluation metrics. The highest PSNR indicates that the proposed method effectively suppresses noise during reconstruction with minimal image quality loss; the highest SSIM indicates that the reconstructed image maintains a high degree of structural consistency with the HR; the highest UQI indicates that the generated image has the best visual quality; the highest IS indicates that the generated image visually meets the human eye's perception criteria; and the lowest FID indicates that the generated image has a distribution similar to the HR with minimal distortion. Comprehensive evaluation of multiple metrics indicates that the proposed method achieves optimal performance, consistent with the subjective analysis above. In summary, the proposed method is capable of reconstructing high-quality images with rich information, fine texture, and high contrast, demonstrating strong robustness.

[0157] Table 1 Average values ​​of five evaluation indicators of reconstructed images on three datasets

[0158]

[0159] Finally, the algorithm's efficiency was evaluated using runtime, model parameters, and FLOPs. The runtime was calculated using a FLIR dataset as an example, with an image size of 224×224 used for FLOPs calculation. The results are shown in Table 2. As can be seen, the proposed method runs faster, but slower than TEN and MFDRN. The algorithm's smaller number of parameters indicates its small scale, but slightly higher than TEN, LESRCNN, and MFDRN. The lower computational complexity indicates that the algorithm requires relatively little computation for reconstruction, but slightly higher than TEN, LESRCNN, and MFDRN. Combining subjective and objective evaluation results demonstrates that the proposed method achieves a better balance between reconstructed image quality and runtime efficiency.

[0160] Table 2 Comparison of running time, model parameters and FLOPs

[0161]

[0162] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may of course make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, characterized by: Inputting a low-resolution infrared image to be reconstructed into a trained infrared image super-resolution network model, and having the model output a reconstructed super-resolution infrared image; wherein the infrared image super-resolution network model includes a multi-input encoder part, a fusion part, a decoder part, and a skip connection part; The multi-input encoder part is used to extract different scale features of the low-resolution infrared image to obtain feature maps of different scales; The fusion part is used to fuse feature maps of different scales from different scales to obtain aggregated feature maps of different scales; The decoder part is used to reconstruct features of aggregated feature maps of different scales to obtain reconstructed feature maps; The jump connection part is used to fuse the reconstructed feature map and the low-resolution infrared image to obtain a super-resolution infrared image; The multi-input encoder part includes a first Bicubic unit to a third Bicubic unit, a first encoder EB1 to a third encoder EB3, a first SCM unit, and a second SCM unit; The first Bicubic unit is used to adjust the size of the low-resolution infrared image to the size required by the final output super-resolution image to obtain a feature map F0; The second Bicubic unit is used to downsample the feature map F0 by a factor of two to obtain a feature map F1; The third Bicubic unit is used to downsample the feature map F1 by a factor of two to obtain a feature map F2; The first encoder EB1 is used to extract features from the feature map F0 to obtain a first scale feature map; The first SCM unit is used to perform rough feature extraction on the feature map F1 to obtain a first rough extracted feature map; The second encoder EB2 is used to perform joint feature extraction on the first scale feature map and the first coarse extraction feature map to obtain a second scale feature map; The second SCM unit is used to perform rough feature extraction on the feature map F2 to obtain a second rough extracted feature map; The third encoder EB3 is used to perform joint feature extraction on the second scale feature map and the second coarse extraction feature map to obtain a third scale feature map; The fusion part includes the first MFF unit, the second MFF unit and the third MFF unit; wherein, any MFF unit is sequentially cascaded by a 1×1 convolution layer, a 3×3 convolution layer and a 3×3 depth-separable convolution layer, and any MFF unit obtains an aggregate feature map F MFF The method is as follows: in, is the output of the first encoder EB1, is the output of the second encoder EB2, is the output of the third encoder EB3, Conv(·) 1×1 The operation of the 1×1 convolution layer; Conv(·) 3×3 The operation of the 3×3 convolution layer; DSConv(·) 3×3 is the operation of a 3×3 depth-wise separable convolutional layer; Cat(·) represents the concatenation operation; R(·) represents the scaling operation using bicubic interpolation; The decoder part includes a first decoder DB1, a second decoder DB2 and a third decoder DB3, wherein the first decoder DB1 is composed of a 3×3 convolution unit I and a local-global feature perception unit I in cascade; the second decoder DB2 is composed of a 3×3 convolution unit II and a local-global feature perception unit II in cascade; the third decoder DB3 is composed of a feature refinement unit FRM, a 3×3 convolution unit III and a local-global feature perception unit III in cascade; The feature refinement unit FRM of the third decoder DB3 is used to enhance the aggregated feature map output by the third MFF unit to obtain an enhanced aggregated feature map; the convolution unit III is used to extract shallow features of the enhanced aggregated feature map to obtain a third decoded shallow feature map; the local-global feature perception unit III is used to extract local features of the third decoded shallow feature map to obtain a third decoded feature map; The convolution unit II of the second decoder DB2 is used to extract shallow features of the upsampled third decoding feature map and the aggregated feature map output by the second MFF unit to obtain a second decoding shallow feature map; the local-global feature perception unit II is used to extract local features of the second decoding shallow feature map to obtain a second decoding feature map; The convolution unit I of the first decoder DB1 is used to extract shallow features of the upsampled second decoding feature map and the aggregated feature map output by the first MFF unit to obtain a first decoding shallow feature map; the local-global feature perception unit I is used to extract local features of the first decoding shallow feature map to obtain a first decoding feature map; The first decoded feature map is used as the reconstructed feature map finally output by the decoder part, and the method for obtaining the reconstructed feature map finally output by the decoder part is as follows: Among them, k = 1, 2, 3 represents the decoder number, Cat(·) is the feature concatenation operation, Up(·) represents the upsampling operation, Conv(·) 3×3 is the operation of a 3×3 convolutional unit; LGFPM(·) is the operation of a local-global feature perception unit; FRM(·) is the operation of a feature refinement unit FRM, F”1~F”3 represent the inputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3 respectively, and F”'1~F”'3 represent the final outputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3 respectively.

2. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 1, characterized in that: The first encoder EB1 is composed of a cascade of convolutional unit I and a local-global feature perception unit I; the second encoder EB2 is composed of a cascade of convolutional unit II, FAM unit I and a local-global feature perception unit II; the third encoder EB3 is composed of a cascade of convolutional unit III, FAM unit II and a local-global feature perception unit III; The convolution unit I in the first encoder EB1 is used to extract shallow features of the feature map F0 to obtain the initial shallow features F0'; the local-global feature perception unit I is used to extract the features of the initial shallow features F0' to obtain the first-scale feature map; The convolution unit II in the second encoder EB2 is used to extract shallow features of the downsampled first-scale feature map to obtain a first shallow feature map; the FAM unit I is used to fuse the first shallow feature map with the first coarse extracted feature map output by the first SCM unit to obtain a first fused feature map; the local-global feature perception unit II is used to extract features from the first fused feature map to obtain a second-scale feature map; The convolution unit III in the third encoder EB3 is used to extract the shallow features of the downsampled second-scale feature map to obtain a second shallow feature map; the FAM unit II is used to fuse the second shallow feature map with the second coarse extracted feature map output by the second SCM unit to obtain a second fused feature map; the local-global feature perception unit III is used to extract the features of the second fused feature map to obtain a third-scale feature map.

3. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 2, characterized in that: Any local-global feature perception unit is composed of a cascade of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM; The local texture attention subunit LTAM is composed of a center difference convolution combination, four ordinary convolution combinations and a 1×1 convolution layer cascade; wherein the center difference convolution combination is composed of a center difference convolution layer CD Conv and an activation layer LReLU cascade, and the ordinary convolution combination is composed of a 3×3 ordinary convolution layer Conv and an activation layer LReLU cascade. The local texture feature map output by the local texture attention subunit LTAM is obtained as follows: F LTAM =Conv((Cat(F5,F4,F3,F2,F1,F in ))) 1×1 +F in F1=Lrelu(CDConv(F in ))+F in F m =F 3 (Cat(F in ,F1,...,F m-1 )),m=2,3,4,5 Among them, F LTAM is the local texture feature map finally output by the current local texture attention subunit LTAM, F in is the feature map of the central difference convolution layer CD Conv of the current local texture attention subunit LTAM, F1~F5 are the outputs of the central difference convolution combination and the four ordinary convolution combinations respectively, Conv(·) 1×1 is the operation of the 1×1 convolution layer; CDConv(·) is the operation of the center difference convolution layer CD Conv, Lrelu is the activation function of the activation layer LReLU, Cat(·) represents the splicing operation; m=1,2,3,4,5 are the numbers of the center difference convolution combination and the four ordinary convolution combinations respectively; F 3 (·)=Lrelu(Conv(·) 3×3 );Conv(·) 3×3 The operation of the 3×3 convolution layer; The global Transformer attention subunit GTAM consists of three cascaded residual structures, where the first residual structure contains a LayerNorm layer and a moving window self-attention mechanism SW-MSA; the second residual structure contains a LayerNorm layer and a channel attention mechanism CA; the third residual structure contains a LayerNorm layer and an affine multi-layer perceptron AffMLP. The scale feature map output by the global Transformer attention subunit GTAM is obtained as follows: F res1 =SW-MAS(LN(F LTAM ))+F LTAM F res2 =CA(LN(F res1 ))+F res1 F GTAM =AffMLP(LN(F res2 ))+F res2 Among them, F res1 is the output feature map of the first residual structure, SW-MAS(·) is the operation of the moving window self-attention mechanism SW-MSA, LN(·) represents the operation of the LayerNorm layer, F res2 is the output feature map of the second residual structure, CA(·) is the operation of the channel attention mechanism CA, AffMLP(·) is the operation of the affine multi-layer perceptron AffMLP, F GTAM is the output feature map of the third residual structure, and F GTAM The scale feature map is the final output of the current local-global feature perception unit.

4. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 1, characterized in that: The feature refinement unit FRM includes two 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, a global average pooling layer, and an attention mechanism AR; The aggregated feature map output by the third MFF unit is divided into five branches, each of which enters a 1×1 convolution layer, three 3×3 dilated convolution layers with dilation coefficients of 4, 8, and 12, and a global average pooling layer. The feature maps obtained by the five branches are superimposed to obtain a high-level feature map; another 1×1 convolution layer is used to perform feature aggregation on the high-level feature map to obtain a high-level aggregated feature map; the attention mechanism AR is used to perform feature enhancement on the high-level aggregated feature map to obtain a high-level enhanced feature map; The high-level aggregate feature map is element-wise multiplied with the high-level enhanced feature map to obtain the final enhanced aggregate feature map.

5. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 1, characterized in that: The jump connection part includes a 3×3 convolutional layer; After the 3×3 convolution layer extracts features from the reconstructed feature map, the extracted feature map is combined with the low-resolution infrared image to obtain a super-resolution infrared image.

6. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 1, characterized in that: The mixed loss function L used in training the infrared image super-resolution network model is total as follows: L total =L pix +λ1L str +λ2L tex Among them, L pix is the pixel loss, L str is the structural loss, L tex is the texture loss, λ1 represents the structural loss L str The weight of λ2 represents the texture loss L tex The weight of .

7. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 6, characterized in that: L pix , L str , L tex The expressions are: Where N is the number of low-resolution infrared images in the training sample, is the pixel matrix of the predicted super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample output by the infrared image super-resolution network model, is the pixel matrix of the actual super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample, ||·||1 is the L1 norm, represents the fast Fourier transform, and SSIM(·) is the structural similarity.

Images