Multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN (Convolutional Neural Network) and Transform

Abstract

According to the multi-input hierarchical infrared image super-resolution reconstruction method based on the collaborative CNN and Transform, through ingenious combination of the advantages of the CNN and the Transform, clearer detail information and edge structures can be reconstructed in a reconstruction task of an infrared image super-resolution image, the contrast ratio and definition are improved to a certain degree, and the reconstruction efficiency of the infrared image super-resolution image is improved. Unnecessary artifacts and noise are reduced, a clearer appearance is kept while high-precision textures and structures are recovered, and a good visual effect is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image super-resolution reconstruction, and in particular relates to a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. Background Art

[0002] Image super-resolution reconstruction technology aims to reconstruct high-quality super-resolution images with rich details and clear edge contours using low-resolution images as input. Infrared thermal imaging systems have the advantages of strong anti-interference, long working distance, strong penetration ability and the ability to work all day long. However, affected by the diffraction effect of infrared optics, the collected images have the disadvantages of low resolution, insufficient detail information and blurred edges. Infrared image super-resolution reconstruction technology aims to restore the high-frequency information lost in the imaging process, thereby reconstructing high-quality images with rich details and fine textures. It has been widely used in security monitoring, military reconnaissance, target detection, remote sensing and other fields.

[0003] At present, using deep learning methods to reconstruct images for super-resolution is a key focus of researchers. According to the network architecture, super-resolution reconstruction algorithms based on deep learning are mainly based on convolutional neural networks (CNNs) and are implemented through specifically designed network architectures, such as densely connected networks, generative adversarial networks, and autoencoders. In 2019, Zhang et al. proposed an RNAN for high-quality image super-resolution reconstruction based on residual structure. The reconstructed image has low noise and artifacts, but the algorithm has a large number of parameters. In view of the fact that too many convolutions and parameters usually consume high computational costs and more memory, in 2020, Tian et al. proposed a lightweight LESRCNN, which can better balance reconstruction performance and operating efficiency, but the reconstruction effect of image texture details needs to be further improved. To solve the problem of information loss, in 2021, the team proposed CFSRCNN by cascading several types of modules to combine low-resolution and high-resolution features. The experimental results show that the network can significantly improve the visual quality of the reconstructed image, but the algorithm occupies a high memory. In order to prevent the loss of original information, in 2022, Tian et al. proposed HGSRCNN based on the symmetric structure, which improved the expressiveness of images. In 2023, Zhang et al. proposed PCS, which expanded the effective receptive field of the convolutional network without adding any additional parameters or FLOPs. The network can reconstruct images with rich details and high clarity, but the algorithm has a high number of parameters.

[0004] In recent years, Transformer has achieved remarkable success in SR. Its powerful self-attention mechanism can learn the global information of the image, thereby improving the performance of the model and has been widely used in computer vision tasks. In 2022, Yang et al. introduced Transformer into SR and proposed CAT. The network mines the deep features of the image through the attention mechanism to accurately reconstruct the HR image, but the high computational complexity limits its practical application. To address this problem, in the same year, Xia et al. proposed a DAT based on a deformable self-attention module. Experiments showed that the model can focus on more relevant parts. In 2024, Zhang et al. proposed a new Transformer network RGT, which can model global spatial information while maintaining low computational cost, but the detailed texture effect is not good.

[0005] The success of deep learning in visible light SR has prompted researchers to use it for infrared SR, but infrared SR is still in the development stage due to insufficient datasets and low image resolution. In 2016, Choi et al. first applied CNN to infrared image super-resolution and proposed the TEN model, which only used three layers of CNN to represent the mapping between LR and HR images, but the network was trained using visible light data to enhance the resolution of infrared images. However, due to the difference between visible light and infrared spectra, the reconstruction accuracy is limited. In 2023, Wu et al. proposed MFDRN based on information distillation, which achieved network lightweight, but the performance needs to be further improved.

[0006] Although CNN-based algorithms have achieved good results, they are limited by the inherent weaknesses of the convolution operator. The network can only extract features locally and it is difficult to capture global dependencies. Transformer has achieved remarkable success in SR. However, the existing self-attention mechanism often ignores the combination of contextual features, which affects network performance. Infrared images usually have problems such as low resolution, insufficient detail information, and blurred edges. How to make full use of the local and global features of infrared images and efficiently aggregate rich features is an important issue. Summary of the invention

[0007] To solve the above problems, the present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, which can make full use of the texture and structure information contained in the low-resolution infrared image, reconstruct a super-resolution image close to the real scene, and improve the image reconstruction performance.

[0008] A multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, wherein a low-resolution infrared image to be reconstructed is input into a trained infrared image super-resolution network model, and the model outputs a reconstructed super-resolution infrared image; wherein the infrared image super-resolution network model includes a multi-input encoder part, a fusion part, a decoder part and a skip connection part;

[0009] The multi-input encoder part is used to extract different scale features of the low-resolution infrared image to obtain feature maps of different scales;

[0010] The fusion part is used to fuse feature maps of different scales from different scales to obtain aggregated feature maps of different scales;

[0011] The decoder part is used to perform feature reconstruction on aggregated feature maps of different scales to obtain reconstructed feature maps;

[0012] The jump connection part is used to fuse the reconstructed feature map and the low-resolution infrared image to obtain a super-resolution infrared image.

[0013] Further, the multi-input encoder part includes a first Bicubic unit to a third Bicubic unit, a first encoder EB1 to a third encoder EB3, a first SCM unit, and a second SCM unit;

[0014] The first Bicubic unit is used to adjust the size of the low-resolution infrared image to the size required by the final output super-resolution image to obtain a feature map F0;

[0015] The second Bicubic unit is used to downsample the feature map F0 by a factor of two to obtain a feature map F1;

[0016] The third Bicubic unit is used to downsample the feature map F1 by a factor of two to obtain a feature map F2;

[0017] The first encoder EB1 is used to extract features from the feature map F0 to obtain a first scale feature map;

[0018] The first SCM unit is used to perform rough feature extraction on the feature map F1 to obtain a first rough extracted feature map;

[0019] The second encoder EB2 is used to perform joint feature extraction on the first scale feature map and the first coarse extraction feature map to obtain a second scale feature map;

[0020] The second SCM unit is used to perform rough feature extraction on the feature map F2 to obtain a second rough extracted feature map;

[0021] The third encoder EB3 is used to perform joint feature extraction on the second scale feature map and the second coarse extraction feature map to obtain a third scale feature map.

[0022] Furthermore, the first encoder EB1 is composed of a convolution unit I and a local-global feature perception unit I in cascade sequence; the second encoder EB2 is composed of a convolution unit II, a FAM unit I and a local-global feature perception unit II in cascade sequence; the third encoder EB3 is composed of a convolution unit III, a FAM unit II and a local-global feature perception unit III in cascade sequence;

[0023] The convolution unit I in the first encoder EB1 is used to extract the shallow features of the feature map F0 to obtain the initial shallow features F0'; the local-global feature perception unit I is used to extract the features of the initial shallow features F0' to obtain the first scale feature map;

[0024] The convolution unit II in the second encoder EB2 is used to extract shallow features of the downsampled first scale feature map to obtain a first shallow feature map; the FAM unit I is used to perform feature fusion on the first shallow feature map and the first coarse extraction feature map output by the first SCM unit to obtain a first fused feature map; the local-global feature perception unit II is used to extract features of the first fused feature map to obtain a second scale feature map;

[0025] The convolution unit III in the third encoder EB3 is used to extract the shallow features of the downsampled second-scale feature map to obtain a second shallow feature map; the FAM unit II is used to fuse the second shallow feature map with the second coarse extraction feature map output by the second SCM unit to obtain a second fused feature map; the local-global feature perception unit III is used to extract the features of the second fused feature map to obtain a third-scale feature map.

[0026] Furthermore, any local-global feature perception unit is composed of a cascade of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM;

[0027] The local texture attention subunit LTAM is composed of a center difference convolution combination, four ordinary convolution combinations and a 1×1 convolution layer cascade in sequence; wherein the center difference convolution combination is composed of a center difference convolution layer CD Conv and an activation layer LReLU cascade, and the ordinary convolution combination is composed of a 3×3 ordinary convolution layer Conv and an activation layer LReLU cascade. The method for obtaining the local texture feature map output by the local texture attention subunit LTAM is as follows:

[0028] F LTAM =Conv((Cat(F5,F4,F3,F2,F1,F in ))) 1×1 +Fin

[0029] F1=Lrelu(CDConv(F in ))+F in

[0030] F m =F 3 (Cat(F in ,F1,...,F m-1 )),m=2,3,4,5

[0031] Among them, F LTAM is the local texture feature map finally output by the current local texture attention subunit LTAM, F in is the feature map of the central difference convolution layer CD Conv that inputs the current local texture attention subunit LTAM, F1~F5 are the outputs of the central difference convolution combination and the four ordinary convolution combinations respectively, Conv(·) 1×1 is the operation of the 1×1 convolution layer; CDConv(·) is the operation of the center difference convolution layer CD Conv, Lrelu is the activation function of the activation layer LReLU, Cat(·) represents the concatenation operation; m=1,2,3,4,5 are the numbers of the center difference convolution combination and the four ordinary convolution combinations respectively; F 3 (·)=Lrelu(Conv(·) 3×3 );Conv(·) 3×3 The operation of the 3×3 convolution layer;

[0032] The global Transformer attention subunit GTAM is composed of three cascaded residual structures, wherein the first residual structure includes a LayerNorm layer and a moving window self-attention mechanism SW-MSA; the second residual structure includes a LayerNorm layer and a channel attention mechanism CA; the third residual structure includes a LayerNorm layer and an affine multilayer perceptron AffMLP, then the method for obtaining the scale feature map output by the global Transformer attention subunit GTAM is as follows:

[0033] F res1 =SW-MAS(LN(F LTAM ))+F LTAM

[0034] F res2 =CA(LN(F res1 ))+F res1

[0035] F GTAM =AffMLP(LN(F res2 ))+Fres2

[0036] Among them, F res1 is the output feature map of the first residual structure, SW-MAS(·) is the operation of the moving window self-attention mechanism SW-MSA, LN(·) represents the operation of the LayerNorm layer, F res2 is the output feature map of the second residual structure, CA(·) is the operation of the channel attention mechanism CA, AffMLP(·) is the operation of the affine multilayer perceptron AffMLP, F GTAM is the output feature map of the third residual structure, and F GTAM The scale feature map that is the final output of the current local-global feature perception unit.

[0037] Furthermore, the fusion part includes a first MFF unit, a second MFF unit and a third MFF unit; wherein any MFF unit is sequentially cascaded by a 1×1 convolution layer, a 3×3 convolution layer and a 3×3 depth-separable convolution layer, and any MFF unit obtains an aggregate feature map F MFF The method is as follows:

[0038]

[0039] in, is the output of the first encoder EB1, is the output of the second encoder EB2, is the output of the third encoder EB3, Conv(·) 1×1 The operation of the 1×1 convolutional layer; Conv(·) 3×3 The operation of the 3×3 convolution layer; DSConv(·) 3×3 is the operation of a 3×3 depthwise separable convolutional layer; Cat(·) represents a concatenation operation; R(·) represents a scaling operation using bicubic interpolation.

[0040] Furthermore, the decoder part includes a first decoder DB1, a second decoder DB2 and a third decoder DB3, wherein the first decoder DB1 is sequentially composed of a 3×3 convolution unit I and a local-global feature perception unit I in cascade; the second decoder DB2 is sequentially composed of a 3×3 convolution unit II and a local-global feature perception unit II in cascade; the third decoder DB3 is sequentially composed of a feature refinement unit FRM, a 3×3 convolution unit III and a local-global feature perception unit III in cascade;

[0041] The feature refinement unit FRM of the third decoder DB3 is used to enhance the features of the aggregate feature map output by the third MFF unit to obtain an enhanced aggregate feature map; the convolution unit III is used to extract the shallow features of the enhanced aggregate feature map to obtain a third decoded shallow feature map; the local-global feature perception unit III is used to extract the local features of the third decoded shallow feature map to obtain a third decoded feature map;

[0042] The convolution unit II of the second decoder DB2 is used to extract the shallow features of the upsampled third decoding feature map and the aggregate feature map output by the second MFF unit to obtain the second decoding shallow feature map; the local-global feature perception unit II is used to extract the local features of the second decoding shallow feature map to obtain the second decoding feature map;

[0043] The convolution unit I of the first decoder DB1 is used to extract the shallow features of the upsampled second decoding feature map and the aggregated feature map output by the first MFF unit to obtain the first decoding shallow feature map; the local-global feature perception unit I is used to extract the local features of the first decoding shallow feature map to obtain the first decoding feature map;

[0044] The first decoded feature map is used as the reconstructed feature map finally output by the decoder part, and the method for obtaining the reconstructed feature map finally output by the decoder part is as follows:

[0045]

[0046] Where k = 1, 2, 3 represents the decoder number, Cat(·) is the feature concatenation operation, Up(·) represents the upsampling operation, Conv(·) represents the decoder number, and Conv(·) represents the decoder number. 3×3 is the operation of a 3×3 convolutional unit; LGFPM(·) is the operation of a local-global feature perception unit; FRM(·) is the operation of a feature refinement unit FRM, F1”~F3” represent the inputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3, respectively, and F1”’~F3”’ represent the final outputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3, respectively.

[0047] Furthermore, the feature refinement unit FRM includes two 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, respectively, a global average pooling layer, and an attention mechanism AR;

[0048] The aggregated feature map output by the third MFF unit is divided into five branches, which enter one 1×1 convolution layer, three 3×3 dilated convolution layers with dilation coefficients of 4, 8, and 12 respectively, and a global average pooling layer. The feature maps obtained by the five branches are superimposed to obtain a high-level feature map; another 1×1 convolution layer is used to perform feature aggregation on the high-level feature map to obtain a high-level aggregated feature map; the attention mechanism AR is used to perform feature enhancement on the high-level aggregated feature map to obtain a high-level enhanced feature map;

[0049] The high-level aggregate feature map is element-wise multiplied with the high-level enhanced feature map to obtain the final enhanced aggregate feature map.

[0050] Furthermore, the jump connection part includes a 3×3 convolutional layer;

[0051] After the 3×3 convolution layer extracts features from the reconstructed feature map, the extracted feature map is combined with the low-resolution infrared image to obtain a super-resolution infrared image.

[0052] Furthermore, the mixed loss function L used in training the infrared image super-resolution network model is total as follows:

[0053] L total =L pix +λ1L str +λ2L tex

[0054] Among them, L pix is the pixel loss, L str is the structural loss, L tex is the texture loss, λ1 represents the structural loss L str The weight of λ2 represents the texture loss L tex The weight of .

[0055] Furthermore, L pix , L str , L tex The expressions are:

[0056]

[0057]

[0058]

[0059] Where N is the number of low-resolution infrared images in the training samples, is the pixel matrix of the predicted super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample output by the infrared image super-resolution network model, is the pixel matrix of the actual super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample, ·1 is the L1 norm, represents fast Fourier transform, and SSIM(·) is the structural similarity.

[0060] Beneficial effects:

[0061] 1. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. By cleverly combining the advantages of CNN and Transformer, in the reconstruction task of infrared image super-resolution images, clearer detail information and edge structure can be reconstructed, and the contrast and clarity can be improved to a certain extent, and unnecessary artifacts and noise can be reduced. While restoring high-precision texture and structure, a clearer appearance is maintained, and a good visual effect is achieved.

[0062] 2. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. The local-global feature perception unit is composed of a carefully designed local texture attention subunit and a global Transformer attention subunit. It can simultaneously capture local texture details and global structures from infrared images of different scales, and can reconstruct high-quality images with rich information, fine texture, and high contrast, and has strong robustness.

[0063] 3. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer. The feature refinement unit can effectively extract high-level context features and local details by combining different convolution receptive fields, realize mutual complementation of features, and enhance the expression ability of image details.

[0064] 4. The present invention provides a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, which fuses the features from different scales and depths and can obtain rich feature representations to better reconstruct high-quality images. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 A flow chart of the design method of the infrared image super-resolution network model provided by the present invention;

[0066] Figure 2 The principle block diagram of the infrared image super-resolution network model provided by the present invention;

[0067] Figure 3 A principle block diagram of a local-global feature perception unit provided by the present invention;

[0068] Figure 4A principle block diagram of a local texture attention subunit in a local-global feature perception unit provided by the present invention;

[0069] Figure 5 A block diagram of the principle of the global Transformer attention subunit in the local-global feature perception unit provided by the present invention;

[0070] Figure 6 A principle block diagram of a feature refinement unit provided by the present invention;

[0071] Figure 7 The principle block diagram of the MFF unit provided by the present invention;

[0072] Figure 8 A comparison chart of reconstruction results of different algorithms provided by the present invention, from top to bottom: HR, RNAN, LESRCNN, CFSRCNN, HGSRCNN, CAT, DAT, PCS, RGT, TEN, MFDRN and the present invention. DETAILED DESCRIPTION

[0073] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0074] In order to make full use of the texture and structure information contained in the low-resolution infrared image and reconstruct an image close to the real scene, the present invention proposes a multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, inputting the low-resolution infrared image to be reconstructed into a trained infrared image super-resolution network model, and the model outputs the reconstructed super-resolution infrared image to improve the image reconstruction performance; wherein, the infrared image super-resolution network model includes a multi-input encoder part, a fusion part, a decoder part and a jump connection part;

[0075] The multi-input encoder part is used to extract different scale features of the low-resolution infrared image to obtain feature maps of different scales;

[0076] The fusion part is used to fuse feature maps of different scales from different scales to obtain aggregated feature maps of different scales;

[0077] The decoder part is used to perform feature reconstruction on aggregated feature maps of different scales to obtain reconstructed feature maps;

[0078] The jump connection part is used to fuse the reconstructed feature map and the low-resolution infrared image to obtain a super-resolution infrared image.

[0079] like Figure 1As shown, the design method of the infrared image super-resolution network model provided by the present invention comprises the following steps:

[0080] Step 1: Use the multi-input encoder-decoder structure as a framework to build an infrared image super-resolution network model. The model is divided into four parts: the multi-input encoder part, the fusion part, the decoder part, and the jump connection part.

[0081] Step 2: Construct a local-global feature perception unit LGFPM consisting of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM to simultaneously capture local texture details and global structures from infrared images of different scales.

[0082] Step 3: Design the feature refinement module FRM. By combining different convolution receptive fields, it can effectively extract high-level contextual features and local details, achieve mutual complementation of features, and enhance the expression ability of image details.

[0083] Step 4: Design a multi-level feature fusion module MFF to fuse features from different scales and depths to obtain rich feature representations to better reconstruct high-quality images.

[0084] Step 5: Construct a hybrid loss function consisting of pixel loss, structure loss, and texture loss to guide network optimization in a supervised learning manner.

[0085] Step 6: Use a multi-input encoder-decoder structure as a framework to build a network model, use a super-resolution network training set to train the network model, determine the optimal values ​​of the parameters in the network model through parameter optimization, and obtain an infrared image super-resolution network model.

[0086] like Figure 2 As shown, the multi-input encoder part includes a first Bicubic unit to a third Bicubic unit, a first encoder EB1 to a third encoder EB3, a first SCM unit, and a second SCM unit;

[0087] The first Bicubic unit is used to adjust the size of the low-resolution infrared image to the size required by the final output super-resolution image to obtain a feature map F0; the second Bicubic unit is used to downsample the feature map F0 by two times to obtain a feature map F1; the third Bicubic unit is used to downsample the feature map F1 by two times to obtain a feature map F2; the first encoder EB1 is used to extract features from the feature map F0 to obtain a first scale feature map; the first SCM unit is used to perform coarse feature extraction on the feature map F1 to obtain a first coarse extracted feature map; the second encoder EB2 is used to perform joint feature extraction on the first scale feature map and the first coarse extracted feature map to obtain a second scale feature map; the second SCM unit is used to perform coarse feature extraction on the feature map F2 to obtain a second coarse extracted feature map; the third encoder EB3 is used to perform joint feature extraction on the second scale feature map and the second coarse extracted feature map to obtain a third scale feature map.

[0088] Furthermore, the first encoder EB1 is composed of a convolution unit I and a local-global feature perception unit I in cascade sequence; the second encoder EB2 is composed of a convolution unit II, a FAM unit I and a local-global feature perception unit II in cascade sequence; the third encoder EB3 is composed of a convolution unit III, a FAM unit II and a local-global feature perception unit III in cascade sequence;

[0089] The convolution unit I in the first encoder EB1 is used to extract the shallow features of the feature map F0 to obtain the initial shallow features F0'; the local-global feature perception unit I is used to extract the local features of the initial shallow features F0' to obtain the first scale feature map;

[0090] The convolution unit II in the second encoder EB2 is used to extract shallow features of the first scale feature map after downsampling to obtain a first shallow feature map; the FAM unit I is used to perform feature fusion on the first shallow feature map and the first coarse extraction feature map output by the first SCM unit to obtain a first fused feature map; the local-global feature perception unit II is used to extract local features of the first fused feature map to obtain a second scale feature map;

[0091] The convolution unit III in the third encoder EB3 is used to extract the shallow features of the downsampled second-scale feature map to obtain a second shallow feature map; the FAM unit I is used to fuse the second shallow feature map with the second coarse extraction feature map output by the second SCM unit to obtain a second fused feature map; the local-global feature perception unit II is used to extract the local features of the second fused feature map to obtain a third-scale feature map.

[0092] That is, the multi-input encoder part is used to extract features of different scales of the input image; the fusion part fuses the encoded features of multiple stages to achieve full feature aggregation. The aggregated features are reconstructed in the decoder part. Finally, the texture details of the input image are transmitted to the output image through the jump connection part to improve the image reconstruction capability.

[0093] Specifically, for the input LR image I LR ∈R H×W×C First, adjust the image size through Bicubic to get the output SR image I SR ∈R tH×tW×C The feature maps of the same size F0∈R tH×tW×C :

[0094] F0=Bicubic ↑t (I LR )

[0095] In the formula, H, W and C in are the height, width and number of channels of the input low-resolution image respectively, and t is the scale factor.

[0096] Next, feature extraction is performed on F0 through the multi-input encoder part, which consists of three encoder blocks (EBs) EB1, EB2, and EB3, where EB1 is used to extract information from the original scale, and EB2 and EB3 extract information from the features after downsampling in the previous encoding stage. Each EB uses a 3×3 convolutional layer to extract shallow features and a local-global feature perception module (LGFPM) to extract local features of the image and build long-range dependencies. To solve the problem of information loss caused by downsampling operations, F0 is continuously downsampled, and features are extracted through SCM, which are then combined with shallow features from the EB2 and EB3 encoding stages through FAM to perform LGFPM operations. The encoding process can be expressed as:

[0097]

[0098] Where k is the index of EB, F′ k EB k Encoded feature map. Conv(·) 3×3 Represents a 3×3 convolutional layer. f1 = Bicubic ↓2 (F0), f2 = Bicubic ↓2 (f1). Down represents the downsampling block, which consists of a 3×3 convolutional layer with a stride of 2 and a LeakyReLU. After downsampling, the size of the output feature map is halved and the number of channels is doubled.

[0099] Furthermore, if Figure 3As shown, any local-global feature perception unit LGFPM is composed of a cascade of local texture attention subunit LTAM and a global Transformer attention subunit GTAM, which are used for local and global feature extraction respectively;

[0100] like Figure 4 As shown, the local texture attention subunit LTAM is composed of a center difference convolution combination, four ordinary convolution combinations and a 1×1 convolution layer cascade in sequence; wherein, the center difference convolution combination is composed of a center difference convolution layer CD Conv and an activation layer LReLU cascade, and the ordinary convolution combination is composed of a 3×3 ordinary convolution layer Conv and an activation layer LReLU cascade. The method for obtaining the local texture feature map output by the local texture attention subunit LTAM is as follows:

[0101] F LTAM =Conv((Cat(F5,F4,F3,F2,F1,F in ))) 1×1 +F in

[0102] F1=Lrelu(CDConv(F in ))+F in

[0103] F m =F 3 (Cat(F in ,F1,...,F m-1 )),m=2,3,4,5

[0104] Among them, F LTAM is the local texture feature map finally output by the current local texture attention subunit LTAM, F in is the feature map of the central difference convolution layer CD Conv that inputs the current local texture attention subunit LTAM, F1~F5 are the outputs of the central difference convolution combination and the four ordinary convolution combinations respectively, Conv(·) 1×1 is the operation of the 1×1 convolution layer; CDConv(·) is the operation of the center difference convolution layer CD Conv, Lrelu is the activation function of the activation layer LReLU, Cat(·) represents the concatenation operation; m=1,2,3,4,5 are the numbers of the center difference convolution combination and the four ordinary convolution combinations respectively; F 3 (·)=Lrelu(Conv(·) 3×3 );Conv(·) 3×3 The operation of the 3×3 convolution layer;

[0105] That is, the local texture attention module (LTAM) consists of a center difference convolution and a traditional convolution. First, a center difference convolution CD Conv and four 3×3 convolution layers are used to extract the deep features of the input. After each layer, LeakyReLU is used to increase the nonlinear expression ability. To enhance the ability to extract details, the features extracted by the previous layer are reused in the DenseNet way. Then, the hierarchical features are concatenated and the number of feature map channels is adjusted through a 1×1 convolution layer. Finally, a jump connection is used to achieve the aggregation of input details and deep feature maps.

[0106] like Figure 5 As shown, the global Transformer attention subunit GTAM is composed of three residual structures cascaded, wherein the first residual structure includes a LayerNorm layer and a moving window self-attention mechanism SW-MSA; the second residual structure includes a LayerNorm layer and a channel attention mechanism CA; the third residual structure includes a LayerNorm layer and an affine multilayer perceptron AffMLP, then the method for obtaining the scale feature map output by the global Transformer attention subunit GTAM is as follows:

[0107] F res1 =SW-MAS(LN(F LTAM ))+F LTAM

[0108] F res2 =CA(LN(F res1 ))+F res1

[0109] F GTAM =AffMLP(LN(F res2 ))+F res2

[0110] Among them, F res1 is the output feature map of the first residual structure, SW-MAS(·) is the operation of the moving window self-attention mechanism SW-MSA, LN(·) represents the operation of the LayerNorm layer, F res2 is the output feature map of the second residual structure, CA(·) is the operation of the channel attention mechanism CA, AffMLP(·) is the operation of the affine multilayer perceptron AffMLP, F GTAM is the output feature map of the third residual structure, and F GTAM The scale feature map that is the final output of the current local-global feature perception unit.

[0111] It should be noted that in order to solve the problem that the moving window self-attention mechanism lacks attention to channel information, the channel attention mechanism CA(·) is used to improve the interdependence between channels. This module does not perform window partitioning, but uses linear mapping to project the input into Q, K and V matrices. By comparing Q T and K, obtain the attention feature map, and compare it with V T Multiply, and the final result is reshaped into a feature map of the same size as the input of the structure. The corresponding formula is as follows:

[0112]

[0113] In the formula, F CA is the output of the channel attention mechanism, Q = XP Q , K = XP K 、V=XP V The X is the input of the channel attention mechanism, P Q , P K and P V is the linear transformation matrix. d is the dimension of Q or K, softmax is the Softmax activation function, and R is the reshaping of the dimension.

[0114] The affine multilayer perceptron is used to improve the robustness of the model. First, the affine transformation layer is used to make the network more stable without increasing the training cost. Then, the features are mapped to a high-dimensional space through the linear transformation layer, and ReLU is used to increase the nonlinear expression ability. Then, the original dimension is restored through the linear transformation layer. Finally, the affine transformation layer is introduced again to increase the flexibility of the network. The corresponding formula is as follows:

[0115] F AffMLP =Aff(linear(relu(linear(Aff(f in )))))

[0116] In the formula, f in is the input of the affine multilayer perceptron, linear is the linear transformation layer, relu is the ReLU activation function, and Aff is the affine transformation layer.

[0117] Furthermore, if Figure 2 As shown, the multi-input fusion part includes a first MFF unit, a second MFF unit and a third MFF unit; wherein, Figure 7 As shown in Figure 2, any MFF unit is composed of a 1×1 convolutional layer, a 3×3 convolutional layer, and a 3×3 depth-separable convolutional layer cascaded in sequence. Then any MFF unit obtains the aggregate feature map F MFF The method is as follows:

[0118]

[0119] in, is the output of the first encoder EB1, is the output of the second encoder EB2, is the output of the third encoder EB3, Conv(·) 1×1 The operation of the 1×1 convolutional layer; Conv(·) 3×3 The operation of the 3×3 convolution layer; DSConv(·) 3×3 is the operation of a 3×3 depthwise separable convolutional layer; Cat(·) represents a concatenation operation; R(·) represents a scaling operation using bicubic interpolation.

[0120] That is to say, in order to make full use of the multi-scale features extracted in the encoding stage, the fusion part simultaneously fuses the features extracted from each EB through the multi-level feature fusion (MFF) unit, and the results at different levels can be expressed as:

[0121]

[0122] In the formula, F k ” is the fusion result of the kth level. (·) ↑2 Bicubic double upsampling, (·) ↓4 、(·) ↓2 Bicubic is downsampled four times and two times respectively.

[0123] It can be seen that the fusion part of the present invention takes the output of the encoder block as input and maps the fusion result to the decoder block. First, the bicubic difference is used to scale the coded features of inconsistent sizes to unify the size of the feature map; then, the scaled features are spliced ​​along the channel dimension and a 1×1 convolution layer is used to generate a preliminary fusion result; then a 3×3 convolution layer is used to extract shallow features. Finally, a 3×3 depth-separable convolution layer is used to better focus on the key details of the features.

[0124] Furthermore, if Figure 2 As shown, the decoder part includes a first decoder DB1, a second decoder DB2 and a third decoder DB3, wherein the first decoder DB1 is sequentially composed of a 3×3 convolution unit I and a local-global feature perception unit I in cascade; the second decoder DB2 is sequentially composed of a 3×3 convolution unit II and a local-global feature perception unit II in cascade; the third decoder DB3 is sequentially composed of a feature refinement unit FRM, a 3×3 convolution unit III and a local-global feature perception unit III in cascade;

[0125] The feature refinement unit FRM of the third decoder DB3 is used to enhance the features of the aggregate feature map output by the third MFF unit to obtain an enhanced aggregate feature map; the convolution unit III is used to extract the shallow features of the enhanced aggregate feature map to obtain a third decoded shallow feature map; the local-global feature perception unit III is used to extract the local features of the third decoded shallow feature map to obtain a third decoded feature map;

[0126] The convolution unit II of the second decoder DB2 is used to extract the shallow features of the upsampled third decoding feature map and the aggregate feature map output by the second MFF unit to obtain the second decoding shallow feature map; the local-global feature perception unit II is used to extract the local features of the second decoding shallow feature map to obtain the second decoding feature map;

[0127] The convolution unit I of the first decoder DB1 is used to extract the shallow features of the upsampled second decoding feature map and the aggregated feature map output by the first MFF unit to obtain the first decoding shallow feature map; the local-global feature perception unit I is used to extract the local features of the first decoding shallow feature map to obtain the first decoding feature map;

[0128] The first decoded feature map is used as the reconstructed feature map finally output by the decoder part, and the method for obtaining the reconstructed feature map finally output by the decoder part is as follows:

[0129]

[0130] Among them, k = 1, 2, 3 represents the decoder number, Cat(·) is the feature concatenation operation, Up(·) represents the upsampling operation, which consists of a 3×3 transposed convolution layer with a step size of 2 and a LeakyReLU. After upsampling, the number of channels in the output feature map is halved and the size is doubled; Conv(·) 3×3 is the operation of a 3×3 convolutional unit; LGFPM(·) is the operation of a local-global feature perception unit; FRM(·) is the operation of a feature refinement unit FRM, F1”~F3” represent the inputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3, respectively, and F1”’~F3”’ represent the final outputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3, respectively.

[0131] like Figure 6 As shown, the feature refinement unit FRM includes two 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, a global average pooling layer, and an attention mechanism AR;

[0132] The aggregated feature map output by the third MFF unit is divided into five branches, which respectively enter one of the 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, and a global average pooling layer. The feature maps obtained by the five branches are superimposed to obtain a high-level feature map; another 1×1 convolutional layer is used to perform feature aggregation on the high-level feature map to obtain a high-level aggregated feature map; the attention mechanism AR is used to perform feature enhancement on the high-level aggregated feature map to obtain a high-level enhanced feature map; the high-level aggregated feature map is element-wise multiplied with the high-level enhanced feature map to obtain the final enhanced aggregated feature map.

[0133] That is to say, the feature refinement unit FRM samples the input tensor in parallel through a 1×1 convolution, three 3×3 dilated convolutions with dilation coefficients of 4, 8, and 12, and a global average pooling layer. The three dilated convolution branches use a layer-by-layer stacking method to enable the network to extract richer high-level features; the feature maps obtained by the five branches are superimposed to increase the information interaction of the feature maps; then, the extracted features are aggregated through a 1×1 convolution; finally, the attention mechanism AR is designed to multiply the AR processed results by the aggregated features element by element to obtain refined features. The AR structure first extracts features and nonlinearly maps the feature map through a 3×3 convolution layer and ReLU; then, the second 3×3 convolution layer is used to further extract features and increase the network's expressiveness. Finally, Sigmoid is used to generate attention weights to enhance attention to input features. The corresponding formula is as follows:

[0134]

[0135]

[0136]

[0137] In the formula, M is the input of the feature refinement module, Cat is the feature concatenation operation, is a 3×3 convolutional layer with a dilation rate of n, AP is a global average pooling layer, is element-by-element multiplication, AR is δ(Conv(relu(Conv(Conv(·) 1×1 ) 3×3 )) 3×3 ), δ is Sigmoid.

[0138] It can be seen that the decoder part contains three decoder blocks (DB), namely DB1, DB2 and DB3, which are designed to gradually restore and refine the feature map to reconstruct high-quality infrared images. DB1 and DB2 take the upsampled features of the previous decoding stage and the cascade of the MFF results of the current level as input and reconstruct them, while DB3 only reconstructs the features fused by the MFF of the current level. The structure of each DB is the same as that of EB. Since all the encoded features converge before DB3, in order to better utilize the deep features, the feature refinement module (FRM) is designed to further refine and enhance the encoded features and introduce the results into DB3.

[0139] Furthermore, the jump connection part includes a 3×3 convolution layer; after the 3×3 convolution layer extracts features from the reconstructed feature map, the extracted feature map is combined with the low-resolution infrared image to enhance the expression ability of the network and obtain a super-resolution infrared image I SR as follows:

[0140] I SR =F0+Conv(F1”) 3×3

[0141] Furthermore, the mixed loss function L used in training the infrared image super-resolution network model is total as follows:

[0142] L total =L pix +λ1L str +λ2L tex

[0143] Among them, L pix is the pixel loss, L str is the structural loss, L tex is the texture loss, λ1 represents the structural loss L str The weight of λ2 represents the texture loss L tex The weight of .

[0144] L pix , L str , L tex The expressions are:

[0145]

[0146]

[0147]

[0148] Where N is the number of low-resolution infrared images in the training samples, is the pixel matrix of the predicted super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample output by the infrared image super-resolution network model, is the pixel matrix of the actual super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample, ·1 is the L1 norm, represents fast Fourier transform, SSIM(·).

[0149] It should be noted that during the training of the infrared image super-resolution network model, the training and test data sets produced include: using the public infrared image dataset FLIR for training, the training set contains 8662 pairs of LR-HR images, where the LR images are obtained by bicubic two-fold downsampling of the HR images. Testing was performed on the FLIR, Thermal700, and Thermal950 datasets, selecting 210, 80, and 80 images as test sets respectively, and using Bicubic to construct LR infrared images with a downsampling factor of 2.

[0150] Set the training parameters, including: using the pytoch framework in Python to implement the algorithm, training the network based on the NVIDIA RTX 4070TiSuper GPU, setting the batch size to 4, the maximum number of iterations (epoch) to 1000, and the learning rate to 0.00001×10 -6 , and it decreases by half every 200 epochs.

[0151] The entire network is trained, λ1 and λ2 in the loss function are set to 0.1, and the Adam optimizer is used to find the optimal solution based on the gradient descent algorithm and back-propagate to update the parameters of the entire model.

[0152] The test set is used to test the multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, and peak signal-to-noise ratio (PSNR), structural similarity (SSIM), image quality factor (UQI), image clarity (IS), and multivariate normal distribution distance (FID) are used as evaluation indicators of the model.

[0153] First, in order to visually observe the performance differences of different methods and verify the effectiveness and generalization ability of the multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, this section selects two images from each of the FLIR, Thermal700 and Thermal950 datasets for subjective evaluation. The results are shown in Figure 2. Figure 8As shown, it can be seen that the reconstructed images of RNAN, CFSRCNN and RGT have similar visual appearance, and there are problems such as over-smoothing and artifacts. LESRCNN, CAT and DAT are difficult to fully restore high-frequency texture details, and the edge contours are blurred. HGSRCNN and PCS eliminate the blurring phenomenon to a certain extent, but the edge contours and details are still not clear enough, and the loss of dark details leads to a decrease in the local contrast of the image. The SR images generated by TEN and MFDRN have a mosaic phenomenon, which gives people a visually chaotic feeling and the overall visual quality is poor. The present invention can reconstruct clearer detail information and edge structures, improve the contrast and clarity to a certain extent, reduce unnecessary artifacts and noise, and have good visual effects.

[0154] By comparing the magnified area, it can be found that the method of the present invention has certain advantages in the restoration of high-frequency details. Figure 8 As can be seen in the first column, the contrast method cannot clearly restore the outline of the fence, and deformation occurs, while the image reconstructed by the present invention has sharper edges and more accurate results, which is also fully demonstrated by the reconstruction of the wooden stakes in the second column and the steps in the fifth column. In addition, the image reconstructed by the present invention has the least noise and is more similar to the original image, such as the window in the third column.

[0155] In summary, the method of the present invention performs stably in different types of image SR, which reflects the robustness of the algorithm. The reconstruction results restore high-precision texture and structure while maintaining a clearer appearance, improving image contrast and visibility, and laying the foundation for subsequent image processing.

[0156] Furthermore, 210, 80 and 80 images were selected from FLIR, Thermal700 and Thermal950 for quantitative analysis respectively. The average values ​​of objective indicators are shown in Table 1. The best result is indicated by bold, the second best result is indicated by underline solid line, and the third best result is indicated by underline dotted line. It can be seen that the proposed method shows excellent performance on the three data sets, and the five evaluation indicators are all optimal values. Among them, the largest PSNR shows that the proposed method effectively suppresses noise during the reconstruction process, and the image quality loss is minimal; the largest SSIM shows that the reconstructed image of the proposed method is highly consistent with HR in structure; the largest UQI shows that the image generated by the proposed method has the best visual quality; the largest IS shows that the image generated by the proposed method visually meets the perception standard of the human eye; the smallest FID shows that the image generated by the proposed method is similar to the distribution of HR and has the smallest distortion. Comprehensive multi-index evaluation shows that the proposed method achieves the best performance, which is consistent with the above subjective analysis. In summary, the proposed method can reconstruct high-quality images with rich information, fine texture and high contrast, and has strong robustness.

[0157] Table 1 Average values ​​of five evaluation indicators of reconstructed images on three datasets

[0158]

[0159] Finally, the efficiency of the algorithm is evaluated by running time, model parameters and FLOPs. The running time is taken as an example to process the FLIR data set. The image size used to calculate FLOPs is 224×224. The results are shown in Table 2. It can be seen that the method of the present invention runs faster, but slower than TEN and MFDRN. The number of parameters of the algorithm in this paper is small, indicating that the scale of the method of the present invention is small, but slightly higher than TEN, LESRCNN and MFDRN. The lower computational complexity indicates that the amount of calculation required by the algorithm during reconstruction is also relatively small, but slightly higher than TEN, LESRCNN and MFDRN. Combining the subjective and objective evaluation results, it is shown that the method of this paper can better balance the quality of the reconstructed image and the running efficiency.

[0160] Table 2 Comparison of running time, model parameters and FLOPs

[0161]

[0162] Of course, the present invention may have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may certainly make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer, characterized in that: Inputting the low-resolution infrared image to be reconstructed into the trained infrared image super-resolution network model, and the model outputs the reconstructed super-resolution infrared image; wherein the infrared image super-resolution network model includes a multi-input encoder part, a fusion part, a decoder part and a skip connection part; The multi-input encoder part is used to extract different scale features of the low-resolution infrared image to obtain feature maps of different scales; The fusion part is used to fuse feature maps of different scales from different scales to obtain aggregated feature maps of different scales; The decoder part is used to perform feature reconstruction on aggregated feature maps of different scales to obtain reconstructed feature maps; The jump connection part is used to fuse the reconstructed feature map and the low-resolution infrared image to obtain a super-resolution infrared image.

2. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer as claimed in claim 1, characterized in that: The multi-input encoder part includes a first Bicubic unit to a third Bicubic unit, a first encoder EB1 to a third encoder EB3, a first SCM unit, and a second SCM unit; The first Bicubic unit is used to adjust the size of the low-resolution infrared image to the size required by the final output super-resolution image to obtain a feature map F0; The second Bicubic unit is used to downsample the feature map F0 by a factor of two to obtain a feature map F1; The third Bicubic unit is used to downsample the feature map F1 by a factor of two to obtain a feature map F2; The first encoder EB1 is used to extract features from the feature map F0 to obtain a first scale feature map; The first SCM unit is used to perform rough feature extraction on the feature map F1 to obtain a first rough extracted feature map; The second encoder EB2 is used to perform joint feature extraction on the first scale feature map and the first coarse extraction feature map to obtain a second scale feature map; The second SCM unit is used to perform rough feature extraction on the feature map F2 to obtain a second rough extracted feature map; The third encoder EB3 is used to perform joint feature extraction on the second scale feature map and the second coarse extraction feature map to obtain a third scale feature map.

3. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer as claimed in claim 2, characterized in that: The first encoder EB1 is composed of a convolution unit I and a local-global feature perception unit I in cascade sequence; the second encoder EB2 is composed of a convolution unit II, a FAM unit I and a local-global feature perception unit II in cascade sequence; the third encoder EB3 is composed of a convolution unit III, a FAM unit II and a local-global feature perception unit III in cascade sequence; The convolution unit I in the first encoder EB1 is used to extract the shallow features of the feature map F0 to obtain the initial shallow features F0'; the local-global feature perception unit I is used to extract the features of the initial shallow features F0' to obtain the first scale feature map; The convolution unit II in the second encoder EB2 is used to extract shallow features of the downsampled first scale feature map to obtain a first shallow feature map; the FAM unit I is used to perform feature fusion on the first shallow feature map and the first coarse extraction feature map output by the first SCM unit to obtain a first fused feature map; the local-global feature perception unit II is used to extract features of the first fused feature map to obtain a second scale feature map; The convolution unit III in the third encoder EB3 is used to extract the shallow features of the downsampled second-scale feature map to obtain a second shallow feature map; the FAM unit II is used to fuse the second shallow feature map with the second coarse extraction feature map output by the second SCM unit to obtain a second fused feature map; the local-global feature perception unit III is used to extract the features of the second fused feature map to obtain a third-scale feature map.

4. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer as claimed in claim 3, characterized in that: Any local-global feature perception unit is composed of a cascade of a local texture attention subunit LTAM and a global Transformer attention subunit GTAM; The local texture attention subunit LTAM is composed of a center difference convolution combination, four ordinary convolution combinations and a 1×1 convolution layer cascade in sequence; wherein the center difference convolution combination is composed of a center difference convolution layer CD Conv and an activation layer LReLU cascade, and the ordinary convolution combination is composed of a 3×3 ordinary convolution layer Conv and an activation layer LReLU cascade. The method for obtaining the local texture feature map output by the local texture attention subunit LTAM is as follows: F LTAM =Conv((Cat(F5,F4,F3,F2,F1,F in ))) 1×1 +F in F1=Lrelu(CDConv(F in ))+F in F m =F 3 (Cat(F in ,F1,...,F m-1 )),m=2,3,4,5 Among them, F LTAM is the local texture feature map finally output by the current local texture attention subunit LTAM, F in is the feature map of the central difference convolution layer CD Conv that inputs the current local texture attention subunit LTAM, F1~F5 are the outputs of the central difference convolution combination and the four ordinary convolution combinations respectively, Conv(·) 1×1 is the operation of the 1×1 convolution layer; CDConv(·) is the operation of the center difference convolution layer CD Conv, Lrelu is the activation function of the activation layer LReLU, Cat(·) represents the concatenation operation; m=1,2,3,4,5 are the numbers of the center difference convolution combination and the four ordinary convolution combinations respectively; F 3 (·)=Lrelu(Conv(·) 3×3 );Conv(·) 3×3 The operation of the 3×3 convolution layer; The global Transformer attention subunit GTAM is composed of three cascaded residual structures, wherein the first residual structure includes a LayerNorm layer and a moving window self-attention mechanism SW-MSA; the second residual structure includes a LayerNorm layer and a channel attention mechanism CA; the third residual structure includes a LayerNorm layer and an affine multilayer perceptron AffMLP, then the method for obtaining the scale feature map output by the global Transformer attention subunit GTAM is as follows: F res1 =SW-MAS(LN(F LTAM ))+F LTAM F res2 =CA(LN(F res1 ))+F res1 F GTAM =AffMLP(LN(F res2 ))+F res2 Among them, F res1 is the output feature map of the first residual structure, SW-MAS(·) is the operation of the moving window self-attention mechanism SW-MSA, LN(·) represents the operation of the LayerNorm layer, and F res2 is the output feature map of the second residual structure, CA(·) is the operation of the channel attention mechanism CA, AffMLP(·) is the operation of the affine multilayer perceptron AffMLP, F GTAM is the output feature map of the third residual structure, and F GTAM The scale feature map that is the final output of the current local-global feature perception unit.

5. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer as claimed in claim 2, characterized in that: The fusion part includes a first MFF unit, a second MFF unit and a third MFF unit; wherein any MFF unit is sequentially cascaded by a 1×1 convolution layer, a 3×3 convolution layer and a 3×3 depth-separable convolution layer, and any MFF unit obtains an aggregate feature map F MFF The method is as follows: in, is the output of the first encoder EB1, is the output of the second encoder EB2, is the output of the third encoder EB3, Conv(·) 1×1 The operation of the 1×1 convolutional layer; Conv(·) 3×3 The operation of the 3×3 convolution layer; DSConv(·) 3×3 is the operation of a 3×3 depthwise separable convolutional layer; Cat(·) represents a concatenation operation; R(·) represents a scaling operation using bicubic interpolation.

6. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer as claimed in claim 5, characterized in that: The decoder part includes a first decoder DB1, a second decoder DB2 and a third decoder DB3, wherein the first decoder DB1 is sequentially composed of a 3×3 convolution unit I and a local-global feature perception unit I in cascade; the second decoder DB2 is sequentially composed of a 3×3 convolution unit II and a local-global feature perception unit II in cascade; the third decoder DB3 is sequentially composed of a feature refinement unit FRM, a 3×3 convolution unit III and a local-global feature perception unit III in cascade; The feature refinement unit FRM of the third decoder DB3 is used to enhance the features of the aggregate feature map output by the third MFF unit to obtain an enhanced aggregate feature map; the convolution unit III is used to extract the shallow features of the enhanced aggregate feature map to obtain a third decoded shallow feature map; the local-global feature perception unit III is used to extract the local features of the third decoded shallow feature map to obtain a third decoded feature map; The convolution unit II of the second decoder DB2 is used to extract the shallow features of the upsampled third decoding feature map and the aggregate feature map output by the second MFF unit to obtain the second decoding shallow feature map; the local-global feature perception unit II is used to extract the local features of the second decoding shallow feature map to obtain the second decoding feature map; The convolution unit I of the first decoder DB1 is used to extract the shallow features of the upsampled second decoding feature map and the aggregated feature map output by the first MFF unit to obtain the first decoding shallow feature map; the local-global feature perception unit I is used to extract the local features of the first decoding shallow feature map to obtain the first decoding feature map; The first decoded feature map is used as the reconstructed feature map finally output by the decoder part, and the method for obtaining the reconstructed feature map finally output by the decoder part is as follows: Where k = 1, 2, 3 represents the decoder number, Cat(·) is the feature concatenation operation, Up(·) represents the upsampling operation, Conv(·) represents the decoder number, and Conv(·) represents the decoder number. 3×3 is the operation of a 3×3 convolutional unit; LGFPM(·) is the operation of a local-global feature perception unit; FRM(·) is the operation of a feature refinement unit FRM, F1”~F3” represent the inputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3, respectively, and F1”’~F3”’ represent the final outputs of the first decoder DB1, the second decoder DB2 and the third decoder DB3, respectively.

7. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 6, characterized in that: The feature refinement unit FRM includes two 1×1 convolutional layers, three 3×3 dilated convolutional layers with dilation coefficients of 4, 8, and 12, a global average pooling layer, and an attention mechanism AR; The aggregated feature map output by the third MFF unit is divided into five branches, which enter one 1×1 convolution layer, three 3×3 dilated convolution layers with dilation coefficients of 4, 8, and 12 respectively, and a global average pooling layer. The feature maps obtained by the five branches are superimposed to obtain a high-level feature map; another 1×1 convolution layer is used to perform feature aggregation on the high-level feature map to obtain a high-level aggregated feature map; the attention mechanism AR is used to perform feature enhancement on the high-level aggregated feature map to obtain a high-level enhanced feature map; The high-level aggregate feature map is element-wise multiplied with the high-level enhanced feature map to obtain the final enhanced aggregate feature map.

8. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer as claimed in claim 1, characterized in that: The jump connection part includes a 3×3 convolutional layer; After the 3×3 convolution layer extracts features from the reconstructed feature map, the extracted feature map is combined with the low-resolution infrared image to obtain a super-resolution infrared image.

9. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 1, characterized in that: The mixed loss function L used when training the infrared image super-resolution network model total as follows: L total =L pix +λ1L str +λ2L tex Among them, L pix is the pixel loss, L str is the structural loss, L tex is the texture loss, λ1 represents the structural loss L str The weight of λ2 represents the texture loss L tex The weight of .

10. The multi-input hierarchical infrared image super-resolution reconstruction method based on collaborative CNN and Transformer according to claim 9, characterized in that: L pix , L str , L tex The expressions are: Where N is the number of low-resolution infrared images in the training samples, is the pixel matrix of the predicted super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample output by the infrared image super-resolution network model, is the pixel matrix of the actual super-resolution infrared image corresponding to the i-th low-resolution infrared image in the training sample, ||·||1 is the L1 norm, represents fast Fourier transform, and SSIM(·) is the structural similarity.

Images