A self-attention infrared image super-resolution reconstruction method based on hybrid convolution

Abstract

The present invention discloses a self-attention infrared image super-resolution reconstruction method based on hybrid convolution, which performs pixel embedding on the initial infrared image to be reconstructed to obtain its shallow feature F0; uses a cascade of N Swin Transformer hybrid convolution groups and one convolution layer to perform deep feature extraction on the shallow feature F0 to obtain the deep feature F0. DF Each Swin Transformer hybrid convolution group consists of L cascaded Swin‑CNN fusion processing blocks and 1 convolution layer; based on the shallow feature F0 and the deep feature F DF , using the reconstruction function to perform super-resolution reconstruction of infrared images. This invention combines the advantages of Swin Transformer and CNN, improving the reconstruction performance of infrared image details and features by fusing global and local information. It shows significant advantages in accuracy and robustness, and ultimately achieves high-quality infrared image reconstruction.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a self-attention infrared image super-resolution reconstruction method based on hybrid convolution. Background Art

[0002] Infrared super-resolution technology, a key means of improving infrared image quality, is widely used in military reconnaissance, industrial inspection, medical imaging, and other fields. Limited by the physical resolution of sensor hardware, infrared images often suffer from low resolution and blurred details, which impact their performance in precision monitoring and recognition tasks. Super-resolution technology allows algorithms to enhance image detail and resolution while maintaining device costs and sensor specifications. Traditional super-resolution methods include interpolation-based algorithms and frequency-domain processing, but these methods perform poorly in complex scenes and detail recovery, and are unable to effectively extract high-frequency information and texture features from images. Summary of the Invention

[0003] In order to overcome the shortcomings of the above-mentioned prior art, the object of the present invention is to provide a self-attention infrared image super-resolution reconstruction method based on hybrid convolution to achieve high-quality infrared image reconstruction.

[0004] The main technical route of the present invention is: in order to address the problem that the Transformer architecture has insufficient ability to understand local information when processing infrared image super-resolution tasks, the present invention combines the local feature extraction capability of the convolutional neural network and the long-distance information capture capability of the Swin Transformer to achieve effective reconstruction of the global structure and local high-frequency details of the infrared image in an efficient manner. In addition, the output of the convolution branch is used as the channel activation information of the Transformer architecture to promote information transfer, thereby further enhancing the performance of the algorithm. At the same time, in view of the typical low contrast and gentle grayscale changes of infrared images, a channel enhancement splitting method is studied and implemented to reduce the number of model parameters, which can maintain efficient infrared super-resolution effects while simplifying the network structure.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A self-attention infrared image super-resolution reconstruction method based on hybrid convolution is characterized by comprising the following steps:

[0007] Step 1: embed pixels of the initial infrared image to be reconstructed to obtain its shallow feature F0;

[0008] Step 2: Use the cascaded N Swin Transformer mixed convolution groups and 1 convolution layer to extract deep features from the shallow feature F0 to obtain the deep feature F DF; Each of the Swin Transformer hybrid convolution groups consists of L cascaded Swin-CNN fusion processing blocks and 1 convolution layer;

[0009] Step 3, based on shallow feature F0 and deep feature F DF , use the reconstruction function to perform super-resolution reconstruction of infrared images.

[0010] In one embodiment, in step 1, a 3×3 convolution operation is used to extract shallow features F0 from the input low-resolution infrared image to be reconstructed, and its mathematical expression is:

[0011] F0=H SF (I LR )

[0012] in, Represents an infrared image with a height of H, a width of W, and a channel number of 1, H SF (·) represents the pixel embedding operation based on 3×3 convolution, Represents shallow features with C channels.

[0013] In one embodiment, among the N Swin Transformer mixed convolution groups, the output feature F of the i-th Swin Transformer mixed convolution group is i Expressed as:

[0014] F i =H STFCGi (F i-1 )=H COVN (H SCFPBL (H SCFPB(L-1) (...H SCFPBj (...H SCFPB1 (F i-1 )))))+F i-1

[0015] Among them, H STFCGi (·) represents the i-th Swin Transformer mixed convolution group, H COVN (·) represents the convolutional layer in the i-th Swin Transformer hybrid convolutional group, H SCFPBj (·) represents the jth Swin-CNN fusion processing block in the i-th Swin Transformer hybrid convolution group, F i-1 Represents the output of the i-1th Swin Transformer mixed convolution group.

[0016] In one embodiment, the Swin-CNN fusion processing block performs the following process:

[0017] The input feature map with height H, width W and number of channels C After normalization, the number of channels is expanded to 2αC through 1 × 1 convolution, where α is the parameter reduction factor and Yα<1;

[0018] The feature map with expanded channel numbers is divided into two parts equally according to the channels. One part is input into the multi-head self-attention branch using the shift window technology to process the self-attention, and the other part is input into the convolution branch to enhance the extraction of local feature information. The output features of the two branches are fused, and then the channels are restored and combined with the input feature map F. T Perform residual connection and finally pass through a feedforward neural network to obtain the final output.

[0019] In one embodiment, the multi-head self-attention branch using the shift window technique divides the input image block into multiple windows of equal size, calculates self-attention in each window, and shifts the window when calculating the self-attention of the next layer, and calculates self-attention in each shifted window. The final multi-head self-attention fusion result Z is expressed as:

[0020] Z=MultiHead(Q,K,V)=Concat(h1,h2,...,h n )·W O

[0021] Among them, Z is the multi-head self-attention fusion result, Q, K and V are the query, key and value when calculating attention respectively, h1, h2, ..., h n is the result of each head of multi-head self-attention, Concat(·) represents channel splicing, W O Represents the fusion weight matrix.

[0022] In one embodiment, the convolution branch performs two channel compression convolutions and one activation function operation on its input, wherein the first convolution is used to reduce the number of channels and the second convolution is used to restore the number of channels. The operation of the convolution branch is expressed as:

[0023] F C1 =Conv3(F C )

[0024] F C2 =Conv3(Act(F C1 ))

[0025] Among them, Conv3(·) represents the 3×3 convolution operation, Act(·) is the activation function, and F C Represents the feature map of the input convolution branch, F C1 is the feature map after the first convolution, with a size of H×W×αC / 3, F C2It is the feature map after the second convolution, with a size of H× W×αC.

[0026] In one embodiment, the output features of the two branches are fused as follows:

[0027] The output of the convolution branch is passed through the CT module to provide channel activation information, and the output of the CT module is multiplied by the multi-head self-attention branch pixel using the shift window technique, and then added to the output pixel of the convolution branch.

[0028] In one embodiment, the CT module performs the following operations:

[0029] The output feature map size of the convolution branch is compressed to 1×1 through global average pooling;

[0030] Use 1×1 convolution to reduce the number of channels to reduce the ratio of 1 / r, where r is the magnification of the reconstructed image;

[0031] The reduced feature map is passed through the ReLU activation function, and then 1×1 convolution is used again to restore the number of channels;

[0032] The feature map of the restored channel number is adjusted to between 0 and 1 through the Sigmoid function and used as the output of the CT module.

[0033] In one embodiment, step 3, using the reconstruction function to perform super-resolution reconstruction of the infrared image, is implemented through two convolutional layers, one activation function and one upsampling block, and its expression is:

[0034] I HR =H REC (F0+F DF )

[0035] Among them, H REC (·) is the reconstruction function, which combines the shallow feature F0 with the deep feature F DF The summed result is used as input to reconstruct a high-resolution infrared image with a magnification of r. The output image resolution is rH×rW.

[0036] In one embodiment, the upsampling block is composed of 3×3 convolutions and sub-pixel convolutions stacked according to a magnification factor. When the magnification factor r is a power of 2, the number of stacks is log2 r.

[0037] Compared with the existing technology, the self-attention infrared image super-resolution reconstruction method based on hybrid convolution in the present invention integrates the advantages of Swin Transformer and CNN. It improves the reconstruction performance of infrared image details and features from the level of fusing global and local information, shows significant advantages in accuracy and robustness, and ultimately achieves high-quality infrared image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the overall structure of the HCSAN in the present invention.

[0039] Figure 2 This is a comparison chart of the Swin Transformer and the improved SCFPB in the present invention.

[0040] Figure 3 This is a schematic diagram of the CT module structure in the present invention.

[0041] Figure 4 Schematic diagram of the shift window in the present invention.

[0042] Figure 5 Schematic diagram of the upsampling block in the present invention.

[0043] Figure 6 Schematic diagram of the parameter reduction strategy flow in the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] With the rapid development of deep learning, super-resolution methods based on convolutional neural networks (CNNs) have made significant progress. Among them, the hybrid convolution structure combines the advantages of standard convolution and dilated convolution. Through multi-scale feature extraction, it can more comprehensively capture local and global information in the image, improving the image reconstruction effect. At the same time, the introduction of the self-attention mechanism further enhances the feature modeling capability of long-distance dependencies in infrared images. By assigning different weights to different areas in the image, the self-attention mechanism can be more flexible and accurate in recovering details, improving the super-resolution effect. Therefore, the infrared super-resolution algorithm based on hybrid convolution and self-attention mechanism has shown significant advantages in accuracy and robustness.

[0046] The present invention designs and implements an infrared image super-resolution reconstruction method based on a hybrid convolutional self-attention model (HCSAN) from the perspective of fusing global and local information, so as to improve the overall performance of infrared image super-resolution reconstruction. The self-attention mechanism is better than traditional convolution in capturing long-distance information, but it is insufficient in extracting local information. To address this problem, the Swin Transformer is combined with CNN to efficiently fuse global and local information, thereby improving the overall performance of infrared image super-resolution reconstruction. In order to better combine the information provided by the convolution and self-attention branches, the results of the convolution branch are used as channel activation information of the self-attention branch. At the same time, in view of the characteristics of flat grayscale changes and low contrast of infrared images, a parameter optimization strategy of channel enhancement splitting is proposed to reduce network redundancy and improve model calculation efficiency.

[0047] refer to Figure 1 As shown, the overall structure of the HCSAN of the present invention is divided into a pixel embedding part, a deep feature extraction part and a high-resolution reconstruction part according to the functional order.

[0048] Among them, the pixel embedding part performs pixel embedding on the initial infrared image to be reconstructed to obtain its shallow feature F0.

[0049] Specifically, this part uses a 3×3 convolution operation to extract the shallow feature F0 from the input low-resolution infrared image to be reconstructed, and its mathematical expression is:

[0050] F0=H SF (I LR )

[0051] in, Represents an infrared image with a height of H, a width of W, and a channel number of 1, H SF (·) represents the pixel embedding operation based on 3×3 convolution, Represents shallow features with C channels.

[0052] The deep feature extraction part is the main part of the present invention, which mainly includes N cascaded Swin Transformer Fused CNN Groups (STFCG) and 1 convolution layer to further extract and process the shallow feature F0 to obtain the deep feature F DF , its mathematical expression is:

[0053] F DF =H DF (F0)

[0054] Among them, F DF Represents the extracted deep features. DF (·) represents the deep feature extraction part, and the output of the internal cascaded STFCG module is expressed as

[0055] F i =H STFCGi (F0) i=1,2,...,N

[0056] Among them, H STFCGi (·) represents the i-th STFCG module.

[0057] The HCSAN of this invention is primarily based on the SwinIR algorithm, the essence of which lies in the stacking of multiple Swin-CNN Fusion Processing Blocks (SCFPBs). Specifically, each SwinTransformer hybrid convolutional group (STFCG module) consists of L cascaded Swin-CNN Fusion Processing Blocks (SCFPB modules) and one convolutional layer.

[0058] SCFPB module structure is as follows Figure 1 As shown in the upper part of the figure, drawing on the ideas of RCAN and SwinIR, the STFCG designed in this paper has residual connections, followed by a stack of L SCFPBs and a convolutional layer at the end. The residual connections of the STFCG facilitate the efficient flow of feature information between different STFCGs, while the convolutional layer at the end increases the stability of the network's translation and variability. The input and output of the i-th STFCG module can be further described as:

[0059] F i =H COVN H SCFPBL (H SCFPB(L-1) (...H SCFPBj (...H SCFPB1 (F i-1 )))))+F i-1 j=0, 1, ..., L

[0060] Among them, H COVN (·) represents the convolutional layer in the i-th STFCG, H SCFPBj (·) represents the jth SCFPB in the i-th STFCG, F i-1 Represents the output of the previous, i.e., the i-1th STFCG.

[0061] SCFPB is the core improvement of this invention. It combines the advantages of the Swin Transformer and integrates convolution operations to enhance the network's ability to capture local information. In addition, considering the gentle grayscale changes of infrared images, the number of parameters added after convolution is optimized to reduce unnecessary redundancy and improve processing efficiency. Figure 2 (a) and (b) show the original structure of Swin Transformer and the improved SCFPB respectively.

[0062] The original Swin Transformer module consists of two main parts: one is the multi-head self-attention (Shifted Window Multi-head Self Attention, SW-MSA) calculation using shifted window technology, and the other is an FNN including two linear mapping layers and GELU activation function. Each part is set with layer normalization and residual connection to support deeper network training. In contrast, the SCFPB designed in the present invention adds a parallel, channel-reducing convolution branch to the SW-MSA module. Before branch feature extraction, SCFPB uses 1×1 convolution and channel separation operations to reduce the number of channels, which will reduce the computational burden in the parallel process, and restore the channel scale at the end of the process through a 1×1 convolution layer. At the same time, due to the addition of 1×1 convolution, the linear mapping performed in SW-MSA to mix the results of multi-head self-attention calculations is omitted. Overall, the SCFPB designed in the present invention conforms to the general macro-architecture of the visual transformer, and its execution process is as follows:

[0063] Input features After normalization, the number of channels is first expanded to 2αC (α<1) through 1×1 convolution. Then, the feature map with the expanded channel number is evenly divided into two parts according to the channel. One part enters the SW-MSA branch to process self-attention, and the other enters the convolution branch to enhance the extraction of local feature information. After the output features of the convolution branch and the SW-MSA branch are fused, the channel is restored and combined with the input feature map F T A residual connection is performed, and finally a feed-forward neural network (FFN) is used to obtain the final output. In the embodiment of the present invention, the feed-forward neural network (FFN) is composed of two linear mappings, one GELU activation, and one layer normalization.

[0064] The parallel convolution branch used for channel compression in this invention performs two channel compression convolutions and one activation function operation on its input, aiming to optimize the number of parameters. The first convolution reduces the number of channels in the feature map, while the second convolution restores the number of channels. The mathematical expression of this convolution branch is:

[0065] FC1 =Conv3(F C )

[0066] F C2 =Conv3(Act(F C1 ))

[0067] Among them, Conv3(·) represents the 3×3 convolution operation, Act(·) is the activation function, and F C Represents the feature map of the input convolution branch, F C1 is the feature map after the first convolution, F C2 is the feature map after the second convolution. In these two convolution steps, the first convolution compresses the feature channel to reduce the calculation and parameter amount of the network. At this time, F C1 The size is H×W×αC / 3. The second convolution is to restore the number of channels of the feature map, and the output F C2 The size is H×W×αC. The floating-point multiplication calculation and parameter calculation formula of convolution are described as:

[0068] FLOPs = C i ×k 2 ×C o ×H×W

[0069] params=C i ×(k 2 ×C o +1)

[0070] Among them, C i is the number of channels of the input feature map, C o is the number of channels of the output feature map, H and W represent the height and width of the feature map respectively, and k is the size of the convolution kernel. According to the above formula, when the channel compression strategy is not adopted, if a 3×3 convolution is used, the total floating-point multiplication calculation amount and parameter amount are 18α 2 C 2 HW and 18α 2 C 2 , and after adopting the channel compression strategy, it is 6α 2 C 2 HW and 3α 2 C 2 Obviously, the channel reduction strategy reduces the amount of computation and parameters to only one third of the original amount.

[0071] To better combine the information provided by the convolution and SW-MSA branches, the SCFPB module fuses the convolution branch results before the SW-MSA branch to provide channel activation information. Specifically, the output of the convolution branch is passed through the CT module to provide channel activation information. The output of the CT module is multiplied with the pixels of the multi-head self-attention branch using the shifted window technique, and then added to the output pixels of the convolution branch.

[0072] The detailed structure of the CT module is as follows Figure 3 As shown in the figure, the CT module performs the following operations: First, the output feature map of the convolution branch is compressed to a 1×1 size through global average pooling. Then, a 1×1 convolution is used to reduce the number of channels to reduce the 1 / r ratio. Next, a ReLU activation function is applied to the convolution result, and a 1×1 convolution is used to restore the number of channels. Finally, the result is adjusted to between 0 and 1 through a sigmoid function and used as the output of the CT module. The output of the CT module is combined with the SW-MSA branch to activate more critical channels, thereby improving the information flow and expression capabilities of the entire network.

[0073] The SW-MSA branch uses the shifted window multi-head self-attention mechanism unique to the Swin Transformer architecture to deeply explore the internal correlations of the image. The SCFPB module uses a 1×1 convolution operation to restore the number of channels, thereby optimizing the calculation of SW-MSA and eliminating the linear mapping step commonly used to integrate the results of multi-head self-attention. In essence, SW-MSA still analyzes the internal connections of the image based on the self-attention mechanism, but compared with the traditional self-attention mechanism, the Swin Transformer reduces the overall amount of computation by performing self-attention calculations within a local window. The single-head self-attention calculation formula contains two matrix multiplications, and its mathematical expression is:

[0074]

[0075] Ignore the Softmax function and divide by the distance d k The amount of calculation, for an image block of dimension h×w×C, the amount of attention calculation in the above formula is:

[0076] N calculate =2(hw) 2 C

[0077] As can be seen from the above formula, the computational cost of self-attention is directly related to the square of the product of the width and height of the image block size. This means that once the image block size increases, the computational cost will increase dramatically. To effectively reduce the computational cost, the PVT model adopts a space-efficient self-attention strategy, reducing the dimensions of the key (K) and value (V) channels to reduce the computational cost of matrix multiplication. This method slows down the increase in computational cost to a certain extent, but at the same time, it inevitably weakens the network's feature representation ability. SW-MSA balances computational efficiency and feature capture by restricting the calculation of self-attention to a local window range. Specifically, SW-MSA divides the input h×w×C image block into M×M windows, resulting in a total of h / M×w / M windows. Self-attention is then calculated in each window, greatly reducing the computational cost of self-attention for large-scale image blocks. The computational cost is now reduced to:

[0078] N′ calculate =2M 2 HkDJ

[0079] However, the self-attention calculation within the local window may cause information transmission to be blocked, and the ability to capture long-range information between different windows is weakened. To solve this problem, SW-MSA shifts the split window when calculating the self-attention in the next layer. The principle is as follows: Figure 4 Show.

[0080] The shifted window design takes an 8×8 image block as input and divides it into 4×4 windows. Self-attention is then performed within these windows. When calculating self-attention in the subsequent layer, the window configured in the previous layer is shifted rightward and downward by two (M / 2) image blocks, resulting in nine windows. Computing self-attention within these nine windows allows information within the four windows in the previous layer to be communicated.

[0081] When calculating multi-head self-attention, the original SW-MSA in Swin Transformer uses linear mapping to fuse the multi-head self-attention results, which is expressed as

[0082] Z=MultiHead(Q,K,V)=Concat(h1,h2,...,h n )·W O

[0083] Among them, Z is the multi-head self-attention fusion result, Q, K and V are the query (Query), key (Key) and value (Value) when calculating attention, h1, h2, ..., h n is the result of each head of multi-head self-attention, Concat(·) represents channel splicing, W Orepresents the fused weight matrix. After adding the results of the convolution branch and the SW-MSA branch, the SCFPB module performs a 1×1 convolution operation, similar to the above formula. Therefore, the SW-MSA in the SCFPB module simplifies the linear mapping step used to fuse the results of multi-head self-attention calculations.

[0084] The high-resolution reconstruction part of the present invention is based on the shallow feature F0 and the deep feature F DF , use the reconstruction function to perform super-resolution reconstruction of infrared images.

[0085] Specifically, the high-resolution reconstruction part of the present invention is implemented by two convolutional layers, one activation function and an upsampling block, and its expression is:

[0086] I HR =H REC (F0+F DF )

[0087] Among them, H REC (·) is the reconstruction function, which represents the entire high-resolution reconstruction stage. It combines the shallow feature F0 with the extracted deep feature F DF The summed result is used as input to reconstruct a high-resolution infrared image with a magnification of r. The output image resolution is rH×rW. In this structure, the "3×3 convolution and sub-pixel convolution" group is stacked according to the magnification, which is the upsampling block. When the magnification r is a power of 2 such as 2, 4 or 8, the number of stacks is log2r. The upsampling block is the core of the high-resolution reconstruction stage, and its structure is as follows Figure 5 shown.

[0088] The parameter reduction strategy of channel enhancement and splitting in the present invention can be further described as follows: in view of the characteristic of smooth grayscale change of infrared images compared with visible light images, a parameter reduction strategy of channel enhancement and splitting is proposed in the SCFPB module to reduce information redundancy and improve computational efficiency. Its structure is as follows: Figure 2 This parameter reduction strategy involves performing channel enhancement by 1×1 convolution before the convolution branch and the SW-MSA branch, then splitting equally according to the number of channels and applying 1×1 convolution in the subsequent processing of the two branches. This strategy can be summarized as Figure 6 Through this strategy, the number of channels of the feature map processed by the convolution branch and the SW-MSA branch is reduced to α times the original number, where α is a real number less than 1.

[0089] To achieve downscaling, for an input feature map of size H×W×C, the number of channels is first increased to 2αC using a 1×1 convolution. The map is then split equally into two parts, each of size H×W×αC. The two split feature maps serve as the input to the convolution branch and the SW-MSA branch, respectively. To restore the number of channels in the feature map, a 1×1 convolution is performed after the features of the convolution branch and the SW-MSA branch are fused to increase the number of channels to C.

[0090] According to the calculation formula of convolution floating-point multiplication and self-attention floating-point multiplication, the total floating-point multiplication calculation amount of the convolution branch and SW-MSA branch without and with channel reduction strategy is respectively

[0091]

[0092]

[0093] In order to verify the effect of the present invention, the following specific embodiment is provided, which mainly includes:

[0094] Step 1: Training Images

[0095] A dataset was constructed by mounting infrared cameras of three different resolutions on a panel and placing them in a vehicle to capture images of various outdoor scenes. The dataset contains infrared images at three resolutions: 160×120, 320×240, and 640×480. The 640×480 images are obtained by cropping images from high-resolution cameras. Each resolution contains 951 training images and 50 test images. The images in the dataset include trees, people, buildings, vehicles, and other common outdoor scenes.

[0096] Although the dataset contains three infrared images of different resolutions, with size ratios of ×2 and ×4, respectively, these images are not fully aligned and therefore cannot be directly used for high- and low-resolution image comparison in training the network. During the training of the HCSAN algorithm and other comparison algorithms, the present invention uses a 640×480 resolution image as the true high-resolution infrared image. The low-resolution image input to the network is degraded according to the image degradation model described below. The degradation model uses a Gaussian blur kernel with a standard deviation of 1, bicubic interpolation downsampling, and additive Gaussian noise.

[0097] I LR =D w B l I HR +n o

[0098] During the training phase, image patches of 48 pixels in width and height were cropped from the input low-resolution images as training input. The corresponding sizes of the infrared high-resolution image patches after ×2, ×4, and ×8 super-resolution were 96, 192, and 384 pixels, respectively. To improve the network's generalization ability, a data augmentation strategy was used during training, randomly flipping the images horizontally or rotating them by 90°, 180°, and 270°.

[0099] (2) Test image

[0100] Three different sets of test images were selected for comprehensive performance evaluation. The first set consisted of 50 test images from a dataset in the literature; the second set consisted of 20 images from the FLIR infrared dataset, a publicly available library of images widely used for infrared imaging and object detection, encompassing a variety of infrared images, including pedestrians, animals, vehicles, and buildings; and the third set consisted of 20 images of campus and road scenes captured in this paper using a DJI drone equipped with an infrared camera.

[0101] (3) Network training

[0102] The HCSAN algorithm adopts a structural design consistent with SwinIR, where the number of stacked STFCG bits N = 6, the number of stacked SCFPBs in STFCG L = 6, and the number of shallow feature channels extracted by pixel embedding C = 180.

[0103] The experimental platform was equipped with an Intel i5-8500 CPU, an NVIDIA TITAN V GPU, and 16GB of RAM. The software environment was built on the Windows 10 Pro operating system, using the PyTorch 1.7.1 deep learning framework and the Python 3.7 programming language. At the training level, the L1 loss function was used, and the ADAM optimizer was selected with parameters of β1 = 0.9 and β2 = 0.999, and ε was set to 1×10 -8 At the same time, set the batch size to 8 and set the -4 is the initial learning rate, and the learning rate is halved after every 20,000 iterations.

[0104] The HCSAN algorithm was compared with several other mainstream algorithms on the three groups of test images. The comparison algorithms include the traditional super-resolution algorithm Bicubic, the deep learning-based algorithms EDSR and RCAN, and the SwinIR algorithm. Each algorithm was trained and tested separately for magnifications of ×2, ×4, and ×8. In order to objectively evaluate the performance of each algorithm, Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) were selected as evaluation indicators. Table 1 records the performance indicators of each algorithm at different magnifications, where the best results in the same group of test images are indicated in bold and the suboptimal results are underlined.

[0105] Table 1 Performance indicators of each algorithm at different magnifications.

[0106]

[0107] The experimental results demonstrate a clear advantage for the Transformer architecture at a 2x upscaling ratio. The proposed HCSAN and SwinIR algorithms significantly outperform the convolutional neural network-based EDSR and RCAN algorithms in all three test image groups. A comparison of our algorithm with the SwinIR algorithm reveals that the proposed method achieves the best PSNR objective evaluation metric across all test groups, while its SSIM performance lags slightly behind that of the SwinIR algorithm in the second test image group. Overall, at a 2x super-resolution upscaling ratio, our proposed method achieves optimal average PSNR and SSIM performance, with PSNR improving by 0.1dB and SSIM by 0.002 compared to the SwinIR algorithm. The Transformer architecture also maintains a clear advantage at a 4x super-resolution upscaling ratio. While HCSAN's PSNR is slightly lower than that of the SwinIR algorithm by 0.03dB on the second test image group, it maintains its lead across the remaining test image groups. HCSAN also comprehensively outperforms the other compared algorithms in terms of SSIM, achieving the highest ranking in both average PSNR and SSIM. When the magnification reaches 8x, the performance advantage between the algorithms becomes less obvious due to the insufficient feature information in the input low-resolution image. However, HCSAN still maintains the best overall performance, followed by the SwinIR algorithm.

[0108] Based on the above analysis, the HCSAN algorithm proposed in this paper shows superior reconstruction performance compared with other algorithms in objective evaluation experiments at various magnification levels, verifying the effectiveness and potential of the algorithm.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A self-attention infrared image super-resolution reconstruction method based on hybrid convolution, characterized in that: The steps include: Step 1: embed pixels of the initial infrared image to be reconstructed to obtain its shallow feature F0; Step 2: Use the cascaded N Swin Transformer mixed convolution groups and 1 convolution layer to extract deep features from the shallow feature F0 to obtain the deep feature F DF Each of the Swin Transformer hybrid convolution groups consists of L cascaded Swin-CNN fusion processing blocks and one convolution layer; the Swin-CNN fusion processing block performs the following process: The input feature map with height H, width W and number of channels C After normalization, the number of channels is expanded to 2αC through 1×1 convolution, where α is the parameter reduction factor, α<1; The feature map with expanded channel numbers is divided into two parts equally according to the channels. One part is input into the multi-head self-attention branch using the shift window technology to process the self-attention, and the other part is input into the convolution branch to enhance the extraction of local feature information. The output features of the two branches are fused, and then the channels are restored and combined with the input feature map F. T Perform residual connection and finally get the final output through a feedforward neural network; The multi-head self-attention branch using the shift window technology divides the input image block into multiple windows of equal size, calculates self-attention in each window, and shifts the window when calculating the self-attention of the next layer, and calculates self-attention in each shifted window. The final multi-head self-attention fusion result Z is expressed as: Z=MultiHead(Q,K,V)=Concat(h1,h2,...,h n )·W O Among them, Z is the result of multi-head self-attention fusion, Q, K and V are the query, key and value when calculating attention respectively, h1,h2,...,h n is the result of each head of multi-head self-attention, Concat(·) represents channel splicing, W O Represents the fusion weight matrix; The convolution branch performs two channel compression convolutions and one activation function operation on its input, where the first convolution is used to reduce the number of channels and the second convolution is used to restore the number of channels. The operation of the convolution branch is expressed as: F C1 =Conv3(F C ) F C2 =Conv3(Act(F C1 )) Among them, Conv3(·) represents the 3×3 convolution operation, Act(·) is the activation function, and F C Represents the feature map of the input convolution branch, F C1 is the feature map after the first convolution, with a size of H×W×αC / 3, F C2 It is the feature map after the second convolution, with a size of H×W×αC; Step 3, based on shallow feature F0 and deep feature F DF , use the reconstruction function to perform super-resolution reconstruction of infrared images.

2. The method for super-resolution infrared image reconstruction based on self-attention and hybrid convolution according to claim 1, characterized in that: In step 1, a 3×3 convolution operation is used to extract the shallow feature F0 from the input low-resolution infrared image to be reconstructed, and its mathematical expression is F0=H SF (I LR ) in, Represents an infrared image with a height of H, a width of W, and a channel number of 1, H SF (·) represents the pixel embedding operation based on 3×3 convolution, Represents shallow features with C channels.

3. The method for super-resolution infrared image reconstruction based on self-attention and hybrid convolution according to claim 1, characterized in that: Among the N Swin Transformer mixed convolution groups, the output feature F of the i-th Swin Transformer mixed convolution group is i Expressed as: F i =H STFCGi (F i-1 )=H COVN (H SCFPBL (H SCFPB(L-1) (...H SCFPBj (...H SCFPB1 (F i-1 )))))+F i-1 Among them, H STFCGi (·) represents the i-th Swin Transformer mixed convolution group, H COVN (·) represents the convolutional layer in the i-th SwinTransformer hybrid convolutional group, H SCFPBj (·) represents the jth Swin-CNN fusion processing block in the i-th Swin Transformer hybrid convolution group, F i-1 Represents the output of the i-1th Swin Transformer mixed convolution group.

4. The method for super-resolution infrared image reconstruction based on self-attention and hybrid convolution according to claim 1, characterized in that: The output features of the two branches are fused as follows: The output of the convolution branch is passed through the CT module to provide channel activation information, and the output of the CT module is multiplied by the multi-head self-attention branch pixel using the shift window technique, and then added to the output pixel of the convolution branch.

5. The method for super-resolution infrared image reconstruction based on self-attention and hybrid convolution according to claim 4, characterized in that: The CT module performs the following operations: The output feature map size of the convolution branch is compressed to 1×1 through global average pooling; Use 1×1 convolution to reduce the number of channels to reduce the ratio of 1 / r, where r is the magnification of the reconstructed image; The reduced feature map is passed through the ReLU activation function, and then 1×1 convolution is used again to restore the number of channels; The feature map of the restored channel number is adjusted to between 0 and 1 through the Sigmoid function and used as the output of the CT module.

6. The method for super-resolution infrared image reconstruction based on self-attention and hybrid convolution according to claim 1, characterized in that: In step 3, the infrared image super-resolution reconstruction is performed using the reconstruction function, which is implemented through two convolutional layers, one activation function and one upsampling block. Its expression is: HR =H REC (F0+F DF ) Among them, H REC (·) is the reconstruction function, which combines the shallow feature F0 with the deep feature F DF The summed result is used as input to reconstruct a high-resolution infrared image with a magnification of r. The output image resolution is rH×rW.

7. The method for super-resolution infrared image reconstruction based on self-attention and hybrid convolution according to claim 6, characterized in that: The upsampling block is composed of 3×3 convolutions and sub-pixel convolutions stacked according to the magnification. When the magnification r is a power of 2, the number of stacks is log2r.

Images