Self-attention infrared image super-resolution reconstruction method based on hybrid convolution

Abstract

The invention discloses a self-attention infrared image super-resolution reconstruction method based on hybrid convolution, and the method comprises the steps: carrying out the pixel embedding of an initial to-be-reconstructed infrared image, and obtaining a shallow feature F0 of the to-be-reconstructed infrared image; n cascaded Swin Transform hybrid convolution groups and one convolution layer are used to carry out deep feature extraction on the shallow feature F0, and a deep feature FDF is obtained; wherein each Swin Transform hybrid convolution group is composed of L Swin-CNN (Convolutional Neural Network) fusion processing blocks which are in cascade connection and one convolution layer; and based on the shallow feature F0 and the deep feature FDF, performing infrared image super-resolution reconstruction by using a reconstruction function. According to the method, the advantages of the Swin Transformer and the CNN are fused, the reconstruction performance of details and features of the infrared image is improved from the level of fusion processing of global and local information, remarkable advantages are shown in the aspects of accuracy and robustness, and finally high-quality infrared image reconstruction is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a self-attention infrared image super-resolution reconstruction method based on hybrid convolution. Background Art

[0002] As a key means to improve the quality of infrared images, infrared super-resolution technology is widely used in military reconnaissance, industrial inspection, medical imaging and other fields. Due to the limitation of the physical resolution of sensor hardware, infrared images often have problems such as low resolution and blurred details, which affect their performance in precision monitoring and recognition tasks. Through super-resolution technology, algorithms can be used to enhance image details and resolution while keeping the equipment cost and sensor specifications unchanged. Traditional super-resolution methods include interpolation-based algorithms and frequency-domain-based processing, but these methods are insufficient in dealing with complex scenes and detail recovery, and cannot effectively extract high-frequency information and texture features in images. Summary of the invention

[0003] In order to overcome the above-mentioned shortcomings of the prior art, the object of the present invention is to provide a self-attention infrared image super-resolution reconstruction method based on hybrid convolution to achieve high-quality infrared image reconstruction.

[0004] The main technical route of the present invention is: in view of the problem that the Transformer architecture has insufficient understanding of local information when processing infrared image super-resolution tasks, the present invention integrates the local feature extraction capability of the convolutional neural network and the long-distance information capture capability of the Swin Transformer to achieve effective reconstruction of the global structure and local high-frequency details of the infrared image in an efficient manner. In addition, the output of the convolution branch is used as the channel activation information of the Transformer architecture to promote information transfer, thereby further enhancing the performance of the algorithm. At the same time, in view of the typical low contrast and gentle grayscale change characteristics of infrared images, a channel enhancement splitting method is studied and implemented to reduce the number of model parameters, which can maintain efficient infrared super-resolution effects while simplifying the network structure.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A self-attention infrared image super-resolution reconstruction method based on hybrid convolution, characterized by comprising the following steps:

[0007] Step 1: embed pixels of the initial infrared image to be reconstructed to obtain its shallow feature F0;

[0008] Step 2: Use the cascaded N Swin Transformer mixed convolution groups and 1 convolution layer to extract deep features from the shallow feature F0 to obtain the deep feature F DF; Each of the Swin Transformer hybrid convolution groups consists of L cascaded Swin-CNN fusion processing blocks and 1 convolution layer;

[0009] Step 3: Based on the shallow feature F0 and the deep feature F DF , infrared image super-resolution reconstruction is performed using the reconstruction function.

[0010] In one embodiment, in step 1, a 3×3 convolution operation is used to extract shallow features F0 from the input low-resolution infrared image to be reconstructed, and its mathematical expression is:

[0011] F0=H SF (I LR )

[0012] in, Represents an infrared image with a height of H, a width of W, and a channel number of 1. SF (·) represents the pixel embedding operation based on 3×3 convolution, Represents shallow features with C channels.

[0013] In one embodiment, among the N Swin Transformer mixed convolution groups, the output feature F of the i-th Swin Transformer mixed convolution group is i It is expressed as:

[0014] F i =H STFCGi (F i-1 )=H COVN (H SCFPBL (H SCFPB(L-1) (...H SCFPBj (...H SCFPB1 (F i-1 ))))))+F i-1

[0015] Among them, H STFCGi (·) represents the i-th Swin Transformer mixed convolution group, H COVN (·) represents the convolutional layer in the i-th Swin Transformer mixed convolutional group, H SCFPBj (·) represents the jth Swin-CNN fusion processing block in the i-th Swin Transformer hybrid convolution group, F i-1 Represents the output of the i-1th Swin Transformer mixed convolution group.

[0016] In one embodiment, the Swin-CNN fusion processing block performs the following process:

[0017] The input feature map with height H, width W and number of channels C After normalization, the number of channels is expanded to 2αC through 1 × 1 convolution, where α is the parameter reduction factor, Yα<1;

[0018] The feature map with expanded channel numbers is divided into two parts equally according to the channels. One part is input into the multi-head self-attention branch using the shift window technology to process self-attention, and the other part is input into the convolution branch to enhance the extraction of local feature information. The output features of the two branches are fused, and then the channels are restored and combined with the input feature map F. T A residual connection is performed, and finally a feedforward neural network is used to obtain the final output.

[0019] In one embodiment, the multi-head self-attention branch using the shift window technology divides the input image block into multiple windows of equal size, calculates self-attention in each window respectively, and shifts the window when calculating the self-attention of the next layer, and calculates self-attention in each shifted window. The final multi-head self-attention fusion result Z is expressed as:

[0020] Z=MultiHead(Q,K,V)=Concat(h1,h2,...,h n )·W O

[0021] Among them, Z is the result of multi-head self-attention fusion, Q, K and V are the query, key and value when calculating attention, h1, h2, ..., h n is the result of each head of multi-head self-attention, Concat(·) represents channel concatenation, W O Represents the fusion weight matrix.

[0022] In one embodiment, the convolution branch performs two channel compression convolutions and one activation function operation on its input, wherein the first convolution is used to reduce the number of channels, and the second convolution is used to restore the number of channels. The operation of the convolution branch is expressed as:

[0023] F C1 =Conv3(F C )

[0024] F C2 =Conv3(Act(F C1 ))

[0025] Among them, Conv3(·) represents a 3×3 convolution operation, Act(·) is the activation function, and F C Represents the feature map of the input convolution branch, F C1 is the feature map after the first convolution, with a size of H×W×αC / 3, F C2It is the feature map after the second convolution, with a size of H× W×αC.

[0026] In one embodiment, the output features of the two branches are fused, and the implementation method is as follows:

[0027] The output of the convolution branch is passed through the CT module to provide channel activation information, and the output of the CT module is multiplied by the multi-head self-attention branch pixel using the shift window technique, and then added to the output pixel of the convolution branch.

[0028] In one embodiment, the CT module performs the following operations:

[0029] The output feature map size of the convolution branch is compressed to 1×1 through global average pooling;

[0030] Use 1×1 convolution to reduce the number of channels to reduce the ratio of 1 / r, where r is the magnification of the reconstructed image;

[0031] The reduced feature map is passed through the ReLU activation function, and then 1×1 convolution is used again to restore the number of channels;

[0032] The feature map of the restored channel number is adjusted to between 0 and 1 through the Sigmoid function as the output of the CT module.

[0033] In one embodiment, the step 3, using the reconstruction function to perform infrared image super-resolution reconstruction, is implemented through two convolutional layers, one activation function and one upsampling block, and its expression is:

[0034] I HR =H REC (F0+F DF )

[0035] Among them, H REC (·) is the reconstruction function, which combines the shallow feature F0 with the deep feature F DF The result of the addition is used as input to reconstruct a high-resolution infrared image with a magnification of r. The output image resolution is rH×rW.

[0036] In one embodiment, the upsampling block is composed of 3×3 convolutions and sub-pixel convolutions stacked according to a magnification factor, and when the magnification factor r is a power of 2, the number of stacks is log2 r.

[0037] Compared with the prior art, the self-attention infrared image super-resolution reconstruction method based on hybrid convolution in the present invention integrates the advantages of Swin Transformer and CNN, improves the reconstruction performance of infrared image details and features from the level of fusing global and local information, shows significant advantages in accuracy and robustness, and ultimately achieves high-quality infrared image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall structure of the HCSAN in the present invention.

[0039] Figure 2 It is a comparison chart between the Swin Transformer in the present invention and the improved SCFPB.

[0040] Figure 3 It is a schematic diagram of the CT module structure in the present invention.

[0041] Figure 4 It is a schematic diagram of the shift window in the present invention.

[0042] Figure 5 Schematic diagram of the upsampling block in the present invention.

[0043] Figure 6 Schematic diagram of the parameter reduction strategy flow in the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] With the rapid development of deep learning, super-resolution methods based on convolutional neural networks (CNNs) have made significant progress. Among them, the hybrid convolution structure combines the advantages of standard convolution and dilated convolution. Through multi-scale feature extraction, it can more comprehensively capture local and global information in the image, improving the image reconstruction effect. At the same time, the introduction of the self-attention mechanism further enhances the feature modeling ability of long-distance dependencies in infrared images. By assigning different weights to different areas in the image, the self-attention mechanism can be more flexible and accurate in restoring details, improving the super-resolution effect. Therefore, the infrared super-resolution algorithm based on hybrid convolution and self-attention mechanism shows significant advantages in accuracy and robustness.

[0046] The present invention designs and implements an infrared image super-resolution reconstruction method based on a hybrid convolutional self-attention model (Hybrid Convolutional Self-Attention Network, HCSAN) from the perspective of fusing global and local information, so as to improve the overall performance of infrared image super-resolution reconstruction. The self-attention mechanism is superior to traditional convolution in capturing long-distance information, but it is insufficient in extracting local information. To address this problem, Swin Transformer is combined with CNN to efficiently fuse global and local information, thereby improving the overall performance of infrared image super-resolution reconstruction. In order to better combine the information provided by the convolution and self-attention branches, the results of the convolution branch are used as the channel activation information of the self-attention branch. At the same time, in view of the characteristics of flat grayscale changes and low contrast of infrared images, a parameter optimization strategy for channel enhancement splitting is proposed to reduce network redundancy and improve model calculation efficiency.

[0047] refer to Figure 1 As shown, the overall structure of the HCSAN of the present invention is divided into a pixel embedding part, a deep feature extraction part and a high-resolution reconstruction part according to the functional order.

[0048] Among them, the pixel embedding part performs pixel embedding on the initial infrared image to be reconstructed to obtain its shallow feature F0.

[0049] Specifically, this part uses a 3×3 convolution operation to extract the shallow feature F0 from the input low-resolution infrared image to be reconstructed, and its mathematical expression is:

[0050] F0=H SF (I LR )

[0051] in, Represents an infrared image with a height of H, a width of W, and a channel number of 1. SF (·) represents the pixel embedding operation based on 3×3 convolution, Represents shallow features with C channels.

[0052] The deep feature extraction part is the main part of the present invention, which mainly includes N cascaded Swin Transformer Fused CNN Groups (STFCG) and 1 convolution layer to further extract and process the shallow feature F0 to obtain the deep feature F DF , its mathematical expression is:

[0053] F DF =H DF (F0)

[0054] Among them, F DF Represents the extracted deep features. DF (·) represents the deep feature extraction part, and the output of the internal cascaded STFCG module is expressed as

[0055] F i =H STFCGi (F0) i=1,2,...,N

[0056] Among them, H STFCGi (·) represents the i-th STFCG module.

[0057] The HCSAN of the present invention is mainly based on the SwinIR algorithm, and its essence lies in the stacking of multiple Swin-CNN fusion processing blocks (SCFPB). Specifically, each SwinTransformer hybrid convolution group (STFCG module) consists of L cascaded Swin-CNN fusion processing blocks (SCFPB modules) and 1 convolution layer.

[0058] The SCFPB module structure is as follows Figure 1 As shown in the upper part of the figure, the STFCG designed by the present invention has residual connections, and then stacks L SCFPBs and a convolutional layer at the end. The residual connection of the STFCG promotes the effective flow of feature information between different STFCGs, and the convolutional layer at the end increases the stability of the network translation and equivariance. The input and output expressions of the above i-th STFCG module can be further described as:

[0059] F i =H COVN H SCFPBL (H SCFPB(L-1) (...H SCFPBj (...H SCFPB1 (F i-1 ))))))+F i-1 j = 0, 1, ..., L

[0060] Among them, H COVN (·) represents the convolutional layer in the i-th STFCG, H SCFPBj (·) represents the jth SCFPB in the i-th STFCG, F i-1 Represents the output of the previous, i.e., the i-1th STFCG.

[0061] SCFPB is the core improvement of the present invention. It combines the advantages of Swin Transformer and integrates convolution operation on this basis to enhance the network's ability to capture local information. In addition, considering the characteristic of the infrared image grayscale change is gentle, the amount of parameters added after convolution is optimized to reduce unnecessary redundancy and improve processing efficiency. Figure 2 (a) and (b) show the original structure of Swin Transformer and the improved SCFPB respectively.

[0062] The original Swin Transformer module is mainly composed of two main parts, one is the multi-head self-attention (Shifted Window Multi-head Self Attention, SW-MSA) calculation using shifted window technology, and the other is an FNN including two linear mapping layers and GELU activation function. Each part is set with hierarchical normalization and residual connection to support deeper network training. In contrast, the SCFPB designed in the present invention adds a parallel convolution branch with reduced channel number to the module of SW-MSA. Before branch feature extraction, SCFPB uses 1×1 convolution and channel separation operations to reduce the number of channels, which will reduce the computational burden in the parallel process, and restore the channel scale at the end of the process through a 1×1 convolution layer. At the same time, due to the addition of 1×1 convolution, the linear mapping performed in SW-MSA to mix the results of multi-head self-attention calculations is omitted. Overall, the SCFPB designed in the present invention conforms to the general macro-architecture of visual Transformer, and its execution process is as follows:

[0063] Input Features After normalization, the number of channels is first expanded to 2αC (α<1) through 1×1 convolution, and then the feature map with expanded channels is evenly divided into two parts according to the channels. One part enters the SW-MSA branch to process self-attention, and the other enters the convolution branch to enhance the extraction of local feature information. After the output features of the convolution branch and the SW-MSA branch are fused, the channels are restored and combined with the input feature map F T A residual connection is performed, and finally a feed-forward neural network (FFN) is passed to obtain the final output. In the embodiment of the present invention, the feed-forward neural network (FFN) is composed of two linear mappings, one GELU activation and one layer normalization.

[0064] The parallel convolution branch used for channel compression in the present invention performs two channel compression convolutions and one activation function operation on its input, aiming to optimize the parameter quantity. Through the first convolution, the number of channels of the feature map is reduced, while the second convolution restores the number of channels. The mathematical expression of the convolution branch is described as:

[0065] FC1 =Conv3(F C )

[0066] F C2 =Conv3(Act(F C1 ))

[0067] Among them, Conv3(·) represents a 3×3 convolution operation, Act(·) is the activation function, and F C Represents the feature map of the input convolution branch, F C1 is the feature map after the first convolution, F C2 is the feature map after the second convolution. In these two convolution steps, the first convolution compresses the feature channel to reduce the network calculation and parameter amount. At this time, F C1 The size is H×W×αC / 3. The second convolution aims to restore the number of channels of the feature map, and the output F C2 The size is H×W×αC. The floating point multiplication calculation and parameter calculation formula of convolution is described as:

[0068] FLOPs = C i ×k 2 ×C o ×H×W

[0069] params=C i ×(k 2 ×C o +1)

[0070] Among them, C i is the number of channels of the input feature map, C o is the number of channels of the output feature map, H and W represent the height and width of the feature map respectively, and k is the size of the convolution kernel. According to the above formula, when the channel compression strategy is not adopted, if 3×3 convolution is used, the total floating point multiplication calculation amount and parameter amount are 18α 2 C 2 HW and 18α 2 C 2 , and after adopting the channel compression strategy, it is 6α 2 C 2 HW and 3α 2 C 2 Obviously, the channel reduction strategy reduces the amount of computation and parameters to only one third of the original amount.

[0071] In order to better combine the information provided by the convolution and SW-MSA branches, the SCFPB module fuses the convolution branch results before the SW-MSA branch to provide channel activation information. Specifically, the output of the convolution branch is passed through the CT module to provide channel activation information, and the output of the CT module is multiplied with the multi-head self-attention branch pixel using the shift window technology, and then added to the output pixel of the convolution branch.

[0072] The detailed structure of the CT module is as follows: Figure 3 As shown in the figure, the operations performed in the CT module are as follows: first, the output feature map of the convolution branch is compressed to 1×1 size through global average pooling; then, the number of channels is reduced using 1×1 convolution to reduce the ratio of 1 / r; then, the ReLU activation function is applied to the convolution result, and the number of channels is restored using 1×1 convolution; finally, the result is adjusted to between 0 and 1 through the Sigmoid function and used as the output of the CT module. The output of the CT module is combined with the SW-MSA branch to activate more critical channels, thereby improving the information flow and expression ability of the overall network.

[0073] The SW-MSA branch uses the shifted window multi-head self-attention mechanism unique to the Swin Transformer architecture to deeply explore the internal correlation of the image. The SCFPB module uses a 1×1 convolution operation to restore the number of channels, thereby optimizing the calculation of SW-MSA and eliminating the linear mapping steps commonly used to integrate the results of multi-head self-attention. In essence, SW-MSA still analyzes the internal connection of the image based on the self-attention mechanism, but compared with the traditional self-attention mechanism, Swin Transformer reduces the overall amount of calculation by performing self-attention calculations within a local window. The single-head self-attention calculation formula contains two matrix multiplications, and its mathematical expression is:

[0074]

[0075] Ignore the Softmax function and divide by the distance d k For an image block of dimension h×w×C, the amount of attention calculation in the above formula is:

[0076] N calculate =2(hw) 2 C

[0077] As can be seen from the above formula, the amount of self-attention calculation is directly related to the square of the product of the width and height of the image block size, which means that once the image block size increases, the amount of calculation will increase dramatically. In order to effectively reduce the amount of calculation, the PVT model adopts a space-efficient self-attention strategy, which reduces the dimensions of the key (K) and value (V) in the channel to reduce the amount of calculation when the matrix is ​​multiplied. This method has slowed down the increase in the amount of calculation to a certain extent, but at the same time, it has inevitably weakened the network's feature representation ability. SW-MSA balances the relationship between computational efficiency and feature capture by limiting the calculation of self-attention to a local window range. Specifically, SW-MSA divides the input h×w×C image block into M×M windows, and obtains a total of h / M×w / M windows. Then the self-attention is calculated in each window, which greatly reduces the amount of self-attention calculation for large-scale image blocks. At this time, the amount of calculation is reduced to:

[0078] N′ calculate =2M 2 HkDJ

[0079] However, the self-attention calculation within the local window may cause information transmission obstruction and weaken the ability to capture long-range information between different windows. To solve this problem, SW-MSA shifts the split window when calculating the self-attention in the next layer. The principle is as follows: Figure 4 Show.

[0080] Input an 8×8 image block, and the shift window design divides it into 4×4 windows, and then performs self-attention calculations in these windows. When calculating self-attention in the subsequent layer, the window configured in the previous layer is shifted to the right and down by two (M / 2) image blocks to obtain 9 windows. Calculating self-attention in these 9 windows can enable the exchange of information in the 4 windows of the previous layer.

[0081] When calculating multi-head self-attention, the original SW-MSA in Swin Transformer uses linear mapping to fuse the multi-head self-attention results, and its expression is

[0082] Z=MultiHead(Q,K,V)=Concat(h1,h2,...,h n )·W O

[0083] Among them, Z is the result of multi-head self-attention fusion, Q, K and V are the query (Query), key (Key) and value (Value) when calculating attention, h1, h2, ..., h n is the result of each head of multi-head self-attention, Concat(·) represents channel concatenation, W ORepresents the fused weight matrix. After adding the results of the convolution branch and the SW-MSA branch, the SCFPB module performs a 1×1 convolution operation, and its principle is similar to the above formula. Therefore, the SW-MSA in the SCFPB module simplifies the linear mapping step used to fuse the multi-head self-attention calculation results.

[0084] The high-resolution reconstruction part of the present invention is based on the shallow feature F0 and the deep feature F DF , infrared image super-resolution reconstruction is performed using the reconstruction function.

[0085] Specifically, the high-resolution reconstruction part of the present invention is implemented by two convolutional layers, one activation function and an upsampling block, and its expression is:

[0086] I HR =H REC (F0+F DF )

[0087] Among them, H REC (·) is the reconstruction function, which represents the entire high-resolution reconstruction stage. It combines the shallow feature F0 with the extracted deep feature F DF The result of the addition is used as input to reconstruct a high-resolution infrared image with a magnification of r. The output image resolution is rH×rW. In this structure, the group of "3×3 convolution and sub-pixel convolution" is stacked according to the magnification, which is the upsampling block. When the magnification r is a power of 2 such as 2, 4 or 8, the number of stacks is log2r. The upsampling block is the core of the high-resolution reconstruction stage, and its structure is as follows Figure 5 shown.

[0088] The parameter reduction strategy of channel enhancement and splitting of the present invention can be further described as follows: in view of the characteristic of gentle grayscale change of infrared images compared with visible light images, a parameter reduction strategy of channel enhancement and splitting is proposed in the SCFPB module to reduce information redundancy and improve calculation efficiency. Its structure is as follows Figure 2 As shown in (b). This parameter reduction strategy involves channel enhancement by 1×1 convolution before the convolution branch and the SW-MSA branch, then splitting equally according to the number of channels and applying 1×1 convolution in the subsequent processing of the two branches. This strategy can be summarized as Figure 6 Through this strategy, the number of channels of the feature map processed by the convolution branch and the SW-MSA branch is reduced to α times the original number, where α is a real number less than 1.

[0089] To achieve reduction, for the feature map of size H×W×C, the number of feature map channels is first increased to 2αC by 1×1 convolution, and then split into two parts with equal channels, each with a size of H×W×αC. The two split feature maps are used as inputs of the convolution branch and the SW-MSA branch respectively. In order to restore the number of feature map channels, 1×1 convolution is used to increase the number of feature map channels to C after the convolution branch and SW-MSA branch feature fusion.

[0090] According to the calculation formula of convolution floating-point multiplication and self-attention floating-point multiplication, the total floating-point multiplication calculation amount of the convolution branch and SW-MSA branch without and with the channel reduction strategy is

[0091]

[0092]

[0093] In order to verify the effect of the present invention, the following specific embodiment is provided, which mainly includes:

[0094] Step 1: Training Images

[0095] The dataset is constructed by installing three infrared cameras of different resolutions on a panel and placing it in a vehicle to capture different outdoor scene images. The dataset contains three infrared images of different resolutions, namely 160×120, 320×240, and 640×480. The 640×480 image is obtained by cropping the high-resolution camera image. Each infrared image of each resolution contains 951 training images and 50 test images. The image content in the dataset covers trees, people, buildings, vehicles, and other common outdoor scenes.

[0096] Although the data set contains three infrared images of different resolutions, with size ratios of ×2 and ×4 respectively, these images are not completely aligned and therefore cannot be directly used as high / low resolution image comparisons for training the network. In the training process of the HCSAN algorithm and other comparison algorithms, the example of the present invention selects images with a resolution of 640×480 as real infrared high-resolution images, and the low-resolution images input to the network are degraded according to the image degradation model described in the following formula. In the degradation model used, a Gaussian blur kernel with a standard deviation of 1 is used, Bicubic interpolation downsampling is used, and additive Gaussian noise is used for noise.

[0097] I LR =D w B l I HR +n o

[0098] In the training phase, image patches with a width and height of 48 pixels are cropped from the input low-resolution images as training input, and the infrared high-resolution image patches corresponding to ×2, ×4, and ×8 super-resolution are 96, 192, and 384 pixels in size, respectively. In order to improve the generalization ability of the network, a data augmentation strategy is used for the dataset during the training process, that is, the dataset images are randomly horizontally flipped or rotated by 90°, 180°, and 270°.

[0099] (2) Test image

[0100] Three different test image sets were selected for comprehensive performance evaluation. The first set of images consists of 50 test images from the dataset in the literature; the second set consists of 20 images selected from the FLIR infrared dataset, which is a public image library widely used in infrared imaging and target detection, including various types of infrared images such as pedestrians, animals, vehicles, and buildings; the third set consists of 20 campus and road scene images taken by a DJI drone equipped with an infrared camera.

[0101] (3) Network training

[0102] The HCSAN algorithm adopts a structural design consistent with SwinIR, where the number of stacked STFCGs N = 6, the number of stacked SCFPBs L in STFCG = 6, and the number of shallow feature channels extracted by pixel embedding C = 180.

[0103] The experimental platform is equipped with an Intel i5-8500 CPU, an NVIDIA TITAN V GPU, and a 16G RAM hardware system configuration. The software environment is built on the Windows 10Pro operating system, using the PyTorch1.7.1 deep learning framework and Python 3.7 programming language. At the training level, the L1 loss function is used, and ADAM is selected as the optimizer, with parameters set to β1=0.9 and β2=0.999, and ε is set to 1×10 -8 At the same time, set the batch size to 8 and set the -4 is the initial learning rate, and the learning rate is halved after every 20,000 iterations.

[0104] The HCSAN algorithm was compared with several other mainstream algorithms on the three groups of test images. The comparison algorithms include the traditional super-resolution algorithm Bicubic, the deep learning-based algorithms EDSR and RCAN, and the SwinIR algorithm. Each algorithm was trained and tested separately for magnifications of ×2, ×4, and ×8. In order to objectively evaluate the performance of each algorithm, Peak Signal-to-Noise Ratio (PSNR) and Structure Similarity Index Measure (SSIM) were selected as evaluation indicators. Table 1 records the performance indicators of each algorithm at different magnifications, where the best results in the same group of test images are in bold and the suboptimal results are underlined.

[0105] Table 1 Performance indicators of each algorithm at different magnifications.

[0106]

[0107] It can be seen from the experimental results that the Transformer architecture has obvious advantages under the 2x magnification. The evaluation indicators of the HCSAN algorithm and the SwinIR algorithm proposed in the present invention in the three groups of test images are significantly better than the EDSR and RCAN algorithms based on convolutional neural networks. In the comparison between the algorithm in this chapter and the SwinIR algorithm, it can be found that the PSNR objective evaluation index of the method of the present invention performs best in all test groups, while SSIM is only slightly inferior to the SwinIR algorithm in the second group of test images. Overall, under the 2x super-resolution magnification ratio, the average PSNR and SSIM indicators of the method of the present invention are both optimal, among which PSNR is improved by 0.1dB compared with the SwinIR algorithm, and SSIM is improved by 0.002. For the 4x super-resolution magnification, the Transformer architecture also has a clear advantage. On the second group of test images, although the PSNR of HCSAN is slightly lower than that of the SwinIR algorithm by 0.03dB, it continues to maintain its leading advantage in the remaining groups of test images. In terms of SSIM, HCSAN also surpasses other comparison algorithms in all aspects. The average PSNR and SSIM indicators also achieved the highest ranking. As for the magnification reaching 8 times, the performance advantage between the algorithms becomes less obvious due to the insufficient feature information in the input low-resolution image. However, HCSAN still maintains the best level in overall performance, followed by the SwinIR algorithm.

[0108] Based on the above analysis, the HCSAN algorithm proposed in the present invention has demonstrated superior reconstruction performance over other algorithms in objective evaluation experiments at various magnification levels, verifying the effectiveness and potential of the algorithm.

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A self-attention infrared image super-resolution reconstruction method based on hybrid convolution, characterized in that: The steps include: Step 1: embed pixels of the initial infrared image to be reconstructed to obtain its shallow feature F0; Step 2: Use the cascaded N Swin Transformer mixed convolution groups and 1 convolution layer to extract deep features from the shallow feature F0 to obtain the deep feature F DF ; Each of the Swin Transformer hybrid convolution groups consists of L cascaded Swin-CNN fusion processing blocks and 1 convolution layer; Step 3: Based on the shallow feature F0 and the deep feature F DF , infrared image super-resolution reconstruction is performed using the reconstruction function.

2. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 1, characterized in that: In step 1, a 3×3 convolution operation is used to extract the shallow feature F0 from the input low-resolution infrared image to be reconstructed, and its mathematical expression is: F0=H SF (I LR ) in, Represents an infrared image with a height of H, a width of W, and a channel number of 1. SF (·) represents the pixel embedding operation based on 3×3 convolution, Represents shallow features with C channels.

3. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 1, characterized in that: Among the N Swin Transformer mixed convolution groups, the output feature F of the i-th Swin Transformer mixed convolution group is i It is expressed as: F i =H STFCGi (F i-1 )=H COVN (H SCFPBL (H SCFPB(L-1) (...H SCFPBj (...H SCFPB1 (F i-1 )))))+F i-1 Among them, H STFCGi (·) represents the i-th Swin Transformer mixed convolution group, H COVN (·) represents the convolutional layer in the i-th SwinTransformer hybrid convolutional group, H SCFPBj (·) represents the jth Swin-CNN fusion processing block in the i-th Swin Transformer hybrid convolution group, F i-1 Represents the output of the i-1th Swin Transformer mixed convolution group.

4. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 1 or 3, characterized in that: The Swin-CNN fusion processing block performs the following process: The input feature map with height H, width W and number of channels C After normalization, the number of channels is expanded to 2αC through 1×1 convolution, where α is the parameter reduction factor, α<1; The feature map with expanded channel numbers is divided into two parts equally according to the channels. One part is input into the multi-head self-attention branch using the shift window technology to process self-attention, and the other part is input into the convolution branch to enhance the extraction of local feature information. The output features of the two branches are fused, and then the channels are restored and combined with the input feature map F. T A residual connection is performed, and finally a feedforward neural network is used to obtain the final output.

5. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 4, characterized in that: The multi-head self-attention branch using the shift window technology divides the input image block into multiple windows of equal size, calculates self-attention in each window respectively, and shifts the window when calculating the self-attention of the next layer, and calculates self-attention in each shifted window. The final multi-head self-attention fusion result Z is expressed as: Z=MultiHead(Q,K,V)=Concat(h1,h2,...,h n )·W O Among them, Z is the result of multi-head self-attention fusion, Q, K and V are the query, key and value when calculating attention respectively, h1,h2,...,h n is the result of each head of multi-head self-attention, Concat(·) represents channel concatenation, W O Represents the fusion weight matrix.

6. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 4, characterized in that: The convolution branch performs two channel compression convolutions and one activation function operation on its input, where the first convolution is used to reduce the number of channels and the second convolution is used to restore the number of channels. The operation of the convolution branch is expressed as: F C1 =Conv3(F C ) F C2 =Conv3(Act(F C1 )) Among them, Conv3(·) represents a 3×3 convolution operation, Act(·) is the activation function, and F C Represents the feature map of the input convolution branch, F C1 is the feature map after the first convolution, with a size of H×W×αC / 3, F C2 It is the feature map after the second convolution, with a size of H×W×αC.

7. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 4, characterized in that: The output features of the two branches are fused, and the implementation method is as follows: The output of the convolution branch is passed through the CT module to provide channel activation information, and the output of the CT module is multiplied by the multi-head self-attention branch pixel using the shift window technique, and then added to the output pixel of the convolution branch.

8. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 7, characterized in that: The CT module performs the following operations: The output feature map size of the convolution branch is compressed to 1×1 through global average pooling; Use 1×1 convolution to reduce the number of channels to reduce the ratio of 1 / r, where r is the magnification of the reconstructed image; The reduced feature map is passed through the ReLU activation function, and then 1×1 convolution is used again to restore the number of channels; The feature map of the restored channel number is adjusted to between 0 and 1 through the Sigmoid function as the output of the CT module.

9. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 1, characterized in that: The step 3 uses the reconstruction function to perform super-resolution reconstruction of the infrared image, which is achieved through two convolutional layers, one activation function and an upsampling block, and its expression is: I HR =H REC (F0+F DF ) Among them, H REC (·) is the reconstruction function, which combines the shallow feature F0 with the deep feature F DF The result of the addition is used as input to reconstruct a high-resolution infrared image with a magnification of r. The output image resolution is rH×rW.

10. The method for super-resolution reconstruction of infrared images based on self-attention based on hybrid convolution according to claim 9, characterized in that: The upsampling block is composed of 3×3 convolutions and sub-pixel convolutions stacked according to the magnification. When the magnification r is a power of 2, the number of stacks is log2 r.

Images