Infrared image super-resolution reconstruction method based on visible light correlation feature fusion

Abstract

The present invention discloses an infrared image super-resolution reconstruction method based on visible light correlation feature fusion, which belongs to the field of image processing technology. The present invention inputs an upsampled low-resolution infrared image into a learnable texture extraction module to extract query features; inputs a visible light image and a high-resolution visible light image that have been processed by downsampling and upsampling into a texture feature encoding module to extract key features and value features; generates a correlation guidance map and a corresponding feature weighted map based on the query features and the key features; obtains a migration feature map based on the corresponding feature weighted map and the value features; inputs shallow features, the correlation guidance map and the migration feature map into a cross-modal feature fusion module to obtain fused features and inputs them into a Transformer image reconstruction module to generate a super-resolution infrared image. The present invention significantly improves the guiding effect of visible light on the reconstruction of infrared details, making the reconstructed image clearer and sharper in texture details.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an infrared image super-resolution reconstruction method based on visible light correlation feature fusion. Background Art

[0002] Infrared images, due to their ability to capture thermal radiation information from objects, have unique advantages in applications at night, in adverse weather conditions, or under obstruction. They are widely used in security monitoring, intelligent driving, industrial inspection, medical diagnosis, and other fields. However, due to the physical characteristics and cost of infrared detectors, obtaining high-resolution infrared images is often challenging. Existing low-resolution infrared images often suffer from blurred details and a lack of texture information, which severely restricts their performance in applications requiring precise analysis and recognition. Therefore, developing efficient infrared image super-resolution (SR) reconstruction techniques has important research significance and practical application value.

[0003] In recent years, deep learning-based single-image super-resolution techniques have made significant progress, demonstrating strong detail recovery capabilities in visible light image super-resolution reconstruction. However, when these methods are directly applied to infrared images, performance is often unsatisfactory. This is primarily due to the inherent low contrast, weak texture, and high noise characteristics of infrared images, which make it difficult to recover rich high-frequency details from their own information alone.

[0004] To overcome the inherent information limitations of infrared images, utilizing other modalities, particularly higher-resolution visible light images, as guidance has become a promising research direction for infrared image super-resolution. Visible light images typically contain rich texture, edge, and structural details, which offer potential guidance for infrared image super-resolution reconstruction. However, existing infrared image super-resolution methods based on visible light guidance still face numerous challenges. First, the challenge of calculating visible light feature correlations and effectively transferring textures is crucial. Accurately calculating the cross-modal correlations between low-resolution infrared features and high-resolution visible light features, and using this information to accurately and robustly transfer useful textures from the visible light domain to the infrared domain, remains a key challenge. Existing methods may ignore feature amplitudes or overly rely on a single local source when calculating correlations, resulting in inaccurate and inflexible transferred textures and limited guidance quality. Second, the refinement and enhancement of the fused features requires optimization. After fusing the visible light guidance features with the infrared native features, designing efficient network modules to refine and enhance these hybrid features is crucial for improving the final reconstruction quality. In particular, utilizing effective activation and attention mechanisms to highlight and restore key details in the infrared image is crucial for improving the final reconstruction quality. Traditional feature processing modules may not be able to fully adapt to the complex feature characteristics after this cross-modal fusion. Third, efficient deep fusion of multimodal features is insufficient: How to effectively deeply fuse the features derived from high-resolution visible light guidance with the basic features extracted from the original low-resolution infrared image to maximize the complementary information of the two while avoiding redundancy and conflict is a key factor affecting reconstruction performance. Simple fusion strategies are unable to cope with complex cross-modal feature interactions, limiting the detail richness and clarity of the final reconstructed image. Summary of the Invention

[0005] To address the problems of blurred details caused by low resolution of infrared images and the shortcomings of existing visible light-guided methods in precise matching and robust transfer of cross-modal information (for example, traditional correlation ignores feature amplitude, and hard attention relies on a single source point) and effective fusion, the present invention proposes an infrared image super-resolution reconstruction method based on visible light correlation feature fusion.

[0006] The present invention is implemented by the following technical solution: a method for super-resolution reconstruction of infrared images based on visible light correlation feature fusion, the steps of which are as follows:

[0007] S1: Low-resolution infrared image Input to the shallow feature extraction module to extract its shallow feature F; upsample the low-resolution infrared image Input to the learnable texture extraction module (LTE), from which the query feature Q is extracted; the visible light image that has been downsampled and then upsampled is processed and high-resolution visible light images , input to the texture feature encoding module (TFE), from which the key feature K and value feature V are extracted; the query feature Q and the key feature K are input to the correlation calculation module, the correlation calculation module calculates the similarity between the two, and generates a correlation guidance map S and a corresponding feature weighted map C for weighting; the corresponding feature weighted map C and the value feature V are input to the corresponding feature weighting module (CFWM), and the corresponding feature weighted map C and the value feature V are weighted to obtain the migration feature map M;

[0008] S2: Input the shallow features F, the correlation guide map S and the migration feature map M into the cross-modal feature fusion module to obtain the fusion features;

[0009] S3: The fused features are input into the Transformer image reconstruction module, which is used to further process, upsample, and restore details of the fused features to finally generate a super-resolution infrared image. .

[0010] Further preferably, the specific process of calculating the corresponding feature weighted graph C is as follows:

[0011] ;

[0012] in, Represents the calculated weight, which is also an element in the corresponding feature weighted graph C. k is the spatial position index in the high-resolution visible light image. is the similarity between the i-th position in the low-resolution infrared image and the j-th position in the high-resolution visible light image, is the similarity between the feature at position i in the low-resolution infrared image and the feature at position k in the high-resolution visible light image.

[0013] Further preferably, the correlation guide map S is calculated as follows:

[0014] ;

[0015] in, Represents the value of the i-th position in the correlation guide graph S.

[0016] Further preferably, the cross-modal feature fusion module (CFFM) first performs channel splicing on the shallow features F and the migration feature map M; then, the correlation guidance map S is used to perform element-by-element multiplication on the spliced ​​features to achieve intelligent modulation and guidance of the fusion process, and the modulated features are then passed through a layer normalization block; the layer-normalized features are fed into two parallel branches: one branch contains a 3×3 convolution and ReLU activation function for capturing local spatial features; the other branch contains a 1×1 convolution and ReLU activation function for channel dimension interaction and feature projection; the outputs of the two branches are then channel-spliced ​​again and the final feature integration is performed through a 1×1 convolution; finally, the integrated features are element-by-element added to the shallow features F of the original input to form a local residual connection to obtain the final fused features.

[0017] Further preferably, the Transformer image reconstruction module consists of n cascaded residual multi-scale attention groups, three 3×3 convolutions, a sub-pixel convolution layer, and a global residual connection.

[0018] Further preferably, the residual multi-scale attention group is composed of m cascaded residual multi-scale attention blocks, an overlapping cross fusion attention block (OCFAB), a 3×3 convolution and a local residual in sequence.

[0019] Further preferably, each residual multi-scale attention block is a functional unit, and the residual multi-scale attention block first performs a first layer normalization on the original input features, and then sends the first normalized features to the multi-scale channel attention block and the shifted window multi-head self-attention block in parallel; after summing the outputs of the multi-scale channel attention block and the shifted window multi-head self-attention block, the first residual connection is performed with the original input features to obtain intermediate features; then, the intermediate features are sequentially subjected to a second layer normalization and a multi-layer perceptron (MLP); finally, the output of the multi-layer perceptron is residually connected with the intermediate features for a second time to obtain the output features of the residual multi-scale attention block.

[0020] Further preferably, the processing flow of the multi-scale channel attention block is as follows: the input features first undergo a 3×3 convolution for feature extraction, and then are sent to the multi-scale large-core attention module for processing. The output of the multi-scale large-core attention module is divided into two paths. The first path is processed by the GELU activation function and the multi-layer perceptron, and the second path is processed by the GELU activation function and the maximum pooling operation. The two results are added element by element, then passed through the Sigmoid activation function, and then the output of the Sigmoid activation function is multiplied element by element with the output feature of the multi-scale large-core attention module. The multiplication result is then subjected to a 3×3 convolution and then sent to the channel attention block for processing.

[0021] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processor implements the above-mentioned infrared image super-resolution reconstruction method.

[0022] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned infrared image super-resolution reconstruction method is implemented.

[0023] The beneficial effects of the present invention are:

[0024] (1) The Visible Correlation Feature Modulation Module (VCFM) designed in this paper can efficiently transfer high-resolution visible light textures, effectively address cross-modal parallax, and significantly improve the detail richness and realism of reconstructed infrared images. Through improved scaled dot product correlation calculation and a softmax-based weighted averaging mechanism, it achieves more accurate and robust transfer of high-resolution visible light texture information, effectively overcoming the problems of traditional methods that ignore amplitude information and rely on a single source point.

[0025] (2) The cross-modal feature fusion module (CFFM) designed in this paper cleverly utilizes the correlation guidance map S generated by the correlation calculation module in the visible light correlation feature modulation module to intelligently modulate and fuse the infrared self-features with the visible light migration texture, maximizing the utilization of complementary information. Furthermore, the multi-scale channel attention block (MS-CAB) is integrated into the Transformer reconstruction module to enhance the expression of the fused features. This achieves a deep and effective integration of the visible light guidance features and the infrared self-features, greatly enhancing the representation capability and robustness of the fused features and laying a solid foundation for high-quality reconstruction.

[0026] (3) The multi-scale channel attention block (MS-CAB) introduced in this paper enhances the model's ability to perceive and utilize key feature channels of different scales, significantly improving the clarity and sharpness of the reconstructed infrared image in terms of edges and textures. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is the network structure diagram of the visible light guided super-resolution infrared image reconstruction of the present invention.

[0028] Figure 2 This is a structural diagram of the cross-modal feature fusion module of the present invention.

[0029] Figure 3 This is a structural diagram of the Transformer image reconstruction module of the present invention.

[0030] Figure 4 This is the residual multi-scale attention group structure diagram of the present invention.

[0031] Figure 5 This is the residual multi-scale attention block structure diagram of the present invention.

[0032] Figure 6 This is the structural diagram of the multi-scale channel attention block of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings.

[0034] The first embodiment of the present invention introduces a method for super-resolution reconstruction of infrared images based on visible light correlation feature fusion. Figure 1 The visible light guided super-resolution infrared image reconstruction network shown in the figure has the following core components: shallow feature extraction module, visible light correlation feature modulation module, cross-modal feature fusion module and Transformer image reconstruction module. Among them, the shallow feature extraction module is used to extract the low-resolution infrared image. The shallow feature F is extracted from the image processing module; the visible light correlation feature modulation module receives multiple inputs and uses the corresponding feature weighting module and the correlation guidance module to generate cross-modal guidance features for guiding reconstruction; the cross-modal feature fusion module is used to deeply fuse the shallow feature F with the cross-modal guidance features to obtain fused features; the Transformer image reconstruction module further processes the fused features and completes upsampling to finally generate a super-resolution infrared image. .

[0035] The specific steps are as follows:

[0036] S1: First, the low-resolution infrared image Input to the shallow feature extraction module to extract its shallow features F. At the same time, the visible light correlation feature modulation module (VCFM) is executed to upsample the low-resolution infrared image Input to the learnable texture extraction module (LTE), from which the query feature Q is extracted; the visible light image that has been downsampled and then upsampled is processed and high-resolution visible light images , is input into the texture feature encoding module (TFE), from which the key feature K and value feature V are extracted. Subsequently, the query feature Q and the key feature K are input into the correlation calculation module, which calculates the similarity between the two and generates a correlation guidance map S and a corresponding feature weighted map C for weighting. Next, the corresponding feature weighted map C and the value feature V are input into the corresponding feature weighting module (CFWM), which uses the corresponding feature weighted map C and the value feature V for weighting to obtain the migration feature map M.

[0037] S2: The shallow features F extracted from the low-resolution infrared image and the obtained correlation guidance map S and migration feature map M are input into the cross-modal feature fusion module CFFM for deep fusion to obtain the fused features.

[0038] S3: The fused features are input into the Transformer image reconstruction module, which is used to further process, upsample, and restore details of the fused features to finally generate a super-resolution infrared image. .

[0039] Each step or module is described in further detail below:

[0040] Reference Figure 1 , firstly, the infrared low-resolution image Perform shallow feature extraction. The specific process is as follows:

[0041] (1);

[0042] in, represents the shallow feature extraction module, represents a low-resolution infrared image, Represents shallow features.

[0043] At the same time, the visible light feature extraction operation is performed, and the visible light correlation feature modulation module proposed in this application receives the low-resolution infrared image ( ), its upsampled version ( ), high-resolution visible light images ( ) and the domain-consistent processed visible light image ( )) as input to achieve accurate extraction and fusion of visible light feature information.

[0044] The processing flow of the visible light correlation feature modulation module is as follows:

[0045] First, the input texture is extracted using a learnable texture extraction module (LTE). Processing, using the texture feature encoding module (TFE) to 、 The extracted texture features (query feature Q, key feature K, value feature V) are processed as the three basic elements for implementing cross-modal guidance. The process is described as follows:

[0046] (2);

[0047] (3);

[0048] (4);

[0049] in, represents a learnable texture extraction operation, Represents the texture feature encoding and extraction operation. From the extracted texture features, the three elements Q, K, and V required for correlation calculation and feature weighting are obtained.

[0050] Then, correlation calculation is performed to calculate the similarity between the query feature Q and the key feature K to calculate the low-resolution infrared image and high-resolution visible light images The similarity between them effectively transfers the texture information related to the low-resolution infrared image from the high-resolution visible light image, while suppressing the transfer of irrelevant textures. The process can be described as:

[0051] (5);

[0052] in, represents the similarity between the i-th position in the low-resolution infrared image and the j-th position in the high-resolution visible light image, Represents the vector inner product operation, and are the expanded vectors of query feature Q and key feature K respectively, is the eigenvector Dimensions (or ), the generated It will be used for the subsequent calculation of the correlation guide map S and migration feature map M.

[0053] Then enter the corresponding feature weighting module for processing. Before this, a corresponding feature weighting graph C needs to be calculated. The specific process is as follows:

[0054] (6);

[0055] in, Represents the calculated weight, which is also an element in the corresponding feature weighted graph C. These weights can be regarded as a correlation measure, which represents the texture contribution of all positions in the high-resolution visible light image to the i-th position in the low-resolution infrared image. k is the index of all possible spatial positions in the high-resolution visible light image. is the similarity between the feature at position i in the low-resolution infrared image and the feature at position k in the high-resolution visible light image.

[0056] The corresponding feature weighted map C and the value feature V are fed into the corresponding feature weighting module, which transfers the value feature V. In the weighted texture transfer module, for each position in the low-resolution image, the value feature V is weighted and summed according to its correlation with all positions in the high-resolution visible light image to obtain the transferred feature map M.

[0057] Finally, the weighted summation of the high-resolution features V of the visible light image is performed according to these weights:

[0058] (7);

[0059] in, Represents the texture feature that is finally migrated to the i-th position in the low-resolution infrared image, that is, the value of the i-th position in the migration feature map M. is the feature vector corresponding to the jth position of the high-resolution visible light image in the value feature V.

[0060] At the same time, a correlation guide graph S needs to be calculated. The specific process is as follows:

[0061] (8);

[0062] in, Represents the value of the i-th position in the correlation bootstrap graph.

[0063] Then, the shallow features F, the correlation guide map S and the migration feature map M are simultaneously input into the cross-modal feature fusion module (CFFM). Figure 2 , the module first performs channel-wise splicing on the shallow features F and the migration feature map M. Then, the correlation guidance map S is used to perform element-wise multiplication on the spliced ​​features to achieve intelligent modulation and guidance of the fusion process. The modulated features are then passed through a layer normalization block. The layer-normalized features are fed into two parallel branches: one branch contains a 3×3 convolution and ReLU activation function to capture local spatial features; the other branch contains a 1×1 convolution and ReLU activation function for channel dimension interaction and feature projection. The outputs of the two branches are then channel-wise spliced ​​again and the final feature integration is performed through a 1×1 convolution. Finally, the integrated features are element-wise added to the shallow features F of the original input to form a local residual connection to obtain the final fused features. Through the above series of operations, the cross-modal feature fusion module (CFFM) can deeply and efficiently fuse infrared shallow features with visible light migration texture features, and use the correlation guidance map S to intelligently modulate the fusion process to generate fused features containing rich cross-modal complementary information and texture details, providing a more powerful representation for subsequent image reconstruction.

[0064] The process is described as:

[0065] (9);

[0066] (10);

[0067] in, represents the intermediate features after fusion, represents the fusion features output by the cross-modal feature fusion module, Indicates channel splicing operation, operator represents the element-wise multiplication between features, represents a 3×3 convolution, represents 1×1 convolution, represents the RELU activation function, Representation layer normalization operation, Represents shallow features.

[0068] Finally, we enter the image reconstruction stage, and input the fused features into the subsequent Transformer image reconstruction module. Figure 3 The Transformer image reconstruction module consists of n cascaded residual multi-scale attention groups, three 3×3 convolutions, a sub-pixel convolution layer, and a global residual connection. The residual multi-scale attention group can extract multi-scale features from the fused features, more efficiently utilize the features for image reconstruction, and thus generate a clearer image. In this example, n=10, and the process is specifically described as follows:

[0069] (11);

[0070] (12);

[0071] (13);

[0072] in, represents the sub-pixel convolution layer, represents the upsampled features, represents the reconstructed super-resolution infrared image, represents the i-th residual multi-scale attention group, and n represents the number of residual multi-scale attention groups ( ), represents the features output by the i-1th residual multi-scale attention group, represents the features output by the i-th residual multi-scale attention group, Represents the features output by the nth residual multi-scale attention group.

[0073] like Figure 4 As shown in Figure 1, the residual multi-scale attention group is composed of m cascaded residual multi-scale attention blocks, an overlapping cross-fusion attention block (OCFAB), a 3×3 convolution, and a local residual. In this embodiment, m=20, and the processing of the residual multi-scale attention group is described as follows:

[0074] (14);

[0075] (15);

[0076] in, represents the jth residual multi-scale attention block of the i-th residual multi-scale attention group, represents the features output by the j-1th residual multi-scale attention block of the i-th residual multi-scale attention group, represents the features output by the jth residual multi-scale attention block of the i-th residual multi-scale attention group, represents the features output by the mth residual multi-scale attention block of the i-th residual multi-scale attention group, represents the input features of the i-th residual multi-scale attention group, represents the output features of the i-th residual multi-scale attention group, Represents the multi-scale extraction operation of overlapping cross-fusion attention blocks.

[0077] like Figure 5 As shown, each residual multi-scale attention block is a functional unit that first performs the first layer normalization on the original input features and then feeds the first normalized features into the multi-scale channel attention block in parallel. and shift window multi-head self-attention block ; After summing the outputs of the multi-scale channel attention block and the shifted window multi-head self-attention block, the first residual connection is performed with the original input features to obtain the intermediate features; then, the intermediate features are sequentially subjected to the second layer normalization and multi-layer perceptron (MLP); finally, the output of the multi-layer perceptron is residually connected with the intermediate features for the second time to obtain the output features of the residual multi-scale attention block, realizing flexible activation of features, adjustment of channel importance and capture of spatial dependencies, and enhancing feature expression ability and training stability through two-level residual connections.

[0078] In general, for a given original input feature , the entire calculation process of the residual multi-scale attention block is:

[0079] (16);

[0080] (17);

[0081] (18);

[0082] in, represents the features after the first layer normalization, represents the intermediate features, represents the output of the residual multi-scale attention block, Representation layer normalization operation, Represents a multilayer perceptron. represents the shift window multi-head self-attention block, Represents a multi-scale channel attention block.

[0083] Reference Figure 6 , multi-scale channel attention block The processing flow is as follows: Input features first undergo a 3×3 convolution for feature extraction and are then fed into the multi-scale large-core attention module for processing. The output of the multi-scale large-core attention module is split into two paths: the first path undergoes GELU activation and a multi-layer perceptron, and the second path undergoes GELU activation and max pooling. The two results are element-wise added, then passed through a sigmoid activation function. The output of the sigmoid activation function is then element-wise multiplied with the output features of the multi-scale large-core attention module. The multiplication result then undergoes a 3×3 convolution and is then fed into the channel attention block for processing. In the channel attention block, the features first undergo a global pooling layer to compress the spatial dimensions and obtain global context. They then undergo a 3×3 convolution, a ReLU activation function, and a 3×3 convolution, followed by a sigmoid activation function to generate per-channel weights. Finally, these channel weights are element-wise multiplied with the input features of the channel attention block (i.e., the convolution results fed into the channel attention block) to obtain channel-weighted features, which serve as the output features of the multi-scale channel attention block. The processing can be described as:

[0084] (19);

[0085] (20);

[0086] (twenty one);

[0087] in, represents the multi-scale large-kernel attention module, represents the channel attention operation, represents the GELU activation function, represents the Sigmoid activation function, represents a 3×3 convolution, Represents the element-wise multiplication operation between features, represents a multilayer perceptron, represents the maximum pooling operation, represents the input features of the multi-scale channel attention block, Represents the output features of the multi-scale large-core attention module, Represents the multi-scale large-core attention feature after Sigmoid, Represents the output features of the multi-scale channel attention block.

[0088] To fully validate the effectiveness of the proposed infrared image super-resolution reconstruction method based on visible light correlation feature fusion, a series of comparative experiments were conducted on the publicly available FLIR-aligned infrared dataset. This dataset contains a rich collection of infrared images of urban scenes, 500 of which were selected as the test set. All experiments were conducted using a bicubic interpolation degradation model.

[0089] In experiments, the method of the present invention was compared with several mainstream and advanced image super-resolution algorithms, including: EDSR (Enhanced Deep Residual Network), RDN (Residual Dense Network), RCAN (Residual Channel Attention Network), SwinIR (Image Restoration Network based on Swin Transformer), HAT (Hybrid Attention Transformer), ATD (Advanced Super-Resolution Transformer with Adaptive Token Dictionary), UGSR (Unaligned Guided Thermal Infrared Super-Resolution), and PAGSR (Multi-Scale Edge Attention Thermal Infrared Super-Resolution). The VCFM is the visible light correlation feature modulation module designed by the present invention, the CFFM is the cross-modal feature fusion module, and the MS-CAB is the multi-scale channel attention block introduced by the present invention.

[0090] Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are selected as core evaluation metrics to assess the quality of super-resolution results. PSNR primarily measures the pixel-level difference between the reconstructed image and the original high-resolution image. Higher values ​​indicate less distortion. SSIM assesses the similarity between images based on brightness, contrast, and structure. Values ​​closer to 1 indicate higher similarity and better reconstruction quality. Both metrics are calculated on the Y channel of the YCbCr color space and reported as averages across the entire test set. The model's parameter count (params) and runtime (runtime) are also considered to assess model complexity and efficiency. The parameter count (in millions) reflects the model size, while the runtime refers to the average time (in seconds) required for the model to process a single image. By comprehensively analyzing these metrics, the performance, efficiency, and practicality of a super-resolution algorithm can be more comprehensively assessed. As shown in Table 1, the proposed method achieves the best results in terms of PSNR and SSIM. While the number of parameters and runtime increase, this translates to significantly improved metrics and visual quality.

[0091] Table 1 Comparative experimental results

[0092]

[0093] A second embodiment of the present invention provides an electronic device including a memory and a processor, wherein the memory stores computer-readable instructions, and when the instructions are executed by the processor, the processor implements the above-mentioned infrared image super-resolution reconstruction method.

[0094] A third embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements the above-mentioned infrared image super-resolution reconstruction method.

[0095] The above description merely represents preferred embodiments of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above disclosure to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A method for super-resolution reconstruction of infrared images based on visible light correlation feature fusion, characterized in that the steps as follows: S1: Low-resolution infrared image Input to the shallow feature extraction module to extract its shallow feature F; upsample the low-resolution infrared image Input to the learnable texture extraction module, from which the query feature Q is extracted; the visible light image that has been downsampled and then upsampled is and high-resolution visible light images , input to the texture feature encoding module, from which the key feature K and value feature V are extracted; The query feature Q and the key feature K are input into the correlation calculation module, which calculates the similarity between the two and generates a correlation guide graph S and a corresponding feature weighted graph C for weighting: ； in, Represents the calculated weight, which is also an element in the corresponding feature weighted graph C. k is the spatial position index in the high-resolution visible light image. is the similarity between the i-th position in the low-resolution infrared image and the j-th position in the high-resolution visible light image, is the similarity between the feature at position i in the low-resolution infrared image and the feature at position k in the high-resolution visible light image; Input the corresponding feature weighted graph C and value feature V into the corresponding feature weighting module, and use the corresponding feature weighted graph C and value feature V for weighting: ； in, Represents the texture feature that is finally migrated to the i-th position in the low-resolution infrared image, that is, the value of the i-th position in the migration feature map M. is the feature vector corresponding to the jth position of the high-resolution visible light image in the value feature V; the migration feature map M is obtained; The correlation bootstrap map S is calculated as follows: ； in, Represents the value of the i-th position in the correlation guide graph S; S2: The shallow features F, the correlation guidance map S and the migration feature map M are input into the cross-modal feature fusion module CFFM. The cross-modal feature fusion module first performs channel splicing on the shallow features F and the migration feature map M. Then, the correlation guidance map S is used to perform element-by-element multiplication on the spliced ​​features to achieve intelligent modulation and guidance of the fusion process. The modulated features are then passed through a layer normalization block. The layer-normalized features are fed into two parallel branches: one branch contains a 3×3 convolution and ReLU activation function to capture local spatial features; the other branch contains a 1×1 convolution and ReLU activation function for channel dimension interaction and feature projection. The outputs of the two branches are then channel-spliced ​​again and finally integrated through a 1×1 convolution. Finally, the integrated features are element-wise added to the shallow features F of the original input to form a local residual connection to obtain the final fused features. S3: The fused features are input into the Transformer image reconstruction module, which is used to further process, upsample, and restore details of the fused features to finally generate a super-resolution infrared image. ; The Transformer image reconstruction module consists of n cascaded residual multi-scale attention groups, 3 3×3 convolutions, a sub-pixel convolution layer and a global residual connection; the residual multi-scale attention group is composed of m cascaded residual multi-scale attention blocks, an overlapping cross-fusion attention block, a 3×3 convolution and a local residual in turn; each residual multi-scale attention block is a functional unit, and the residual multi-scale attention block first performs a first layer normalization on the original input features, and then sends the first normalized features to the multi-scale channel attention block and the shifted window multi-head self-attention block in parallel; the outputs of the multi-scale channel attention block and the shifted window multi-head self-attention block are summed, and then the first residual connection is performed with the original input features to obtain the intermediate features; then, the intermediate features are sequentially subjected to a second layer normalization and a multi-layer perceptron; finally, the output of the multi-layer perceptron is residually connected with the intermediate features for a second time to obtain the output features of the residual multi-scale attention block; The processing flow of the multi-scale channel attention block is as follows: the input feature first undergoes a 3×3 convolution for feature extraction, and then is sent to the multi-scale large-core attention module for processing; the output of the multi-scale large-core attention module is divided into two paths, the first path is processed by the GELU activation function and the multi-layer perceptron, and the second path is processed by the GELU activation function and the maximum pooling operation. The two results are added element by element, and then passed through the Sigmoid activation function. The output of the Sigmoid activation function is then multiplied element by element with the output feature of the multi-scale large-core attention module. The multiplication result is then passed through a 3×3 convolution and then sent to the channel attention block for processing.

2. An electronic device comprising a memory and a processor, wherein the memory stores computer-readable instructions, wherein: When the instructions are executed by the processor, the processor implements the infrared image super-resolution reconstruction method according to claim 1.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the infrared image super-resolution reconstruction method according to claim 1 is implemented.

Images