Multi-level attention visible light guided infrared image super-resolution method and system

Abstract

The invention discloses a visible light guided infrared image super-resolution method and system based on multilevel attention, and belongs to the field of image processing and deep learning. Respectively extracting a multi-scale first feature and a multi-scale second feature of the original low-resolution infrared image and the high-resolution visible light image; a cross-modal feature fusion module is adopted to carry out bidirectional guidance on each pair of first features and second features with the same scale, and multi-scale fusion features are generated; and decoding and reconstructing the multi-scale fusion features in sequence from small scale to large scale to obtain a high-resolution thermal infrared image. The multi-layer cross-modal feature fusion module adopts a serial three-network structure, bidirectional guidance is realized through the synergistic effect of a mobile window self-attention block and a cross attention block, and interaction and fusion of deep information between modals are enhanced. The super-resolution reconstruction performance of the method is superior to that of an existing method, and indexes such as SSIM and PSNR are obviously improved.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and deep learning, and in particular to a multi-level attention visible light guided infrared image super-resolution method and system. Background Art

[0002] Thermal infrared images are capable of imaging in environments without visible light and have good anti-interference and target recognition capabilities. However, due to the high manufacturing cost and limited resolution of thermal infrared image sensors, the acquired images often have problems such as low resolution, blurred edges, and loss of details, which seriously affect the accuracy and stability of downstream target detection and recognition tasks. In contrast, visible light images are easy to acquire, have high resolution, and are rich in texture information. Therefore, in recent years, a large number of studies have attempted to use high-resolution visible light images as guidance to assist in improving the resolution of thermal infrared images. This type of task is usually called guided super-resolution. The core challenge lies in how to effectively extract and fuse the structure, edge, and semantic information between the two images under conditions of large modal differences, thereby restoring clearer and more realistic high-resolution infrared images.

[0003] Existing guided image super-resolution techniques fall into two main categories. One relies heavily on shallow feature mapping techniques like bilateral filtering and dictionary learning, which have limited fusion capabilities and struggle to capture complex cross-modal relationships. The other relies on deep learning methods based on convolutional neural networks (CNNs) or Transformer architectures. Pure CNN architectures are limited in modeling long-range dependencies and maintaining structural consistency. Mainstream Transformer-based models all use a cascade architecture, resulting in high computational cost and a large number of parameters. Direct application to cross-modal tasks can easily lead to modal confusion.

[0004] Therefore, there is an urgent need for an infrared image super-resolution reconstruction method with high fusion efficiency and strong cross-modal modeling capabilities to better meet the dual requirements of image resolution and detail restoration capabilities in practical applications. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of existing visible light guided thermal infrared image super-resolution methods in terms of insufficient cross-modal information fusion, low fusion efficiency, and insufficient image detail restoration capabilities, and to provide a multi-level attention visible light guided infrared image super-resolution method and system that can achieve efficient deep-level feature interaction while maintaining modal structure independence, and effectively improve the edge clarity and structural restoration capabilities of the reconstructed image.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a visible light guided infrared image super-resolution method based on multi-level attention, comprising:

[0008] Extracting multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image respectively;

[0009] A cross-modal feature fusion module is used to perform bidirectional guidance on each pair of first and second features of the same scale to generate multi-scale fused features; the cross-modal feature fusion module is composed of a first network, a second network, and a third network connected in series; the first network is a symmetrical dual-branch structure with dual input and dual output obtained by stacking one or more (preferably 2-4) moving window self-attention blocks; the second network is a cross-branch structure with dual input and dual output obtained by stacking one or more (preferably 1) moving window cross-attention blocks, where the dual output results are first added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more (preferably 2-4) moving window self-attention blocks; the moving window cross-attention block contains a cross-attention block formed by bidirectional cross-attention of the self-attention blocks;

[0010] The multi-scale fusion features are decoded and reconstructed in order from small to large scale to obtain a high-resolution thermal infrared image; in the decoding and reconstruction process of each layer, the input smaller-scale fusion features are reconstructed to the same spatial resolution as the larger-scale fusion features, and added to the larger-scale fusion features in the channel dimension as the decoding and reconstruction result of this layer; the decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added to the original low-resolution infrared image in the channel dimension as the final high-resolution thermal infrared image.

[0011] Furthermore, taking the combination of the original low-resolution infrared image and the corresponding high-resolution visible light image as input, a two-branch convolutional neural network is used to extract multi-scale first features and multi-scale second features respectively;

[0012] The dual-branch convolutional neural network has a symmetrical structure. Each branch is a multi-level coding network. The first coding layer contains a convolutional layer, and the remaining coding layers contain a series of pooling layers and convolutional layers. The output of each layer of the multi-level coding network corresponds to a feature of a scale.

[0013] Furthermore, a multi-level decoding network is used to decode and reconstruct the multi-scale fusion features in order from small to large scale. The last decoding layer contains a convolutional layer, and the remaining decoding layers contain concatenated convolutional layers and upsampling layers.

[0014] Furthermore, the cross-modal feature fusion module for bidirectionally guiding the first features and the second features of different scales has the same structure and independent parameters.

[0015] Furthermore, the moving window self-attention block is obtained by connecting alternating self-attention blocks and moving windows in series, and the moving window cross-attention block is obtained by connecting alternating cross-self-attention blocks and dual-channel moving windows in series.

[0016] Furthermore, the cross self-attention block is a cross dual-branch structure formed by a first self-attention block and a second self-attention block that cross in both directions. The two self-attention blocks respectively receive thermal infrared features and visible light features to calculate query vectors, key vectors and value vectors, and after swapping the query vector based on thermal infrared features with the query vector based on visible light features, the multi-head attention outputs are calculated respectively. The multi-head attention outputs of the two branches are swapped and added to the input features received by each branch before layer normalization is performed. The normalized results are then passed through a feedforward neural network, residual connection, and layer normalization to obtain the final output.

[0017] Furthermore, the method further includes a step of cropping the input original low-resolution infrared image and high-resolution visible light image, wherein the size of the cropped image is a maximum multiple of the minimum acceptable image size unit.

[0018] In a second aspect, the present invention provides a visible light guided infrared image super-resolution system based on multi-level attention, which is used to implement the above-mentioned visible light guided infrared image super-resolution method based on multi-level attention.

[0019] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the above-mentioned visible light-guided infrared image super-resolution method based on multi-level attention is implemented.

[0020] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0021] The memory is used to store computer programs;

[0022] The processor is configured to implement the above-mentioned visible light guided infrared image super-resolution method based on multi-level attention when executing the computer program.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] This paper introduces a dual-branch, multi-scale feature extraction architecture, achieving comprehensive representation of both modal internal and external features and improving the accuracy of cross-modal feature alignment. A multi-level cross-modal feature fusion module is designed, employing a cascaded three-network architecture. This architecture achieves bidirectional guidance through the synergy of moving window self-attention blocks and cross-attention blocks, enhancing the interaction and fusion of deep information between modalities. Specifically, the dual-branch self-attention architecture of the first network independently extracts deep features from both modalities. The cross-branch dual-branch architecture of the second network implements bidirectional information exchange through a query vector swap mechanism (exchanging query vectors generated by thermal infrared features with those generated by visible light features). The third network finally integrates and optimizes the fused features. This hierarchical fusion approach effectively addresses modal confusion and inadequate modeling of long-range dependencies in traditional methods, laying the foundation for high-quality reconstruction. During the decoding and reconstruction phase, a bottom-up approach is used to hierarchically process the fused features, with low-level features providing fine structural cues and high-level features carrying semantic information. The final output layer innovatively performs channel-wise summation of the decoded results with the original low-resolution infrared image, preserving the original radiometric characteristics while incorporating detailed information into the reconstruction. Experimental results show that the super-resolution reconstruction performance of the proposed method is better than that of existing methods, and has obvious improvements in indicators such as SSIM and PSNR. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the multi-level attention visible light guided infrared image super-resolution method proposed in the present invention;

[0026] Figure 2 is a model block diagram in an embodiment of the present invention;

[0027] Figure 3 This is the structural diagram of the cross-modal fusion module;

[0028] Figure 4 This is the structural block diagram of the moving window self-attention module;

[0029] Figure 5 This is the structural block diagram of the moving window cross self-attention module;

[0030] Figure 6 Schematic diagram of random cropping strategy;

[0031] Figure 7 Schematic diagram of the effect of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0033] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to actual circumstances.

[0035] like Figure 1 As shown in the figure, the multi-level attention visible light guided infrared image super-resolution method proposed in this paper, given a low-resolution thermal infrared image and a high-resolution visible light image, aims to reconstruct a super-resolution thermal infrared image. It mainly includes the following steps:

[0036] S1, obtain a pair of registered input images, including the original low-resolution infrared image and the high-resolution visible light image.

[0037] For example, the pair of registered input images obtained in step S1 need to be normalized and cropped randomly in a certain direction according to a preset rule. Figure 6 As shown in the figure, to improve the generalization ability of the model, the present invention introduces a structured random cropping strategy during training. Given that the smallest effective spatial unit in the network of the present invention is 64×64 pixels, each 640×512 image is cropped to a fixed resolution of 576×448. Specifically, a total of 64 pixels are cropped in width and height, and the cropped area is randomly assigned to the left or right (for width) and the top or bottom (for height). This enables the model to observe different spatial contexts while maintaining alignment with the network architecture. This approach not only ensures compatibility with the network but also enhances robustness to changes in image boundaries.

[0038] S2, extracts the multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image respectively.

[0039] It should be noted that the first feature and the second feature are multi-scale features of the low-resolution infrared image and the high-resolution visible light image, respectively. Here, the combination of the original low-resolution infrared image and the corresponding high-resolution visible light image is used as input, and a two-branch convolutional neural network is used to extract the multi-scale first feature and the multi-scale second feature respectively. In this embodiment, Figure 2As shown in the figure, the dual-branch convolutional neural network has a symmetrical structure. Each branch is a multi-layer encoding network. The first encoding layer consists of a convolutional layer, and the remaining encoding layers consist of a series of pooling layers and convolutional layers. Each layer of the multi-layer encoding network outputs features corresponding to a scale. For example, each branch of the dual-branch convolutional neural network has a three-layer structure: the first layer is a 3×3 convolutional layer, the second layer is a series of pooling layers and 3×3 convolutional layers, and the third layer is a series of pooling layers and 3×3 convolutional layers. The number of feature channels in the three layers is 64, 128, and 256, respectively, which are used for subsequent multi-layer fusion.

[0040] S3, a cross-modal feature fusion module is used to perform bidirectional guidance on each pair of first features and second features of the same scale to generate multi-scale fusion features.

[0041] like Figure 2 and Figure 3 As shown, the cross-modal feature fusion module consists of a first network, a second network, and a third network connected in series; the first network is a symmetrical dual-branch structure with dual input and dual output obtained by stacking one or more moving window self-attention blocks; the second network is a cross-branch structure with dual input and dual output obtained by stacking one or more moving window cross-attention blocks, where the dual output results are first added in the channel dimension and then halved before being input into the third network; the third network is a single-branch structure obtained by stacking one or more moving window self-attention blocks. It should be noted that the moving window cross-attention block contains a cross-attention block formed by bidirectional cross-attention of the self-attention blocks, and is not a traditional unidirectional guided structure.

[0042] In a specific implementation of the present invention, the cross-modal feature fusion modules for bidirectionally guiding the first and second features of different scales have the same structure and independent parameters. Also taking three layers as an example, this embodiment designs three cross-modal feature fusion modules.

[0043] For example, Figure 3As shown, in the cross-modal feature fusion module, the first network is a symmetrical dual-branch structure with dual inputs and dual outputs, formed by stacking N moving window self-attention blocks. The first branch inputs the infrared feature map, and the second branch inputs the visible light feature map. The moving window self-attention block in each branch is a single-input, single-output structure that processes the feature maps of the two modalities independently and outputs the processed infrared feature map and visible light feature map. The second network is a dual-input, dual-output, cross-branch structure formed by a moving window cross-attention block. Its dual input features are the processed infrared feature map and visible light feature map output by the first network, respectively. The bimodal feature map is bidirectionally cross-guided in the moving window cross-attention block, and the processed infrared feature map and visible light feature map are output. The output bimodal feature map is summed in the channel dimension and then input into the third network. The third network is a single-branch structure composed of M moving window self-attention blocks. Its input is the infrared feature map and visible light feature map summed in the channel dimension, and it outputs the fused feature map. In this embodiment, N and M are set to 3 and 4 respectively; the three cross-modal feature fusion modules are used to fuse infrared feature maps and visible light feature maps of different scales and generate fusion results of corresponding scales.

[0044] S4, decode and reconstruct the multi-scale fusion features in order from small to large scale to obtain a high-resolution thermal infrared image.

[0045] For example, taking three layers as an example, this embodiment designs a three-layer decoding, in which the multi-scale fusion features are decoded and reconstructed in order from small to large scale. The last decoding layer includes a convolution layer, and the remaining decoding layers include a series of convolution layers and upsampling layers. In the decoding and reconstruction process of each layer, the input smaller-scale fusion features are reconstructed to the same spatial resolution as the larger-scale fusion features, and added to the larger-scale fusion features in the channel dimension as the decoding and reconstruction result of this layer; the decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added to the original low-resolution infrared image in the channel dimension as the final high-resolution thermal infrared image.

[0046] like Figure 2 As shown, the first decoding layer and the second decoding layer include a 3×3 convolution layer and an upsampling layer, and the third decoding layer includes a 3×3 convolution layer.

[0047] S5, the model is trained using the loss between the reconstructed high-resolution thermal infrared image and the real high-resolution thermal infrared image.

[0048] In order to enhance the pixel accuracy, structural consistency and edge sharpness of the generated image, this embodiment adopts a composite loss function that combines three complementary components: mean square error (MSE) loss, structural similarity (SSIM) loss and image gradient loss. SSIM loss is used to improve the structural integrity of the reconstructed image. It evaluates local structural similarity in terms of brightness, contrast and texture, which helps to improve the perceived quality. In order to preserve edge sharpness and fine details, gradient loss is calculated by measuring the image gradient difference between the generated image and the real image in the horizontal and vertical directions, guiding the network to restore sharper boundaries and contours. Finally, combined with the mean square error loss , structural similarity loss and image gradient loss , the weighted result of the three losses is used as the final loss ; In this embodiment, the three weights are The MSE loss measures the direct difference between the generated super-resolution thermal infrared image and the real high-resolution thermal infrared image, which effectively encourages both overall brightness and pixel-level accuracy.

[0049] S6, under the guidance of high-resolution visible light images, uses the trained model to reconstruct low-resolution thermal infrared images into high-resolution thermal infrared images.

[0050] The model architecture employed in this paper is a U-shaped encoder-decoder structure consisting of multiple layers of downsampling and upsampling stages. In the encoder stage, two parallel convolutional neural network branches extract local features from thermal infrared and visible light images, respectively. Each branch generates multi-scale feature maps with 64, 128, and 256 channels, capturing hierarchical texture and structural information. These multi-level features are then fed into a cross-modal fusion module, which first uses a series of self-attention modules to extract deep features. A cross-attention module is then introduced to enable effective interaction between thermal infrared and visible light features. Additional self-attention modules are then used to refine and reconstruct the fused representation. Residual connections are applied throughout this process to stabilize training and enhance feature learning. The multi-layered cross-modal feature fusion module reduces the number of computational layers, improving training efficiency and reducing memory usage. In the decoder stage, a series of upsampling and convolutional layers gradually restore the spatial resolution of the feature maps. At each scale, the upsampled features are concatenated with the corresponding feature map from the cross-modal fusion module and then refined by convolutional layers to enhance structural detail and visual quality. The output of the last convolutional layer passes through a tanh activation function and is added to the original low-resolution thermal infrared image through a residual connection to generate the final high-resolution output.

[0051] In one embodiment of the present invention, the moving window self-attention block is obtained by connecting alternating self-attention blocks and moving windows in series. Figure 4 As shown, the moving window self-attention block consists of two self-attention blocks, and a window offset is applied after each self-attention block to achieve cross-window interaction (the entire feature map is moved by half the window size to enable cross-window interaction between adjacent regions). Within each self-attention block, it is applied to non-overlapping windows to model spatial dependencies within the image, as shown below:

[0052]

[0053]

[0054] Where B is the learnable relative position bias, is the projection matrix, is the input feature of the self-attention block, the superscript T is the transpose, is the feature dimension, They are query vector, key vector and value vector respectively.

[0055] The entire calculation process of the self-attention block is as follows:

[0056]

[0057]

[0058] Among them, LN is layer normalization and FFN is a feedforward network, which is used to further refine features.

[0059] like Figure 5 As shown, the moving window cross-attention block is obtained by alternating cross-self-attention blocks and dual-channel moving windows in series. The cross-self-attention block is a cross-branch structure formed by a first self-attention block and a second self-attention block that cross in both directions. The two self-attention blocks receive thermal infrared features and visible light features respectively to calculate the query vector, key vector, and value vector. The query vector based on thermal infrared features is swapped with the query vector based on visible light features, and the multi-head attention output is calculated respectively. The multi-head attention outputs of the two branches are swapped and added to the input features received by each branch before layer normalization. The normalized results are then passed through a feedforward neural network, residual connection, and layer normalization to obtain the final output.

[0060] In the moving window cross-attention block, bidirectional cross-attention is calculated between corresponding windows of the two modalities. In the first direction, thermal infrared features are used as queries and visible light features as keys and values to update the thermal infrared image features using fine-grained visual cues. At the same time, in the second direction, visible light features are used as queries and thermal infrared features as keys and values to inject thermal infrared context into the visible light branch. The updated features are added back to their respective branches through residual connections, so that each modality can be enhanced by the other while maintaining their structural independence for subsequent processing. The entire computation process of the cross-attention block is as follows:

[0061]

[0062]

[0063]

[0064] in, 、 They are the thermal infrared features and visible light features received by the cross attention block, They are respectively the query vector obtained based on thermal infrared features, and the key and value vectors obtained based on visible light features. 、 These are the thermal infrared and visible light features output by the cross-attention block, respectively. In the cross-attention block, the attention result is added to the original input via a residual connection, and the combined output is normalized by the layer. This residual-normalized representation preserves the original modal structure while incorporating cross-modal information from the other modality.

[0065] After each branch incorporates complementary information via a moving window cross-attention block, the updated feature maps from the two modalities are concatenated along the channel dimension. To reduce the added number of channels, a simple 3×3 convolutional layer is applied to halve the number of channels for computational efficiency. The fused feature maps are then fed into a series of moving window self-attention modules to reconstruct and refine the fused multimodal representation. This process enables the network to leverage cross-modal context while maintaining a unified high-level semantic understanding of the image. In the self-attention block, the present invention modifies the placement of the layer normalization (LN) layer after the residual connection to ensure consistent feature distribution during propagation. Features in the main feature stream are normalized after each attention and feed-forward layer; this adjustment results in more stable training, especially for deeper networks.

[0066] To validate the effectiveness of our method, experiments were conducted on the VGTSR dataset, which contains 1025 image pairs. Each pair consists of six images: a high-resolution thermal infrared (TIR) image, a high-resolution visible light image, two TIR images downsampled by a factor of 4 (one using Gaussian blur and one using bicubic interpolation), and two TIR images downsampled by a factor of 8 (blurred and interpolated). The high-resolution images have a resolution of 640 × 512. The dataset was divided into 800 groups for training. After 300 rounds of training, the remaining 225 images were used to validate the model's performance.

[0067] This paper introduces a novel training strategy that uses cropped full-size images for training. That is, each input image is cropped to 64 pixels, and the cropping position is randomly selected from the left or right edge or the top or bottom edge of the image, so that the cropped image size is changed from the original 640×512 to 576×448, ensuring that it can be aligned with the network window partition structure.

[0068] In the 4x and 8x super-resolution scenarios, two different downsampling methods were used to create low-resolution infrared images as input, and the following experimental data was obtained.

[0069] Table 1: 4x super-resolution experimental results

[0070]

[0071] Table 2: 8x super-resolution experimental results

[0072]

[0073] SwinIR is a Transformer-based single-image super-resolution method without visible light input as a guide; UGSR is a visible light-guided super-resolution method based on a convolutional neural network and an encoder-decoder architecture; MGNet uses a convolutional neural network to perform three different transformations on the visible light input, and then uses a unidirectionally guided cross-attention module for feature fusion; CENet / SwinFuSR uses multiple alternating self-attention and cross-attention modules for feature fusion to reconstruct high-resolution infrared images, requiring significantly higher computing power and video memory than the present invention. The Transformer modules in the above methods are all connected in series, and do not contain multiple parallel Transformer branches.

[0074] Two common evaluation metrics are used: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). Higher PSNR and SSIM values are preferred. As shown in Tables 1 and 2, our method surpasses other methods in both PSNR and SSIM, demonstrating its ability to restore image details and preserve image structure.

[0075] In addition to achieving excellent quantitative results, the present invention also exhibits visually better perceptual quality, e.g. Figure 7 As shown in the figure, the first column is the full high-resolution thermal infrared (TIR) image GT(full), the second column is the low-resolution result of the local subimage of GT(full), that is, the original low-resolution infrared image LR, the third column is the high-resolution infrared image reconstructed by the present invention, the fourth column is the ground truth image GT of the local subimage in the fourth column, and the fifth column is the high-resolution visible light image RGB corresponding to the low-resolution result of the local subimage. It can be seen that the reconstruction result of the present invention preserves fine structural details and restores high-frequency texture, highlighting the effectiveness of the multi-scale cross-modal fusion and layered reconstruction strategy of the present invention.

[0076] Based on the same inventive concept, this embodiment also provides a visible light guided infrared image super-resolution system based on multi-level attention, including:

[0077] A dual-modal image input module, which is used for users to input low-resolution infrared images and high-resolution visible light images and pre-process them;

[0078] A dual-branch encoding module extracts multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image respectively;

[0079] A cross-modal feature fusion module is used to bidirectionally guide each pair of first and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network, and a third network connected in series; the first network is a symmetrical dual-branch structure with dual input and dual output obtained by stacking one or more moving window self-attention blocks; the second network is a cross-branch structure with dual input and dual output obtained by stacking one or more moving window cross-attention blocks, and the dual output results are first added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more moving window self-attention blocks; the moving window cross-attention block contains a cross-attention block formed by bidirectional cross-attention of the self-attention blocks;

[0080] A single-branch decoding module is used to decode and reconstruct multi-scale fusion features in order from small to large scale to obtain a high-resolution thermal infrared image. In the decoding and reconstruction process of each layer, the input smaller-scale fusion features are reconstructed to the same spatial resolution as the larger-scale fusion features, and added to the larger-scale fusion features in the channel dimension as the decoding and reconstruction result of this layer. The decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added to the original low-resolution infrared image in the channel dimension as the final high-resolution thermal infrared image.

[0081] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0082] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0083] It should also be noted that the visible light guided infrared image super-resolution method based on multi-level attention in the above embodiment can essentially be executed by a computer program. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiment, which includes a memory and a processor;

[0084] The memory is used to store computer programs;

[0085] The processor is configured to implement a visible light guided infrared image super-resolution method based on multi-level attention in the above embodiment when executing the computer program.

[0086] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) to execute all or part of the steps of the methods described in various embodiments of the present invention.

[0087] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiment, and a computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement a visible light-guided infrared image super-resolution method based on multi-level attention in the above embodiment.

[0088] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a removable hard drive, a magnetic disk, or an optical disk.

[0089] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0090] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they are not to be construed as limiting the scope of the present invention. Persons skilled in the art will readily appreciate that variations and modifications may be made without departing from the spirit of the present invention, and all such variations and modifications fall within the scope of protection of the present invention.

Claims

1. A visible light guided infrared image super-resolution method based on multi-level attention, characterized by: include: Extracting multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image respectively; A cross-modal feature fusion module is used to perform bidirectional guidance on each pair of first and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network, and a third network connected in series; the first network is a symmetrical dual-branch structure with dual input and dual output obtained by stacking one or more moving window self-attention blocks; the second network is a cross-branch structure with dual input and dual output obtained by stacking one or more moving window cross-attention blocks, and the dual output results are first added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more moving window self-attention blocks; the moving window cross-attention block contains a cross-attention block formed by bidirectional cross-attention of the self-attention blocks; The multi-scale fusion features are decoded and reconstructed in order from small to large scale to obtain a high-resolution thermal infrared image; in the decoding and reconstruction process of each layer, the input smaller-scale fusion features are reconstructed to the same spatial resolution as the larger-scale fusion features, and added to the larger-scale fusion features in the channel dimension as the decoding and reconstruction result of this layer; the decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added to the original low-resolution infrared image in the channel dimension as the final high-resolution thermal infrared image.

2. The visible light guided infrared image super-resolution method based on multi-level attention according to claim 1 is characterized in that: Taking the combination of the original low-resolution infrared image and the corresponding high-resolution visible light image as input, a two-branch convolutional neural network is used to extract multi-scale first features and multi-scale second features respectively; The dual-branch convolutional neural network has a symmetrical structure. Each branch is a multi-level coding network. The first coding layer contains a convolutional layer, and the remaining coding layers contain a series of pooling layers and convolutional layers. The output of each layer of the multi-level coding network corresponds to a feature of a scale.

3. The visible light guided infrared image super-resolution method based on multi-level attention according to claim 1 is characterized in that: A multi-level decoding network is used to decode and reconstruct the multi-scale fusion features in order from small to large scale. The last decoding layer contains a convolutional layer, and the remaining decoding layers contain concatenated convolutional layers and upsampling layers.

4. The visible light guided infrared image super-resolution method based on multi-level attention according to claim 1 is characterized in that: The cross-modal feature fusion module for bidirectionally guiding the first features and second features of different scales has the same structure but independent parameters.

5. The visible light guided infrared image super-resolution method based on multi-level attention according to claim 1 is characterized in that: The moving window self-attention block is obtained by connecting alternating self-attention blocks and moving windows in series, and the moving window cross-attention block is obtained by connecting alternating cross-self-attention blocks and dual-channel moving windows in series.

6. The visible light guided infrared image super-resolution method based on multi-level attention according to claim 5 is characterized in that: The cross self-attention block is a cross dual-branch structure formed by a first self-attention block and a second self-attention block that cross in both directions. The two self-attention blocks receive thermal infrared features and visible light features respectively to calculate the query vector, key vector and value vector. The query vector based on the thermal infrared feature and the query vector based on the visible light feature are exchanged and the multi-head attention outputs are calculated respectively. The multi-head attention outputs of the two branches are exchanged and added to the input features received by each branch and then layer normalized. The normalized results are then passed through a feedforward neural network, residual connection, and layer normalization to obtain the final output.

7. The visible light guided infrared image super-resolution method based on multi-level attention according to claim 1, characterized in that: The method also includes a step of cropping the input original low-resolution infrared image and high-resolution visible light image, wherein the size of the cropped image is a maximum multiple of the minimum acceptable image size unit and is smaller than the original size.

8. A visible light guided infrared image super-resolution system based on multi-level attention, characterized in that: include: A dual-modal image input module, which is used for users to input low-resolution infrared images and high-resolution visible light images and pre-process them; A dual-branch encoding module extracts multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image respectively; A cross-modal feature fusion module is used to bidirectionally guide each pair of first and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network, and a third network connected in series; the first network is a symmetrical dual-branch structure with dual input and dual output obtained by stacking one or more moving window self-attention blocks; the second network is a cross-branch structure with dual input and dual output obtained by stacking one or more moving window cross-attention blocks, and the dual output results are first added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more moving window self-attention blocks; the moving window cross-attention block contains a cross-attention block formed by bidirectional cross-attention of the self-attention blocks; A single-branch decoding module is used to decode and reconstruct multi-scale fusion features in order from small to large scale to obtain a high-resolution thermal infrared image. In the decoding and reconstruction process of each layer, the input smaller-scale fusion features are reconstructed to the same spatial resolution as the larger-scale fusion features, and added to the larger-scale fusion features in the channel dimension as the decoding and reconstruction result of this layer. The decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added to the original low-resolution infrared image in the channel dimension as the final high-resolution thermal infrared image.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the visible light guided infrared image super-resolution method based on multi-level attention as described in any one of claims 1 to 7 is implemented.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the visible light guided infrared image super-resolution method based on multi-level attention as described in any one of claims 1 to 7 when executing the computer program.

Images