Multilevel attention visible light guided infrared image super-resolution method and system

Abstract

The application discloses a kind of visible light guided infrared image super-resolution method and system based on multi-level attention, belong to image processing and deep learning field.Extract the multiscale first feature and multiscale second feature of original low-resolution infrared image and high-resolution visible light image respectively;Cross-modal feature fusion module is used to bidirectional guide each pair of first feature and second feature of same scale, generate multiscale fusion feature;The multiscale fusion feature is decoded and reconstructed in order according to scale from small to large, and high-resolution thermal infrared image is obtained.The multi-level cross-modal feature fusion module of the application adopts series three network structure, bidirectional guide is realized through the synergistic effect of moving window self-attention block and cross-attention block, and the interaction and fusion of deep information between modes are enhanced.The super-resolution reconstruction performance of the application is better than existing method, and there is obvious improvement in SSIM, PSNR and other indicators.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and deep learning, in particular to a multi-level attention visible light guided infrared image super-resolution method and system. BACKGROUND

[0002] Thermal infrared images can image in the absence of visible light, with good anti-interference and target recognition ability. However, due to the high manufacturing cost of thermal infrared image sensors and the limited resolution, the images collected often have problems such as low resolution, blurred edges, and loss of details, which seriously affect the accuracy and stability of downstream target detection and recognition tasks. In contrast, visible light images are easy to obtain, have high resolution, and are rich in texture information, so in recent years there have been a large number of studies trying to use high-resolution visible light images as guidance to help improve the resolution of thermal infrared images. This kind of task is usually referred to as guided super-resolution, and the core challenge is how to effectively extract and fuse the structure, edge and semantic information between the two images under the condition of large modal difference, and then restore a clearer and more realistic high-resolution infrared image.

[0003] Existing guided image super-resolution techniques can be mainly divided into two categories. One class relies heavily on shallow feature mapping methods such as bilateral filtering and dictionary learning, and has limited fusion capability, making it difficult to capture complex cross-modal relationships. The other class is deep learning methods based on convolutional neural networks (CNN) or Transformer architecture. Pure CNN structures still have limitations in modeling long-distance dependencies and maintaining structural consistency, and the mainstream models based on Transformer are all in series architecture, which has high computational cost and large parameter quantity, and directly applying them to cross-modal tasks can easily cause modal confusion.

[0004] Therefore, there is an urgent need for an infrared image super-resolution reconstruction method that has high fusion efficiency and strong cross-modal modeling capability to better meet the dual requirements of image resolution and detail restoration capability in practical applications. SUMMARY

[0005] The present application aims to overcome the defects of existing visible light guided thermal infrared image super-resolution methods in terms of insufficient cross-modal information fusion, low fusion efficiency, and insufficient image detail restoration capability, and to provide a multi-level attention visible light guided infrared image super-resolution method and system that can efficiently interact with deep features while maintaining modal structural independence and effectively improving the edge sharpness and structural restoration capability of the reconstructed image.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a visible light guided infrared image super-resolution method based on multi-level attention, comprising:

[0008] Multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image are extracted respectively;

[0009] A cross-modal feature fusion module is used to bidirectionally guide each pair of first features and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network and a third network connected in series; the first network is a symmetric double-branch structure with double inputs and double outputs obtained by stacking one or more (preferably 2-4) mobile window self-attention blocks; the second network is a cross double-branch structure with double inputs and double outputs obtained by stacking one or more (preferably 1) mobile window cross-attention blocks, and the double output results are added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more (preferably 2-4) mobile window self-attention blocks; the mobile window cross-attention block contains a cross-attention block formed by bidirectional cross of self-attention blocks;

[0010] The multi-scale fusion features are sequentially decoded and reconstructed in order of scale from small to large to obtain a high-resolution thermal infrared image; in each layer of decoding and reconstruction, the input fusion features of a smaller scale are reconstructed to the same spatial resolution as the fusion features of a larger scale, and are added in the channel dimension with the fusion features of the larger scale as the decoding and reconstruction result of the layer; the decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added in the channel dimension with the original low-resolution infrared image as the final high-resolution thermal infrared image.

[0011] Further, the combination of the original low-resolution infrared image and the corresponding high-resolution visible light image is used as input, and a double-branch convolutional neural network is used to extract multi-scale first features and multi-scale second features respectively;

[0012] The double-branch convolutional neural network is a symmetric structure, and each branch is a multi-level encoding network; the first encoding layer contains a convolutional layer, and the remaining encoding layers contain a pooling layer and a convolutional layer connected in series; each layer output of the multi-level encoding network corresponds to a scale of features.

[0013] Further, a multi-level decoding network is used to sequentially decode and reconstruct the multi-scale fusion features in order of scale from small to large, and the last decoding layer contains a convolutional layer, and the remaining decoding layers contain a convolutional layer and an up-sampling layer connected in series.

[0014] Further, the cross-modal feature fusion modules used for bidirectionally guiding first features and second features of different scales have the same structure and independent parameters.

[0015] Further, the mobile window self-attention block is obtained by connecting the self-attention block and the mobile window alternately, and the mobile window cross-attention block is obtained by connecting the cross self-attention block and the double-channel mobile window alternately.

[0016] Further, the cross self-attention block is a cross double-branch structure formed by a bidirectional cross first self-attention block and a second self-attention block, the two self-attention blocks respectively receive the thermal infrared feature and the visible light feature to calculate the query vector, the key vector and the value vector, the query vector based on the thermal infrared feature is exchanged with the query vector based on the visible light feature to calculate the multi-head attention output, the multi-head attention outputs of the two branches are exchanged and added with the input features received respectively, and then the layer normalization is performed, and the normalized results are respectively passed through the feedforward neural network, the residual connection and the layer normalization to obtain the final output.

[0017] Further, the method further includes the steps of cropping the input original low-resolution infrared image and the high-resolution visible light image, and the size of the cropped image is the maximum multiple of the minimum acceptable image size unit.

[0018] In a second aspect, the present application provides a visible light guided infrared image super-resolution system based on multi-level attention, which is used to realize the above-mentioned visible light guided infrared image super-resolution method based on multi-level attention.

[0019] In a third aspect, the present application provides a computer readable storage medium, and the storage medium stores a computer program, when the computer program is executed by a processor, the above-mentioned visible light guided infrared image super-resolution method based on multi-level attention is realized.

[0020] In a fourth aspect, the present application provides a computer electronic device, which includes a memory and a processor.

[0021] The memory is used to store a computer program.

[0022] The processor is used to realize the above-mentioned visible light guided infrared image super-resolution method based on multi-level attention when the computer program is executed.

[0023] Compared with the prior art, the present application has the following advantages:

[0024] The application introduces a double-branch multi-scale feature extraction structure, realizes comprehensive expression of internal and external features of modes, and improves the accuracy of cross-modal feature alignment; a multi-level cross-modal feature fusion module is designed, which adopts a series three-network structure, realizes bidirectional guidance through the synergistic effect of the moving window self-attention block and the cross-attention block, and enhances the interaction and fusion of deep information between modes. Specifically, the double-branch self-attention structure of the first network independently extracts deep features of two modes, the cross double-branch structure of the second network realizes bidirectional information interaction through the query vector exchange mechanism (interchanges the query vector generated by the thermal infrared feature and the query vector generated by the visible light feature), and the third network finally integrates and optimizes the fusion features. This hierarchical fusion method effectively solves the problems of mode confusion and insufficient long-distance dependence modeling in traditional methods, and lays a foundation for high-quality reconstruction. In the decoding reconstruction stage, the fusion features are processed in a bottom-up manner, the low-level features provide fine structure clues, and the high-level features carry semantic information. Finally, the decoding result and the original low-resolution infrared image are added in the channel, which not only retains the original radiation characteristics, but also injects the details of the reconstructed information. The experimental results show that the super-resolution reconstruction performance of the application is better than that of the existing methods, and has obvious improvement in SSIM, PSNR and other indicators. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The flowchart of the multi-level attention visible light guided infrared image super-resolution method proposed by the application;

[0026] Figure 2 The model block diagram in the embodiment of the application;

[0027] Figure 3 The structure block diagram of the cross-modal fusion module;

[0028] Figure 4 The structure block diagram of the moving window self-attention module;

[0029] Figure 5 The structure block diagram of the moving window cross self-attention module;

[0030] Figure 6 The schematic diagram of the random cropping strategy;

[0031] Figure 7 The effect diagram of the application. DETAILED DESCRIPTION

[0032] The application will be further described and explained in conjunction with specific embodiments. The embodiments are only exemplary and do not circumscribe the scope of the disclosure. The technical features of each embodiment in the application can be combined accordingly without conflict.

[0033] The accompanying drawings are merely illustrative and are not necessarily drawn to scale. Some of the blocks in the drawings are functional blocks, which can be implemented in software, hardware, or a combination of the two. Some of the blocks in the drawings can also be implemented in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the drawings are merely illustrative and do not necessarily include all steps. For example, some steps can be further divided, and some steps can be combined or partially combined, so the actual execution order can be changed according to the actual situation.

[0035] As Figure 1 shown, the multi-level attention visible light guided infrared image super-resolution method proposed by the present application gives a low-resolution thermal infrared image and a high-resolution visible light image, and the goal is to reconstruct a super-resolution thermal infrared image. Mainly includes the following steps:

[0036] S1, obtaining a pair of registered input images, including an original low-resolution infrared image and a high-resolution visible light image.

[0037] Exemplarily, the pair of registered input images obtained in step S1 needs to be normalized and randomly cropped according to a predetermined rule. As Figure 6 shown, in order to improve the generalization ability of the model, a structured random cropping strategy is introduced in the training process. Since the smallest effective spatial unit in the network of the present application is 64x64 pixels, each 640x512 image is cropped to a fixed resolution of 576x448. Specifically, a total of 64 pixels are cropped in width and height, and the cropped area is randomly allocated to the left or right (for width) and top or bottom (for height). This allows the model to observe different spatial contexts while maintaining alignment with the network architecture. This method not only ensures compatibility with the network, but also enhances robustness to changes in image boundaries.

[0038] S2, respectively extracting multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image.

[0039] It should be noted that the first feature and the second feature are respectively the multi-scale features of the low-resolution infrared image and the high-resolution visible light image. Here, the combination of the original low-resolution infrared image and the corresponding high-resolution visible light image is taken as input, and a double-branch convolutional neural network is used to extract the multi-scale first feature and the multi-scale second feature. In this embodiment, as Figure 2As shown, the double-branch convolutional neural network is a symmetric structure, each branch is a multi-level encoding network, the first encoding layer contains a convolutional layer, and the remaining encoding layers contain a cascaded pooling layer and a convolutional layer; each layer of the multi-level encoding network outputs a feature corresponding to a scale. For example, each branch of the double-branch convolutional neural network has a three-layer structure, the first layer is a 3x3 convolutional layer, the second layer is a cascaded pooling layer and a 3x3 convolutional layer, and the third layer is a cascaded pooling layer and a 3x3 convolutional layer, the number of feature channels of the three layers is 64, 128 and 256 in turn, which is used for subsequent multi-level fusion.

[0040] S3, using a cross-modal feature fusion module to bidirectionally guide each pair of first features and second features of the same scale to generate multi-scale fusion features.

[0041] As shown in Figure 2 and Figure 3 As shown, the cross-modal feature fusion module is composed of a cascaded first network, a second network and a third network; the first network is a symmetric double-branch structure with double inputs and double outputs obtained by stacking one or more mobile window self-attention blocks; the second network is a cross double-branch structure with double inputs and double outputs obtained by stacking one or more mobile window cross-attention blocks, the double output results are first added in the channel dimension and then halved, and then input into the third network; the third network is a single-branch structure obtained by stacking one or more mobile window self-attention blocks. It should be noted that the mobile window cross-attention block contains a cross-attention block formed by bidirectional cross of self-attention blocks, which is not a traditional one-way guiding structure.

[0042] In one specific implementation of the present application, the cross-modal feature fusion module for bidirectionally guiding first features and second features of different scales has the same structure and independent parameters. Also taking three layers as an example, three cross-modal feature fusion modules are designed in this embodiment.

[0043] Exemplarily, as shown in Figure 3As shown, in the cross-modal feature fusion module, the first network is formed by stacking N moving window self-attention blocks, which is a symmetrical dual-branch structure with dual inputs and dual outputs. The first branch takes an infrared feature map as input, and the second branch takes a visible light feature map as input. The moving window self-attention block in each branch is a single-input, single-output structure, processing the feature maps of the two modalities separately and outputting the processed infrared and visible light feature maps. The second network is a dual-input, dual-output, cross-branch structure formed by a moving window cross-attention block. Its dual input features are the processed infrared and visible light feature maps output by the first network, respectively. The dual-modal feature maps are bidirectionally cross-guided in the moving window cross-attention block, outputting the processed infrared and visible light feature maps. The output dual-modal feature maps are summed in the channel dimension before being input into the third network. The third network is a single-branch structure formed by M moving window self-attention blocks. Its inputs are the infrared and visible light feature maps summed in the channel dimension, and its output is the fused feature map. In this embodiment, N and M are set to 3 and 4 respectively; the three cross-modal feature fusion modules are used to fuse infrared feature maps and visible light feature maps of different scales and generate fusion results of corresponding scales.

[0044] S4 decodes and reconstructs the multi-scale fusion features sequentially in ascending order of scale to obtain a high-resolution thermal infrared image.

[0045] For example, also using a three-layer approach, this embodiment designs a three-layer decoding method. Multi-scale fusion features are decoded and reconstructed sequentially in ascending order of scale. The last decoding layer contains a convolutional layer, and the remaining decoding layers contain cascaded convolutional layers and upsampling layers. During each layer's decoding and reconstruction process, the smaller-scale fusion features are reconstructed to the same spatial resolution as the larger-scale fusion features, and then added to the larger-scale fusion features in the channel dimension to obtain the decoding and reconstruction result for that layer. The decoding and reconstruction result of the previous layer serves as the input for the next layer. The final decoding and reconstruction result is added to the original low-resolution infrared image in the channel dimension to obtain the final high-resolution thermal infrared image.

[0046] like Figure 2 As shown, the first and second decoding layers contain 3×3 convolutional layers and upsampling layers, and the third decoding layer contains 3×3 convolutional layers.

[0047] S5 uses the loss between the reconstructed high-resolution thermal infrared image and the real high-resolution thermal infrared image to train the model.

[0048] To enhance the pixel accuracy, structural consistency and edge sharpness of the generated image, the embodiment adopts a composite loss function, which combines three complementary components: mean square error (MSE) loss, structural similarity (SSIM) loss and image gradient loss. The SSIM loss is used to improve the structural integrity of the reconstructed image, which evaluates the local structural similarity in terms of brightness, contrast and texture, and helps to improve the perceptual quality. The gradient loss is used to preserve the edge sharpness and fine details, which calculates the gradient loss by measuring the difference in image gradient between the generated image and the real image in the horizontal and vertical directions, guiding the network to restore sharper boundaries and contours. Finally, the MSE loss 、SSIM loss and image gradient loss are combined to obtain the final loss , and the three weights in the embodiment are . The MSE loss measures the direct difference between the generated super-resolution thermal infrared image and the real high-resolution thermal infrared image, which effectively encourages overall brightness and pixel-level accuracy.

[0049] S6, under the guidance of the high-resolution visible light image, the low-resolution thermal infrared image is reconstructed into a high-resolution thermal infrared image by using the trained model.

[0050] The model architecture adopted by the present application is a U-shaped encoder-decoder structure, which includes multiple levels of downsampling and upsampling stages. In the encoder stage, two parallel convolutional neural network branches are used to extract local features from thermal infrared and visible light images respectively. Each branch generates multi-scale feature maps with 64, 128 and 256 channels, capturing hierarchical texture and structural information. These multi-level features are then sent to a cross-modal fusion module. In this module, a series of self-attention modules are first used to extract deep features, followed by the introduction of cross-attention modules to achieve effective interaction between thermal infrared and visible light features. Then, additional self-attention modules are used to refine and reconstruct the fused representation. Throughout the process, residual connections are applied to stabilize training and enhance feature learning. The structural design of the multi-level cross-modal feature fusion module helps to reduce the number of calculation layers to improve training efficiency and reduce memory occupation. In the decoder stage, a series of upsampling layers and convolutional layers are used to gradually restore the spatial resolution of the feature maps. At each scale, the upsampled features are concatenated with the corresponding feature maps from the cross-modal fusion module, and then refined through convolutional layers to enhance structural details and visual quality. The output of the last convolutional layer is activated by the tanh activation function and added to the original low-resolution thermal infrared image through a residual connection to generate the final high-resolution output.

[0051] In one implementation of the present application, the moving window self-attention block is obtained by concatenating an alternating self-attention block and a moving window. As shown in Figure 4 The moving window self-attention block is composed of two self-attention blocks, and a window offset is applied after each self-attention block to enable cross-window interaction (the entire feature map is moved by half the window size to enable cross-window interaction between adjacent regions). Within each self-attention block, a non-overlapping window is applied to model the spatial dependencies within the image, as shown below:

[0052]

[0053]

[0054] where B is a learnable relative position bias, is a projection matrix, is the input feature of the self-attention block, and the superscript T is the transpose, is the feature dimension, is the query vector, the key vector, and the value vector, respectively.

[0055] The entire calculation process of the self-attention block is as follows:

[0056]

[0057]

[0058] where LN is layer normalization, and FFN is a feedforward network for further refining the features.

[0059] As shown in Figure 5 The moving window cross-attention block is obtained by concatenating an alternating cross self-attention block and a double-channel moving window. The cross self-attention block is a cross double-branch structure formed by a bidirectional cross first self-attention block and a second self-attention block. The two self-attention blocks respectively receive the thermal infrared feature and the visible light feature to calculate the query vector, the key vector, and the value vector. After interchanging the query vector based on the thermal infrared feature with the query vector based on the visible light feature, the two self-attention blocks respectively calculate the multi-head attention output. The multi-head attention outputs of the two branches are interchanged and added to the input features received by each branch, and then subjected to layer normalization. The normalized results are respectively passed through a feedforward neural network, a residual connection, and layer normalization to obtain the final output.

[0060] In the mobile window cross-attention block, bidirectional cross-attention is calculated between corresponding windows of the two modalities. In the first direction, the thermal infrared features serve as queries, and the visible light features serve as keys and values, updating the thermal infrared image features with fine-grained visual cues. Meanwhile, in the second direction, the visible light features serve as queries, and the thermal infrared features serve as keys and values, injecting thermal infrared context into the visible light branch. The updated features are added back to the respective branches through residual connections, allowing each modality to be enhanced by the other while maintaining its structural independence for subsequent processing. The entire calculation process of the cross-attention block is as follows:

[0061]

[0062]

[0063]

[0064] wherein, , are the thermal infrared features and visible light features received by the cross-attention block, are the query vector based on the thermal infrared features, and the key and value vectors based on the visible light features, , are the thermal infrared features and visible light features output by the cross-attention block. In the cross-attention block, the attention results are added to the original input through a residual connection, and the combined output is normalized by a layer normalization. This residual normalized representation preserves the original modality structure while incorporating cross-modality information from the other modality.

[0065] After each branch is fused with complementary information through the mobile window cross-attention block, the updated feature maps from the two modalities are concatenated along the channel dimension. To reduce the increased number of channels, a simple 3x3 convolution layer is applied to halve the number of channels, ensuring computational efficiency. The fused feature maps are then fed into a series of mobile window self-attention modules to reconstruct and refine the fused multi-modal representation. This process enables the network to utilize cross-modality context while maintaining a unified high-level semantic understanding of the image. In the self-attention block, the position of the layer normalization (LN) layer after the residual connection is modified to ensure consistent feature distribution during propagation. The features of the main feature stream are normalized after each attention and feed-forward layer, which brings more stable training, especially for deeper networks.

[0066] To verify the effect of the present application, experiments were conducted on the VGTSR dataset, which contains 1025 groups of image pairs. Each group includes six images: one high-resolution thermal infrared (TIR) image, one high-resolution visible light image, two 4-fold down-sampled TIR images (one using Gaussian blur and one using bicubic interpolation), and two 8-fold down-sampled TIR images (blur and interpolation). The resolution of the high-resolution images is 640x512. The dataset is divided into 800 groups for training, and after 300 rounds of training, the remaining 225 images are used to verify the effect of the model.

[0067] The present application introduces a novel training strategy, which uses cropped full-size images for training, i.e., each input image is cropped to a size of 64 pixels, and the cropping position is randomly selected from the left and right edges or the top and bottom edges of the image, so that the size of the cropped image changes from the original 640x512 to 576x448, ensuring that it can be aligned with the network window partition structure.

[0068] In the 4-fold and 8-fold super-resolution scenarios, two different down-sampling methods are used to produce low-resolution infrared images as input, and the following experimental data is obtained.

[0069] Table 1: 4-fold super-resolution experimental results

[0070]

[0071] Table 2: 8-fold super-resolution experimental results

[0072]

[0073] SwinIR is a single-image super-resolution based on Transformer, without visible light input as a guide; UGSR is a visible light guided super-resolution based on convolutional neural networks and encoder-decoder architecture; MGNet uses convolutional neural networks to perform three different transformations on the visible light input, and then uses a one-way guided cross-attention module for feature fusion; CENet / SwinFuSR uses multiple alternating self-attention and cross-attention modules for feature fusion to reconstruct high-resolution infrared images, requiring significantly more computing power and memory than the present application. The Transformer modules in the above methods are all in series and do not contain multiple parallel Transformer branches.

[0074] Two common evaluation indicators are used: Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). The higher the PSNR and SSIM values, the better. As shown in Tables 1 and 2, the method of the present application outperforms other methods in both peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), demonstrating the ability of the method to restore image details and preserve image structure.

[0075] In addition to achieving excellent quantitative results, the present application also exhibits visually better perceptual quality, as Figure 7 As shown, the first column is the full GT (full) of high-resolution thermal infrared (TIR) images, the second column is the low-resolution result of the local subgraph of GT (full), i.e. the original low-resolution infrared image LR, the third column is the high-resolution infrared image reconstructed by the present application, the fourth column is the local subgraph thereof as the true value graph GT of the fourth column, and the fifth column is the high-resolution visible light image RGB corresponding to the low-resolution result of the local subgraph. It can be seen that the reconstruction result of the present application retains fine structural details and restores high-frequency textures, highlighting the effectiveness of the multi-scale cross-modal fusion and hierarchical reconstruction strategy of the present application.

[0076] Based on the same inventive concept, in the present embodiment, a visible light guided infrared image super-resolution system based on multi-level attention is also provided, comprising:

[0077] A dual-modal image input module for a user to input and pre-process a low-resolution infrared image and a high-resolution visible light image;

[0078] A dual-branch encoding module for extracting multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image, respectively;

[0079] A cross-modal feature fusion module for bidirectional guidance of each pair of first features and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network and a third network in series; the first network is a symmetric dual-branch structure with double inputs and double outputs obtained by stacking one or more mobile window self-attention blocks; the second network is a cross dual-branch structure with double inputs and double outputs obtained by stacking one or more mobile window cross-attention blocks, and the double output results are first added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more mobile window self-attention blocks; the mobile window cross-attention block contains a cross-attention block formed by bidirectional cross of self-attention blocks;

[0080] A single-branch decoding module for sequentially decoding reconstruction of the multi-scale fusion features in order of scale from small to large to obtain a high-resolution thermal infrared image; in each layer of decoding reconstruction, the input fusion features of a smaller scale are reconstructed to the same spatial resolution as the fusion features of a larger scale, and are added in the channel dimension with the fusion features of a larger scale as the decoding reconstruction result of this layer; the decoding reconstruction result of the previous layer is used as the input of the next layer, and the decoding reconstruction result of the last layer is added in the channel dimension with the original low-resolution infrared image as the final high-resolution thermal infrared image.

[0081] For the system embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the method embodiment, and the implementation method of the remaining modules is not described here. The system embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present application. Those skilled in the art can understand and implement without creative labor.

[0082] The embodiment of the system of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiment can be realized by software, hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability.

[0083] In addition, it should be noted that the above-mentioned embodiment of the visible light guided infrared image super-resolution method based on multi-level attention can be essentially executed by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer electronic device corresponding to the method provided by the above-mentioned embodiment, which includes a memory and a processor;

[0084] The memory is used to store a computer program;

[0085] The processor is used to implement the visible light guided infrared image super-resolution method based on multi-level attention in the above-mentioned embodiment when executing the computer program.

[0086] In addition, the logical instructions in the above-mentioned memory can be realized in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application.

[0087] Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer readable storage medium corresponding to the method provided by the above-mentioned embodiment, and the storage medium stores a computer program. When the computer program is executed by a processor, the method for visible light guided infrared image super-resolution based on multi-level attention can be realized.

[0088] It can be understood that the storage medium can include a random access memory (RAM) and a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various storage program codes.

[0089] It can be understood that the processor can be a general processor, including a central processing unit (CPU), a network processor (NP), etc. It can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0090] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the present application. For ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

Claims

1. A method for visible light guided infrared image super-resolution based on multi-level attention, characterized in that, The method comprises the following steps: cropping the input original low-resolution infrared image and high-resolution visible light image, the cropping position being randomly selected from the left and right edges or the upper and lower edges of the image, the size of the cropped image being the maximum multiple of the minimum acceptable image size unit and being smaller than the original size; extracting multi-scale first features and multi-scale second features of the original low-resolution infrared image and the high-resolution visible light image, respectively; using a cross-modal feature fusion module to conduct bidirectional guidance on each pair of first features and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network and a third network connected in series; the first network is a double-input double-output symmetric double-branch structure obtained by stacking one or more mobile window self-attention blocks; the second network is a double-input double-output cross double-branch structure obtained by stacking one or more mobile window cross-attention blocks, and the double-output results are added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more mobile window self-attention blocks; the mobile window cross-attention block is obtained by connecting an alternating cross self-attention block and a double-channel mobile window in series, and the cross self-attention block is formed by bidirectional cross of self-attention blocks; the cross self-attention block is a cross double-branch structure formed by bidirectional cross of a first self-attention block and a second self-attention block, and the two self-attention blocks respectively receive the thermal infrared features and the visible light features to calculate the query vector, the key vector and the value vector, the query vector based on the thermal infrared features is exchanged with the query vector based on the visible light features, and then the multi-head attention output is calculated, the multi-head attention outputs of the two branches are exchanged and added with the input features received by each branch, and then layer normalization is performed, and the normalized results are respectively passed through a feedforward neural network, a residual connection and layer normalization to obtain the final output; decoding and reconstructing the multi-scale fusion features in order of increasing scale to obtain a high-resolution thermal infrared image; in each layer of the decoding and reconstruction process, the input fusion features of a smaller scale are reconstructed to the same spatial resolution as the fusion features of a larger scale, and are added with the fusion features of the larger scale in the channel dimension to serve as the decoding and reconstruction result of the layer; the decoding and reconstruction result of a previous layer serves as the input of a subsequent layer, and the decoding and reconstruction result of the last layer is added with the original low-resolution infrared image in the channel dimension to serve as the final high-resolution thermal infrared image.

2. The multi-level attention based visible light guided infrared image super-resolution method according to claim 1, characterized in that, The combination of the original low-resolution infrared image and the corresponding high-resolution visible light image is input, and a double-branch convolutional neural network is used to extract multi-scale first features and multi-scale second features, respectively. The double-branch convolutional neural network is a symmetric structure, each branch is a multi-level encoding network, the first encoding layer contains a convolutional layer, and the remaining encoding layers contain a pooling layer and a convolutional layer connected in series; each layer of the multi-level encoding network outputs features of a corresponding scale.

3. The multi-level attention based visible light guided infrared image super-resolution method of claim 1, wherein, A multi-level decoding network is used to decode and reconstruct the multi-scale fusion features in order of increasing scale, and the last decoding layer contains a convolutional layer, and the remaining decoding layers contain a convolutional layer and an upsampling layer connected in series.

4. The multi-level attention based visible light guided infrared image super-resolution method of claim 1, wherein, The cross-modal feature fusion module for bidirectional guiding different scales of first features and second features has the same structure but independent parameters.

5. The multi-level attention based visible light guided infrared image super-resolution method of claim 1, wherein, The mobile window self-attention block is obtained by connecting the self-attention blocks alternately and the mobile window cross-attention block is obtained by connecting the cross-attention blocks alternately and the double-channel mobile window.

6. A visible light guided infrared image super-resolution system based on multi-level attention, characterized in that, Comprise: A dual-modal image input module for a user to input and pre-process low-resolution infrared images and high-resolution visible light images; the pre-processing includes cropping the input original low-resolution infrared images and high-resolution visible light images, the cropping position is randomly selected from the left and right edges or the top and bottom edges of the images, and the size of the cropped images is the maximum multiple of the minimum acceptable image size unit and is smaller than the original size; A dual-branch encoding module for extracting multi-scale first features and multi-scale second features of the original low-resolution infrared images and high-resolution visible light images respectively; A cross-modal feature fusion module for bidirectional guiding each pair of first features and second features of the same scale to generate multi-scale fusion features; the cross-modal feature fusion module is composed of a first network, a second network and a third network connected in series; the first network is a symmetric dual-branch structure with double inputs and double outputs obtained by stacking one or more mobile window self-attention blocks; the second network is a cross dual-branch structure with double inputs and double outputs obtained by stacking one or more mobile window cross-attention blocks, and the double output results are added in the channel dimension and then halved; the third network is a single-branch structure obtained by stacking one or more mobile window self-attention blocks; the mobile window cross-attention block is obtained by connecting the cross-attention blocks alternately and the double-channel mobile window; the cross-attention block is a cross dual-branch structure formed by bidirectional cross of the first self-attention block and the second self-attention block; two self-attention blocks respectively receive thermal infrared features and visible light features to calculate query vectors, key vectors and value vectors, and then calculate multi-head attention outputs after interchanging the query vectors based on the thermal infrared features and the query vectors based on the visible light features; the multi-head attention outputs of the two branches are interchanged and added to the input features received respectively, and then layer normalization is performed, and the normalized results are respectively passed through a feedforward neural network, a residual connection and layer normalization to obtain the final output; A single-branch decoding module for decoding and reconstructing the multi-scale fusion features in order from small to large scales to obtain a high-resolution thermal infrared image; in each decoding and reconstruction process, the input fusion features of a smaller scale are reconstructed to the same spatial resolution as the fusion features of a larger scale, and are added to the fusion features of the larger scale in the channel dimension as the decoding and reconstruction result of the layer; the decoding and reconstruction result of the previous layer is used as the input of the next layer, and the decoding and reconstruction result of the last layer is added to the original low-resolution infrared image in the channel dimension as the final high-resolution thermal infrared image.

7. A computer readable storage medium characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, a visible light guided infrared image super-resolution method based on multi-level attention as claimed in any one of claims 1 to 5 is realized.

8. A computer electronic device, comprising: comprising a memory and a processor; The memory is used to store a computer program; The processor is used to realize the visible light guided infrared image super-resolution method based on multi-level attention as claimed in any one of claims 1 to 5 when the computer program is executed.

Images