Unmanned aerial vehicle thermal infrared image super-resolution model and method for guiding decoupling
Through the guided decoupled drone thermal infrared image super-resolution model, multiple attention mechanisms are used to realize the interaction between visible light features and infrared features, solving the problem of thermal infrared image reconstruction under complex scenes and dark light conditions, and achieving more High detection accuracy and detail recovery effect.
Patent Information
- Application Number
- CN202411492639.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2025-02-18
AI Technical Summary
The prior art is difficult to accurately reconstruct fine details and textures in thermal infrared images under complex scenes and dark light conditions, and the existing guided super-resolution methods rely heavily on high-quality optical inputs and lack the ability to adapt to different environments.
A guided decoupling of the thermal infrared image super-resolution model of the UAV is proposed, including the AGM module, the AFM module and the MOGM module. The texture and edge information in the visible light image is extracted through the feature extraction network, combined with overlapping window attention, cross-modal attention and sliding window attention, the full interaction between the visible light features and the infrared features is achieved, and a high-resolution thermal infrared image is generated.
Improve object detection accuracy in complex scenes and dark light conditions, restore richer texture and structural details, and show better evaluation indicators and visual perception effects.
Smart Images
Figure CN119477700A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a guided decoupling unmanned aerial vehicle thermal infrared image super-resolution model and method. Background Art
[0002] UAV thermal imaging technology can capture infrared radiation emitted by objects, and has shown unique application advantages in many fields such as security inspection, agricultural measurement, and target detection. Therefore, it has become a research focus in the field of remote sensing. However, limited by the physical imaging principle and the technical level of airborne thermal imaging equipment, the resolution of thermal infrared images captured directly from the perspective of drones is limited, which restricts the analysis and understanding of thermal infrared images. To this end, it is necessary to study technical methods for generating high-resolution thermal infrared images. Single Image Super-Resolution (SISR), that is, recovering high-resolution (HR) images from low-resolution (LR) images, has broad application prospects and significant practical value.
[0003] Existing single-image super-resolution methods based on deep learning, such as infrared image super-resolution methods combining compressed sensing with deep learning and image restoration models based on Swin Transformer, have solved some limitations of traditional methods, but it is often difficult to accurately reconstruct fine details and textures in thermal images because thermal images themselves lack obvious edge information and texture features. In addition, existing single-image super-resolution methods face major challenges in restoring complex details and generating realistic images, especially in complex high-altitude drone environments, where thermal infrared imaging is interfered by multiple factors in the atmosphere.
[0004] In order to achieve better reconstruction, recent studies have used visible light images to guide thermal infrared image super-resolution. By using a feature extraction network to extract texture and edge information from visible light images to guide low-resolution infrared image reconstruction, the limitations of single image super-resolution are overcome and great potential is shown. However, existing guided super-resolution methods rely heavily on high-quality optical inputs and usually use a single network framework to process all scenes, lacking the ability to adapt to different environments. This can produce excellent results under ideal conditions, but in complex scenes and challenging drone perspectives, such as foggy weather and dark light conditions, the significant degradation of visible light image quality will cause the performance of existing guided super-resolution methods to drop significantly. Summary of the invention
[0005] The technical problem to be solved by the present invention is how to improve the target detection accuracy in complex scenes and dark light conditions.
[0006] The present invention solves the above technical problems through the following technical means:
[0007] Guided decoupled UAV thermal infrared image super-resolution model, including: AGM module, AFM module, MOGM module;
[0008] The visible light image is used as the input of the AGM module, and feature extraction is performed through three branches: normal light scene branch, low light scene branch, and foggy occlusion scene branch to obtain visible light image information under three types of scenes. The visible light image information under the three types of scenes is used as the input of the AFM module. The AFM module selectively activates and utilizes the most relevant branch features according to the input visible light image information, and fuses it with other branch features to obtain the visible light image fusion features under the three types of scenes; the visible light image fusion features and the thermal infrared image are simultaneously used as the input of the MOGM module. In MOGM, overlapping window attention, cross-modal attention and sliding window attention are used to achieve full interaction between visible light features and infrared features, and finally upsampling is performed to generate a high-resolution thermal infrared image.
[0009] Furthermore, the normally illuminated scene branch includes a backbone network and two multi-spectral guidance layers according to the data flow, and enhances the guidance information by combining cross-modal self-attention and shift window mechanism.
[0010] Furthermore, the low-light scene branch includes three parts in sequence according to the data flow: a pre-trained Retinex decomposition network, a backbone network, and four multi-spectral guidance layers. The Retinex decomposition network performs feature enhancement on the image based on the retinal imaging principle.
[0011] Furthermore, the foggy occlusion scene branch includes a backbone network and six gated attention layers in sequence according to the data flow direction; the visible light features under foggy conditions are first compressed along the channel dimension, and then a weight factor is generated for each channel; the feature map is mapped and weighted along the channel direction through the weight factor to enhance the key features.
[0012] Furthermore, the MOGM module includes four RMAG groups according to the data flow, and each RMAG contains an overlapping multi-head cross-modal attention layer, a multi-spectral guidance layer, a Swin Transformer attention layer and an original overlapping Transformer attention layer.
[0013] The present invention also provides a guided decoupling UAV thermal infrared image super-resolution method, comprising the following steps:
[0014] S1. Use the AGM module to extract features from visible light images under different weather and lighting conditions;
[0015] The visible light image is used as the input of the AGM module, and features are extracted through three branches: normal light scene branch, low light scene branch, and foggy occlusion scene branch, to obtain visible light image information under three types of scenes;
[0016] S2. Adaptively integrate visible light image features from different branches through a selection mechanism;
[0017] The visible light image information under the three types of scenes is used as the input of the AFM module. The AFM module selectively activates and utilizes the most relevant branch features according to the input visible light image information, and fuses it with other branch features to obtain the visible light image fusion features under the three types of scenes;
[0018] S3. Use the obtained adaptive aggregation features to guide the infrared features, thereby achieving infrared feature enhancement.
[0019] The visible light image fusion features and the thermal infrared image are simultaneously used as inputs of the MOGM module. In MOGM, overlapping window attention, cross-modal attention and sliding window attention are used to achieve full interaction between visible light features and infrared features.
[0020] S4. Generate a high-resolution thermal infrared image by upsampling the feature map after full interaction in step S3.
[0021] Furthermore, the normally illuminated scene branch includes a backbone network and two multi-spectral guidance layers according to the data flow, and enhances the guidance information by combining cross-modal self-attention and shift window mechanism.
[0022] Furthermore, the low-light scene branch includes three parts in sequence according to the data flow: a pre-trained Retinex decomposition network, a backbone network, and four multi-spectral guidance layers. The Retinex decomposition network performs feature enhancement on the image based on the retinal imaging principle.
[0023] Furthermore, the foggy occlusion scene branch includes a backbone network and six gated attention layers in sequence according to the data flow direction; the visible light features under foggy conditions are first compressed along the channel dimension, and then a weight factor is generated for each channel; the feature map is mapped and weighted along the channel direction through the weight factor to enhance the key features.
[0024] The present invention also provides a training method for a guided decoupled UAV thermal infrared image super-resolution model, which includes three stages:
[0025] Phase 1: Trunk Training
[0026] All VGTSR2.0 training data are used to train feature extraction of optical images under general weather conditions, i.e., the backbone network. In this stage, the three guide branches and the AFM module are removed, and a general feature extractor is trained for all attribute data.
[0027] Stage 2: Training three attribute-specific branches:
[0028] After obtaining the trained feature extractor, three attribute-based branches are trained in sequence, namely the normal lighting scene branch, the low lighting scene branch, and the foggy occlusion scene branch. The corresponding attribute-based training data is used to learn the optical representation of specific attributes, while freezing all other parameters. When training a specific branch, only the data of the corresponding scene is used, and the data of other scenes and branch structures will be excluded from the network.
[0029] Stage 3: Training of AFM module and fine-tuning of MOGM:
[0030] After obtaining the learned feature extractor and three attribute-specific branches, the attribute-aware fusion module is trained using all the data and the MOGM is fine-tuned while freezing all other parameters.
[0031] The advantages of the present invention are:
[0032] 1. The present invention proposes a guided decoupling network GDNet. The network integrates a specific attribute guidance module, and designs different branches for feature processing for images under low light, foggy, and normal light conditions. Ensure effective feature extraction of visible light images in different scenes. The perception attribute fusion module realizes adaptive aggregation of features of the three branches. The multi-scene visible light guidance module uses multiple attention mechanisms to realize the feature interaction of visible light images and thermal infrared images, and realizes efficient guidance of visible light images to thermal infrared images.
[0033] 2. Compared with the existing state-of-the-art single image super-resolution methods and guided super-resolution methods, the present invention restores richer texture and structural details and has better performance in terms of evaluation indicators and visual perception. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a schematic diagram of the process (model structure) of the super-resolution reconstruction method of thermal infrared images of unmanned aerial vehicles guided by decoupling of the present invention;
[0035] Figure 2 This is a schematic diagram of the RMAG structure in an embodiment of the present invention;
[0036] Figure 3 Schematic diagram of the overall structure of the three branches in the training phase in an embodiment of the present invention;
[0037] Figure 4This is a schematic diagram of the GAL structure in an embodiment of the present invention;
[0038] Figure 5 Schematic diagram of the MGL structure in an embodiment of the present invention. DETAILED DESCRIPTION
[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] Example 1
[0041] This embodiment provides a guidance disentanglement network (GDNet) UAV thermal infrared image super-resolution model. Figure 1 As shown, it includes: AGM module (Attribute-specific Guidance Module (AGM)), AFM module (Attribute-aware Fusion Module (AFM)), MOGM module (Multi-scenario Optical Guidance Module (MOGM)); the visible light image is used as the input of the AGM module, and feature extraction is performed through three branches, namely, the normal illumination scene branch (NC), the low illumination scene branch (LI), and the foggy occlusion scene branch (FO), to obtain visible light image information under three types of scenes, and the visible light image information under the three types of scenes is used as the input of the AFM module. The AFM module selectively activates and utilizes the most relevant branch features according to the input visible light image information, and fuses them with other branch features to obtain visible light image fusion features under the three types of scenes; the visible light image fusion features and the thermal infrared image are simultaneously used as the input of the MOGM module. In the MOGM, overlapping window attention, cross-modal attention and sliding window attention are used to realize full interaction between visible light features and infrared features, and finally upsampling is performed to generate a high-resolution thermal infrared image.
[0042] Specifically, the input of GDNet is the visible light image and thermal infrared image from the perspective of the drone. The binocular camera carried on the drone platform simultaneously captures visible light-infrared image pairs at high altitudes. The original visible light image and the downsampled thermal infrared image are used as the input of the network. The drone platform used in this embodiment is Matrice 3TD. The drone under this platform is equipped with an uncooled micro-thermal radiation sensor to obtain reliable thermal imaging. All visible light images will be resized and cropped with reference to the corresponding thermal infrared images, and manual alignment will be performed during the cropping process. In order to meet the needs of multimodal research methods and maintain the consistency of the entire data set.
[0043] First, different branches are used to extract features from visible light images under three different conditions: standard lighting, low light, and fog. In the normal lighting state branch, in order to alleviate the noise problem in thermal infrared images from a high-altitude perspective, this embodiment adopts a cross-modal attention mechanism that integrates thermal infrared features. Under low-light conditions, a pre-trained Retinex decomposition network is used to enhance visible light images affected by low light. In order to better simulate the visual scene from the perspective of a drone in low light, this embodiment trains a feature decomposition network from scratch based on RetinexFormer. Under foggy conditions, in order to enhance clear local details and suppress foggy occlusions in the image, this embodiment uses a gated attention mechanism integrated into the original visual Transformer to achieve effective feature extraction for foggy images.
[0044] After obtaining the visible light image information in different scenes, in order to adaptively aggregate the features from multiple guiding branches, the features of the three branches are input into the AFM module. This module can selectively activate and utilize the most relevant branch features according to the input feature representation, and efficiently fuse them with other branch features to achieve efficient feature extraction of visible light images in different scenes.
[0045] After the AFM module extracts the features of the visible light image information in complex scenes. In order to promote the efficient guidance of the visible light image features to the thermal infrared image features, this embodiment designs a multi-scene visible light guidance module (MOGM). In MOGM, overlapping window attention, cross-modal attention and sliding window attention are used to achieve full interaction between visible light features and infrared features, promote the network to more effectively integrate multimodal information, and finally generate high-resolution thermal infrared images by upsampling the obtained fusion features.
[0046] This embodiment trains the entire network through the L1 pixel loss between the thermal infrared image predicted by the network and the real thermal infrared image. The calculation method of this loss is as follows:
[0047]
[0048] in, is the loss function, and Φ is the parameter set to be optimized in GDNet. Output the jth pixel of the thermal infrared image for GDNet, is the jth pixel corresponding to the original thermal infrared image.
[0049] In this embodiment, the AGM module consists of three branches: normal lighting scene branch (NC), low lighting scene branch (LI), and foggy occlusion scene branch (FO). Under normal lighting conditions, the NC branch consists of a basic CNN backbone network and two multispectral guidance layers (MGL), which enhance the guidance information by combining cross-modal self-attention and shift window mechanism. The visible light image is first subjected to preliminary feature extraction through a basic backbone network composed of multiple 3×3 convolutional layers, and then input into multiple multispectral guidance layers simultaneously with infrared features. In the multispectral guidance layer, cross-modal self-attention and sliding window mechanism (Window Multi-head Cross-Attention, W-MCA) are used to enhance the guidance information of the normal lighting visible light scene image. In the sliding window mechanism, the calculation method of these operations is as follows:
[0050] Q=YW Q ,K=YW K ,V=XW V
[0051] Among them, WQ, WK and WV are learnable projection matrices shared between different windows, X represents the visible light image features from the NC branch input, and Y represents the thermal infrared features. The calculation process of the cross-modal self-attention mechanism in the local window is as follows:
[0052]
[0053] Where B is a learnable relative position encoding matrix, dk is the dimension of the query vector, and the Attention(·) function represents multi-head self-attention and multi-head cross-modal attention. In W-MCA, Q and K come from thermal infrared features and V comes from visible light features. The complete process of MGL is as follows:
[0054] X=W-MCA(LN(X))+X,
[0055] X=MLP(LN(X))+X.
[0056] Among them, W-MCA represents window-based multi-head cross-modal attention, MLP and LN represent multi-layer perceptron and layer normalization, respectively.
[0057] In low-light conditions, visible light image information is significantly reduced, but the low-light image reflectance map contains a large amount of color and texture information that is beneficial to the super-resolution of thermal infrared images. Therefore, this embodiment requires guidance information in low-light scenes. Unlike conventional feature extractors, the LI branch consists of three parts: a pre-trained Retinex decomposition network, a basic CNN backbone, and four multispectral guidance layers. The Retinex decomposition network performs feature enhancement on the image based on the principle of retinal imaging. The present invention can use visible light-thermal infrared image pairs as input to train the Retinex network from scratch, or simply load publicly available pre-trained weights. In order to better simulate the scene from the perspective of a drone under low-light conditions, the present invention trains a Retinex network from scratch based on multimodal image data from the perspective of a drone. The network models the noise in the reflectance and illumination maps through a Transformer feature extractor, thereby providing a more robust feature representation.
[0058] In foggy conditions, the visible light image from the perspective of the drone has blurred local details due to the occlusion of the fog layer. In order to enhance the detail information of the clear part of the image and suppress the local foggy representation, the present invention integrates a gating mechanism in the original Transformer. The foggy occlusion scene branch includes a backbone network and six gated attention layers (GAL). The visible light features under foggy conditions are first compressed along the channel dimension, and then a weight factor is generated for each channel. The feature map is mapped and weighted along the channel direction by the weight factor. To enhance the key features, thereby suppressing invalid information and enhancing valid information. In the entire gated attention layer, the visible light features are input into W-MCA and GAB respectively after layer normalization. The whole process of GAL is as follows:
[0059] X LN =LN(X),
[0060] X G =W-MCA(X LN )+GAB(X LN )+X,
[0061] X = MLP(LN(X G ))+X G .
[0062] Where XLN and XG represent intermediate features. MLP and LN are multi-layer perceptron and layer normalization functions, respectively.
[0063] The multi-scene visible light guidance module (MOGM) includes four residual multi-attention groups (RMAGs), each of which contains four components: overlapping multi-head cross-modal attention layer (OMCL), multi-spectral guidance layer (MGL), Swin Transformer attention layer (STL) and original overlapping Transformer attention layer (OTL). These network structures will be introduced one by one. For OMCL, this network structure is similar to the structure of the traditional Swin Transformer layer. The present invention aggregates an overlapping multi-head cross-modal attention (OMCA) and a multi-layer perceptron (MLP) in it, and uses a variable-size window to capture more dynamic features of the projection matrix. The visible light feature is used as the Query in the attention mechanism, and the infrared feature is used as the Key and Value in the attention mechanism. The window size of the Key and Value is larger than the window size in the conventional attention design to ensure that the windows overlap. The window design of the Query is a regular size to ensure that there is no window overlap. The size of the overlapping window is determined by the following formula:
[0064]
[0065] Where β is the coefficient that determines the degree of window overlap and is set to 0.5. O represents the standard size used when dividing the window size, O t Represents the window size after overlapping division and adjustment. This method expands the connection relationship between windows and ensures that visible light information can query a larger window during the multimodal fusion process. This is conducive to the global fusion of multimodal features.
[0066] For OTL, the attention layer combines self-attention and channel attention to perform feature extraction in a parallel manner, so that global and local information can be captured simultaneously. The multispectral guidance layer (MGL) has been explained in the previous introduction. For STL, unlike MGL, the feature input has changed from multimodal features to single modal input. At the same time, in order to capture long-term dependencies and model global context information, STL uses self-attention and sliding window mechanisms to further enhance the fused information. The whole process in STL is as follows:
[0067] Y=W-MSA(LN(Y))+Y,
[0068] Y=MLP(LN(Y))+Y.
[0069] Among them, W-MSA represents the multi-head self-attention mechanism. MLP and LN represent multi-layer perceptron and layer normalization respectively.
[0070] This embodiment proposes a guided decoupling network GDNet. The network integrates a specific attribute guidance module, and designs different branches for feature processing for images under low light, foggy, and normal light conditions. Ensure effective feature extraction of visible light images in different scenes. The perception attribute fusion module realizes adaptive aggregation of features of the three branches. The multi-scene visible light guidance module uses multiple attention mechanisms to realize feature interaction between visible light images and thermal infrared images, and realizes efficient guidance of thermal infrared images by visible light images.
[0071] Compared with the existing state-of-the-art single image super-resolution methods and guided super-resolution methods, this embodiment restores richer texture and structural details and has better performance in terms of evaluation indicators and visual perception.
[0072] Example 2
[0073] Based on the model of Example 1, this embodiment provides an application method of the above model, comprising the following steps:
[0074] Step 1: Use the specific attribute guidance module to extract features of visible light images under different weather and lighting conditions to obtain feature information of three branches.
[0075] Step 2: Adaptively integrate visible light image features from different branches through a selection mechanism
[0076] Step 2.1 concatenates the feature information features output by the three branches along the channel dimension and merges them into one feature.
[0077] In step 2.2, the merged features are average pooled and max pooled in the channel direction to extract the spatial relationship between the three branch features and achieve simple aggregation of the features.
[0078] Step 2.3 models the aggregated features through spatial attention and convolution operations to obtain three spatial selection masks.
[0079] Step 2.4: Multiply the original features of the three branches by their corresponding spatial selection masks respectively, and add the results with weighted features to obtain adaptive fusion features.
[0080] Step 3: Use the obtained adaptive aggregation features to guide the infrared features, thereby achieving infrared feature enhancement.
[0081] Step 4: Upsample the fused feature map to obtain a high-resolution UAV thermal infrared image.
[0082] Example 3
[0083] This embodiment provides a training method for the model of Example 1. During the training phase, three key issues must be addressed. First, if the entire training data set is used directly to train the network, the errors caused by the low light and fog conditions in the training data will propagate through all branches. Second, since the scenes from which these data are derived are not available during the testing phase, the specific challenges posed by optical representation in the OTUAV-SR task cannot be predicted. The ultimate goal of this embodiment is to improve the ability to utilize optical representations while suppressing responses to irrelevant representations. To address these issues, a three-stage training strategy is proposed to promote efficient learning of guided thermal super-resolution tasks in complex scenes.
[0084] Phase 1. Trunk Training:
[0085] In the first stage, all VGTSR2.0 training data are used to train feature extraction of optical images under general weather conditions. This module includes 4 convolutional layers and LeakyReLU activation functions when the super-resolution is 4 times, and 6 convolutional layers and LeakyReLU activation functions when the super-resolution is 8 times. Note that all three guide branches and attribute-aware fusion layers are removed in this stage. The main purpose is to train a universal feature extractor for all attribute data.
[0086] Stage 2: Training three branches:
[0087] After obtaining the trained feature extractor, this embodiment trains three attribute-based branches in sequence, using the corresponding attribute-based training data to learn the optical representation of the specific attribute, while freezing all other parameters. It should be noted that when training a specific branch, only the data of the corresponding scene is used, and the data of other scenes and branch structures will be excluded from the network. The training settings remain unchanged from the first stage.
[0088] Stage 3: Training of attribute-aware fusion module and fine-tuning of MOGM:
[0089] After obtaining the learned feature extractor and three attribute-specific branches, this embodiment trains the attribute-aware fusion module using all the data and fine-tunes the MOGM while freezing all other parameters.
[0090] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A guided decoupled UAV thermal infrared image super-resolution model, characterized by: include: AGM module, AFM module, MOGM module; The visible light image is used as the input of the AGM module, and feature extraction is performed through three branches: normal light scene branch, low light scene branch, and foggy occlusion scene branch to obtain visible light image information under three types of scenes. The visible light image information under the three types of scenes is used as the input of the AFM module. The AFM module selectively activates and utilizes the most relevant branch features according to the input visible light image information, and fuses it with other branch features to obtain the visible light image fusion features under the three types of scenes; the visible light image fusion features and the thermal infrared image are simultaneously used as the input of the MOGM module. In MOGM, overlapping window attention, cross-modal attention and sliding window attention are used to achieve full interaction between visible light features and infrared features, and finally upsampling is performed to generate a high-resolution thermal infrared image.
2. The guided decoupled UAV thermal infrared image super-resolution model according to claim 1 is characterized in that: The normally illuminated scene branch includes a backbone network and two multi-spectral guidance layers according to the data flow, and enhances the guidance information by combining cross-modal self-attention and shift window mechanism.
3. The guided decoupled UAV thermal infrared image super-resolution model according to claim 1, characterized in that: The low-light scene branch includes three parts in sequence according to the data flow: a pre-trained Retinex decomposition network, a backbone network, and four multi-spectral guidance layers. The Retinex decomposition network performs feature enhancement on the image based on the retinal imaging principle.
4. The guided decoupled UAV thermal infrared image super-resolution model according to claim 1, characterized in that: The foggy occlusion scene branch includes a backbone network and six gated attention layers in sequence according to the data flow direction; the visible light features under foggy conditions are first compressed along the channel dimension, and then a weight factor is generated for each channel; the feature map is mapped and weighted along the channel direction through the weight factor to enhance the key features.
5. The guided decoupled UAV thermal infrared image super-resolution model according to claim 1, characterized in that: The MOGM module includes four RMAG groups according to the data flow, each RMAG contains an overlapping multi-head cross-modal attention layer, a multi-spectral guidance layer, a Swin Transformer attention layer and the original overlapping Transformer attention layer.
6. A guided decoupled UAV thermal infrared image super-resolution method, characterized in that: The following steps are involved: S1. Use the AGM module to extract features from visible light images under different weather and lighting conditions; The visible light image is used as the input of the AGM module, and features are extracted through three branches: normal light scene branch, low light scene branch, and foggy occlusion scene branch, to obtain visible light image information under three types of scenes; S2. Adaptively integrate visible light image features from different branches through a selection mechanism; The visible light image information under the three types of scenes is used as the input of the AFM module. The AFM module selectively activates and utilizes the most relevant branch features according to the input visible light image information, and fuses it with other branch features to obtain the visible light image fusion features under the three types of scenes; S3. Use the obtained adaptive aggregation features to guide the infrared features, thereby achieving infrared feature enhancement. The visible light image fusion features and the thermal infrared image are simultaneously used as inputs of the MOGM module. In MOGM, overlapping window attention, cross-modal attention and sliding window attention are used to achieve full interaction between visible light features and infrared features. S4. Generate a high-resolution thermal infrared image by upsampling the feature map after full interaction in step S3.
7. The guided decoupled UAV thermal infrared image super-resolution method according to claim 6, characterized in that: The normally illuminated scene branch includes a backbone network and two multi-spectral guidance layers according to the data flow, and enhances the guidance information by combining cross-modal self-attention and shift window mechanism.
8. The guided decoupled UAV thermal infrared image super-resolution method according to claim 6, characterized in that: The low-light scene branch includes three parts in sequence according to the data flow: a pre-trained Retinex decomposition network, a backbone network, and four multi-spectral guidance layers. The Retinex decomposition network performs feature enhancement on the image based on the retinal imaging principle.
9. The guided decoupled UAV thermal infrared image super-resolution method according to claim 6, characterized in that: The foggy occlusion scene branch includes a backbone network and six gated attention layers in sequence according to the data flow direction; the visible light features under foggy conditions are first compressed along the channel dimension, and then a weight factor is generated for each channel; the feature map is mapped and weighted along the channel direction through the weight factor to enhance the key features.
10. The training method of the guided decoupled UAV thermal infrared image super-resolution model according to any one of claims 1 to 5, characterized in that: It consists of three stages: Phase 1: Trunk Training All VGTSR2.0 training data are used to train feature extraction of optical images under general weather conditions, i.e., the backbone network. In this stage, the three guide branches and the AFM module are removed, and a general feature extractor is trained for all attribute data. Stage 2: Training three attribute-specific branches: After obtaining the trained feature extractor, three attribute-based branches are trained in sequence, namely the normal lighting scene branch, the low lighting scene branch, and the foggy occlusion scene branch. The corresponding attribute-based training data is used to learn the optical representation of specific attributes, while freezing all other parameters. When training a specific branch, only the data of the corresponding scene is used, and the data of other scenes and branch structures will be excluded from the network. Stage 3: Training of AFM module and fine-tuning of MOGM: After obtaining the learned feature extractor and three attribute-specific branches, the attribute-aware fusion module is trained using all the data and the MOGM is fine-tuned while freezing all other parameters.