Airborne infrared image super-resolution method based on degradation perception

Abstract

The invention provides an airborne infrared image super-resolution reconstruction system based on degradation perception. The airborne infrared image super-resolution reconstruction system comprises a degradation representation learning framework, a residual state space block and a cross-modal feature integration framework. The method comprises the following steps: firstly, extracting degradation information from an infrared image through a self-supervised learning framework; then, the spatial dependence of the respective features of the visible light image and the infrared image is captured through a feature extraction method based on a spatial model; and finally, the visible light is fully utilized to carry out feature supplement on the infrared image, and an unmanned aerial vehicle infrared super-resolution result with a more real visual perception effect is generated.

Description

Technical Field

[0001] The present invention relates to a UAV infrared image restoration technology, which is suitable for the reconstruction and restoration of UAV infrared images at low resolution, and in particular to an airborne infrared image super-resolution system based on degradation perception. Background Art

[0002] Drones flying at high altitudes are easily affected by motion blur and atmospheric turbulence, which results in low resolution and loss of details of images collected by drone infrared sensors. Therefore, improving the clarity of acquired low-resolution infrared images is an urgent problem to be solved. Enhancing the resolution of infrared images through infrared super-resolution restoration technology has become an important means to solve this problem. Infrared super-resolution reconstruction is the process of predicting a high-resolution infrared image based on one or more observed low-resolution infrared images, which is a typical ill-posed problem.

[0003] Super-resolution algorithms for infrared images are mainly divided into reconstruction-based and learning-based methods, among which learning-based methods can be subdivided into shallow learning and deep learning methods. Reconstruction-based methods use a specific module to generate new image information from low-resolution images. However, in actual application scenarios, since the resolution of the acquired infrared images is usually low, a larger magnification is required. However, as the magnification increases, the performance of reconstruction-based super-resolution algorithms decreases significantly and is difficult to meet actual needs. Learning-based methods can reconstruct the high-frequency edge and texture information missing from the original low-resolution infrared images through the training data set method.

[0004] Early single-UAV infrared super-resolution methods lack global context awareness, and they cannot accurately reconstruct textures or local structures in the UAV scene. Instead, these methods may generate artificial high-frequency signals to fill in the missing details, resulting in unnatural textures or repeated artifacts, which severely degrade the image quality.

[0005] In recent years, infrared super-resolution methods have been developed to compensate for missing information in infrared images by introducing rich details and texture information from visible light images, which can improve the clarity and detail representation of infrared images. However, due to significant modality differences, the texture information from visible light images to infrared images may not be accurate enough; in addition, due to the complex imaging conditions on UAV platforms, the scene alignment between infrared and visible light images is not always accurate, resulting in unnatural or unrealistic details in the restored infrared images. Summary of the invention

[0006] The present invention aims to provide an airborne infrared image super-resolution system based on degradation perception, which extracts long-distance spatial features of images through the Mamba framework and optimizes the infrared super-resolution technology in combination with the degradation module, and obtains accurate infrared high-resolution results in combination with feature interaction, refinement and enhancement modules.

[0007] In order to solve the above technical problems, the specific scheme adopted by the present invention is: an airborne infrared image super-resolution system based on degradation perception, sampling the degradation learning module to collect infrared degradation information; using a visual attention state space module, combined with long-range correlation acquisition, dynamic feature modulation and efficient feature integration to improve the super-resolution reconstruction of low-resolution infrared images; using a cross-modal feature integration module to integrate infrared images and visible light features from three levels of cross-modal feature interaction, refinement and enhancement, effectively improving the quality of the reconstructed infrared image.

[0008] Optionally, include the following steps:

[0009] S1: Acquire the visible light image and the corresponding infrared image of the target image;

[0010] S2: A self-supervised learning framework is used to extract degradation information from low-resolution UAV infrared images, and the restoration process is guided by the degradation perception module;

[0011] S3: Uses residual state space module to capture visible and infrared drone image features;

[0012] S4: Adopt cross-modal integrated feature framework to perform feature interaction between different modalities;

[0013] S5: Adopt loss function constraints to finally generate high-resolution visible light infrared images.

[0014] Optionally, the target image VGTSR dataset in S1 contains 1025 pairs of visible light drone images and infrared drone image pairs, and the shooting scenes are campus and city street scenes taken at a distance of 200 meters.

[0015] Optionally, S2 uses a self-supervised learning framework to extract degradation information from low-resolution UAV infrared images, and uses this information to guide the restoration process through the degradation perception module:

[0016] Degenerate means:

[0017] I LR =(I HR #k σ )↓ s +n δ .

[0018] Among them I LR Represents a low-resolution infrared image; IHR represents a high-resolution infrared image; k σ represents four blur kernels of different sizes, namely: Gaussian blur kernel of 3≤σ≤15, average blur kernel of 3≤σ≤15, intermediate blur kernel of 3≤σ≤15, and motion blur kernel of 5≤σ≤25; n δ Indicates different noise levels; # indicates convolution operation; ↓ s Represents the downsampling factor.

[0019] The image re-degradation step is designed as:

[0020] First, an encoder with 5 layers of 5×5 convolutional kernels is used to extract the low-resolution infrared image I LR1 The degenerate representation feature F d , and I LR1 Low-resolution infrared images with the same degradation but different image contents I LR2 .

[0021] Then, the degenerate representation feature F d and high-resolution infrared images I HR2 Send it to the re-degradation network to generate a pseudo low-resolution infrared image

[0022] Finally, by calculating and I LR2 The loss between.

[0023] Image degradation loss in S2:

[0024]

[0025] Where L rd represents the degradation loss, I LR2 is a low-resolution infrared image with the same degradation as the input image but with different content, It is a pseudo low-resolution infrared image synthesized through degradation training.

[0026] Energy loss over distance in S2:

[0027] L ed =2A-BC,

[0028]

[0029] Where L ed is the energy distance loss; the encoder randomly selects m images and encodes them into x 1 ,x 2 ...x m , where x i ∈R CRepresents the degradation information of the i-th image; then sample n times from the Gaussian distribution to get y 1 ,y 2 ...y n ,y j ∈R C is the jth sample sampled from a Gaussian distribution. The bounded constraint prevents different degraded representations from crowding into a small space and can better distinguish subtle degradation details.

[0030] Optionally, S3 uses a residual state space module to capture long-range dependencies in feature space, designed as:

[0031] Focused Visual State Space Module in S3:

[0032] Z i =VFSSM(LN(F i ))+s·F i .

[0033] Where Z i represents the i-th residual state space module, F i is the input deep feature of the i-th layer, s∈R C is a learnable scaling factor, VFSSM is the focused visual state space module, and LN is the normalization layer.

[0034] The focused visual state space module is designed as follows:

[0035] F 1 =LN(SSM 2D (σ(DWConv(Linear(F in ))))),

[0036] F 2 =SA(Linear(F in )),

[0037]

[0038] Among them, DWConv is the depth convolution, σ is the activation layer, SSM 2D is the 2D selective scanning module layer, LN is the normalization layer, F in is the input feature, SA is the spatial attention activation module, and Linear is the linear layer.

[0039] Optionally, the cross-modal integrated feature framework in S4 is designed to interact features between different modalities as follows:

[0040] The steps of the feature-feature interaction module are designed as follows:

[0041] First, the extracted infrared and visible light features are added together to ensure that the features of both modalities are reflected to a certain extent in the initial interaction stage;

[0042] The fused features are input into two parallel branches, namely the spatial attention branch and the channel attention branch;

[0043] Both branches are sequentially subjected to 1×1 convolutional layer, 3×3 convolutional layer and 1×1 convolutional layer for feature extraction, and the results are multiplied by the attention coefficient corresponding to each channel;

[0044] Finally, the two channel features are added.

[0045] Feature refinement module:

[0046] Q=Pool 4 (Conv(F 2 )),

[0047] K=Pool 2 (Conv(F 3 )),

[0048] V=Conv(F 4 ),

[0049]

[0050] Among them, Conv is the convolution layer, Pool i is an average pooling operation with kernel size and stride i, F j represents the features of the jth layer, SoftMax is the activation function; Q, K, and V are the query vector, key vector, and value vector, respectively.

[0051] The steps of feature enhancement module are designed as follows:

[0052] The first parallel branch uses dilated convolution to capture spatial features from visible light images, effectively enhancing details and global context;

[0053] A second, parallel branch will add the optical and infrared signatures together, ensuring that comprehensive information from both modalities is preserved and integrated;

[0054] The parallel third branch extracts the channel information of infrared features and uses average pooling and maximum pooling to ensure robust features;

[0055] Finally, the third branch result is added to the spatial feature of the first branch, and the result is calculated using the activation function and then added to the second branch feature to enhance and optimize the deep information.

[0056] Optionally, the cross-modal integrated feature framework in S5 is designed to interact features between different modalities as follows:

[0057]

[0058] in, represents the i-th layer trained by VGG, λ MSE and λ perc represents the loss weight parameter, MSE represents the mean square error of pixel and feature space, and the second loss is the perceptual loss; I HR It is a high-resolution UAV infrared image. SR It is a super-resolution drone infrared image.

[0059] The present invention provides an airborne infrared image super-resolution method based on degradation perception, which has the following beneficial effects:

[0060] This degradation-aware airborne infrared image super-resolution method uses a degradation representation learning module to collect infrared degradation information and significantly enhance the reconstructed features, thereby improving the fidelity of the final output. Then, a visual attention state space module is used to cleverly combine the advantages of long-range correlation acquisition, dynamic feature modulation and efficient feature integration to improve the super-resolution reconstruction of low-resolution infrared images. A cross-modal feature integration module is used to perform feature integration at three levels: cross-modal feature interaction, refinement and enhancement, effectively improving the quality of the reconstructed infrared image. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the present invention;

[0062] Figure 2 is the visual state space module of the present invention;

[0063] Figure 3 It is a cross-modal feature integration module of the present invention; DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] Specifically, the following steps are included:

[0066] S1: Acquire the visible light image and the corresponding infrared image of the target image;

[0067] S2: A self-supervised learning framework is used to extract re-degraded information from low-resolution UAV infrared images, and the restoration process is guided by the re-degraded network and energy distance loss constraints;

[0068] S3: Uses residual state space module to capture visible and infrared drone image features;

[0069] S4: Adopt cross-modal integrated feature framework to perform feature interaction between different modalities;

[0070] S5: Adopt loss function constraints to finally generate high-resolution visible light infrared images.

[0071] In S1, the visible light image and the corresponding infrared image of the target image are obtained:

[0072] The target image VGTSR dataset contains 1025 pairs of visible light and infrared drone images, and the shooting scenes are campus and city street scenes taken at 200 meters.

[0073] The low-resolution infrared images of S2 are designed as follows:

[0074] I LR =(I HR #k σ )↓ s +n δ

[0075] Among them I LR represents a low-resolution infrared image; k σ represents four blur kernels of different sizes, namely: Gaussian blur kernel of 3≤σ≤15, average blur kernel of 3≤σ≤15, intermediate blur kernel of 3≤σ≤15, and motion blur kernel of 5≤σ≤25; n δ Indicates different noise levels; # indicates convolution operation; ↓ s Represents the downsampling factor.

[0076] The image re-degradation step is designed as:

[0077] First, an encoder with 5 layers of 5×5 convolutional kernels is used to extract the low-resolution infrared image I LR1 The re-degenerate representation feature F d , and I LR1 Low-resolution infrared images with the same degradation but different contents I LR2 .

[0078] Then, the degenerate representation feature F d and high-resolution infrared images I LR2 Send it to the re-degradation network to generate a pseudo low-resolution infrared image

[0079] Finally, by calculating and I LR2 The loss between.

[0080] Image loss in S2

[0081]

[0082] Where L rd represents the degradation loss, is a low-resolution infrared image with the same degradation as the input image but with different content, are synthetic low-resolution images that were trained.

[0083] Energy loss over distance in S2:

[0084] L ed =2A-BC,

[0085]

[0086] L ed is the energy distance loss, where the encoder randomly selects m images and encodes them into x 1 ,x 2 ...x m , where x i ∈R C Represents the degradation information of the i-th image; then sample n times from the Gaussian distribution to get y 1 ,y 2 ...y n ,y j ∈R C is the jth sample sampled from the Gaussian distribution. The bounded constraint prevents the degenerate representation from crowding into a small space and can better distinguish subtle degenerate details.

[0087] like Figure 3 As shown, the S3 residual state space module:

[0088] Z i =VFSSM(LN(F i ))+s·F i .

[0089] Where Z i represents the i-th residual state space module, F i is the input deep feature of the i-th layer, s∈R C is a learnable scaling factor, VFSSM is the state-space module for focused vision, and LN is the normalization layer.

[0090] The focused visual state space module in S3 contains two parallel paths. The first branch involves deep convolutional layers and activation functions; the second branch involves the spatial attention activation module, which is specifically expressed as:

[0091] F1 =LN(SSM 2D (σ(DWConv(Linear(F in ))))),

[0092] F 2 =SA(Linear(F in )),

[0093]

[0094] Among them, DWConv is the deep convolution layer, σ is the activation layer, and SSM 2D is a two-dimensional focused visual state space module layer, LN is a normalization layer, SA is a spatial attention activation module, It is Hadamard.

[0095] like Figure 3 As shown, the extracted visible light and infrared drone images are input into the cross-modal feature integration module, and the cross-modal feature integration module includes a feature interaction module, a feature refinement module, and a feature interaction module.

[0096] The design steps of the feature interaction module in S4 are:

[0097] First, the extracted infrared and visible light features are added together to ensure that the features of both modalities are reflected to a certain extent in the initial interaction stage;

[0098] The fused features are input into two parallel branches, namely the spatial attention branch and the channel attention branch;

[0099] Both branches are sequentially subjected to 1×1 convolutional layer, 3×3 convolutional layer and 1×1 convolutional layer for feature extraction, and then multiplied by the attention coefficient corresponding to each branch;

[0100] Finally, the two branch features are added.

[0101] Feature refinement module in S4:

[0102] Q=Pool 4 (Conv(F 2 )),

[0103] K=Pool 2 (Conv(F 3 )),

[0104] V=Conv(F 4 ),

[0105]

[0106] Among them, Conv is the convolution layer; Pooli is an average pooling operation with kernel size and stride i; F j represents the visible light or infrared drone features of the jth layer; SoftMax is the activation function; Q, K, and V are the query vector, key vector, and value vector, respectively.

[0107] The design steps of the three-branch feature interaction module in S4 are as follows:

[0108] First, the first molecule uses dilated convolution to capture spatial features from visible light images, effectively enhancing details and global context;

[0109] The second branch will then add the optical and infrared signatures together, ensuring that comprehensive information from both modalities is preserved and integrated;

[0110] Then the third branch is used to extract the channel information of infrared features, and average pooling and maximum pooling are used to ensure robust features;

[0111] Finally, the third branch result is added to the spatial feature of the first branch, and the result is calculated using the activation function and then added to the second branch feature to enhance and optimize the deep information.

[0112] The S5 loss function constraint is expressed as:

[0113]

[0114] in, represents the i-th layer feature after VGG training, set i∈[1,2,3,4]; λ MSE and λ perc represents the loss weight parameter; MSE represents the mean square error of pixel and feature space, Perc is the perceptual loss; I HR It is a high-resolution UAV infrared image. SR It is a super-resolution drone infrared image.

[0115] In this embodiment, the test set proposed in S1 is used according to steps 1-5 to obtain the final UAV infrared super-resolution result. The peak signal-to-noise ratio evaluation index of the reconstructed UAV infrared image reaches 32.04dB in ×4 super-resolution and 26.67dB in ×8 super-resolution.

[0116] It should be noted that, in this article, the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.

[0117] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A degradation-aware airborne infrared image super-resolution method, characterized in that: A self-supervised learning framework is used to directly extract degradation information from the input low-resolution infrared image. A residual state space module is used to capture the long-range spatial dependencies of visible light and infrared drone images respectively, thereby improving the clarity and detail representation of the super-resolution results. A cross-modal feature integration framework is used to solve modal differences, and the quality of infrared image reconstruction is improved by interacting, refining and enhancing infrared features with visible light features.

2. The degradation-aware airborne infrared image super-resolution method according to claim 1, characterized in that: The following steps are involved: S1: Acquire the visible light image and the corresponding infrared image of the target image; S2: A constrained self-supervised learning framework is used to extract degradation information from low-resolution UAV infrared images, and the restoration process is guided by the degradation perception module; S3: Uses residual state space module to capture visible and infrared drone image features; S4: Adopt cross-modal integrated feature framework to perform feature interaction between different modalities; S5: Adopt loss function constraints to finally generate high-resolution visible light infrared images.

3. The method for obtaining a visible light image of a target image and a corresponding infrared drone image according to claim 2, characterized in that: The target image VGTSR dataset contains 1025 pairs of visible light images and infrared drone image pairs. The shooting scenes are campus and city street scenes taken at an altitude of 200 meters.

4. The self-supervised constrained degradation perception module according to claim 2, characterized in that: The self-supervised learning framework in S2 is designed as follows: Degenerate means: I LR =(I HR #k σ )↓ s +n δ . Among them I LR Represents a low-resolution infrared image; I HR represents a high-resolution infrared image; k σ represents four blur kernels of different sizes, namely: Gaussian blur kernel of 3≤σ≤15, average blur kernel of 3≤σ≤15, intermediate blur kernel of 3≤σ≤15, and motion blur kernel of 5≤σ≤25; n δ Indicates different noise levels; # indicates convolution operation; ↓ s Represents the downsampling factor. The image re-degradation step is designed as: (1) First, an encoder with 5 layers of 5×5 convolutional kernels is used to extract the low-resolution infrared image I LR1 The degenerate representation feature F d , and I LR1 Low-resolution infrared images with the same degradation but different image contents I LR2 . (2) Then, the degenerate representation feature F d and high-resolution infrared images I HR2 Send it to the re-degradation network to generate a pseudo low-resolution infrared image (3) Finally, by calculating and I LR2 The loss between. Image loss in S2: Where L rd represents the degradation loss, is the same degraded image as the input image but with different content. It is a pseudo low-resolution image synthesized through degradation training. Energy distance loss constraint in S2: L ed =2A-B-C, Where L ed is the energy distance loss; the encoder randomly selects m images and encodes them into x1,x2...x m , where x i ∈R C represents the degradation information of the i-th image; then sample n times from the Gaussian distribution to obtain y1, y2...y n ,y j ∈R C is the jth sample sampled from a Gaussian distribution. The bounded constraint prevents different degraded representations from crowding into a small space and can better distinguish subtle degradation details.

5. The residual state space module according to claim 2 captures the long-range dependency of the feature space, characterized in that: Visual Focus State Space Module in S3: Z i =VFSSM(LN(F i ))+s·F i . Where Z i represents the i-th residual state space module, F i is the deep feature of the i-th layer input, s∈R C is a learnable scaling factor, VFSSM is the focused visual state space module, and LN is the normalization layer. Focused Visual State Space Module in S3: F1=LN(SSM 2D (σ(DWConv(Linear(F in ))))), F2=SA(Linear(F in )), Among them, DWConv is the depth convolution, σ is the activation layer, SSM 2D is a two-dimensional selective scanning module layer, LN is a normalization layer, SA is a spatial attention activation module, and Linear is a linear layer.

6. According to the cross-modal integrated feature framework of claim 2, feature interaction between different modalities is performed, characterized in that: The feature interaction module steps in S4 are designed as follows: (1) First, the extracted infrared and visible light features are added together to ensure that the features of the two modalities are integrated to a certain extent in the initial interaction stage; (2) Input the fused features into two parallel branches, namely the spatial attention branch and the channel attention branch; (3) Both branches are sequentially subjected to 1×1 convolutional layer, 3×3 convolutional layer and 1×1 convolutional layer for feature extraction, and then multiplied by the attention coefficient corresponding to each branch; (4) Finally, the features of the two branches are added. Feature refinement module in S4: Q=Pool4(Conv(F 2 )), K=Pool2(Conv(F 3 )), V=Conv(F 4 ), Among them, Conv is the convolution layer; Pool i is an average pooling operation with kernel size and stride i; F j represents the features of the jth layer; SoftMax is the activation function; Q, K, and V are the query vector, key vector, and value vector, respectively. The three-branch feature enhancement module steps in S4 are designed as follows: (1) First, the first branch uses dilated convolution to capture spatial features from visible light images, effectively enhancing detail features and global context associations; (2) The optical and infrared features are then summed and fed into the second branch, ensuring that comprehensive information from both modalities is preserved and integrated; (3) Then use the third branch to extract the channel information of the infrared features, and use average pooling and maximum pooling to ensure robust features. (4) Finally, the result of the third branch is added to the spatial feature of the first branch, and the result is calculated using the activation function and then added to the second branch feature to enhance and optimize the deep information.

7. According to the constraint loss function of claim 2, a high-resolution visible light infrared image is finally generated, characterized in that: Loss function in S5 in, represents the i-th layer feature after VGG training, λ MSE and λ perc represents the loss weight parameter, MSE represents the mean square error loss of pixel and feature space, and Perc is the perceptual loss; I HR It is a high-resolution UAV infrared image. SR It is a super-resolution drone infrared image.

Images