Multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanism

Abstract

The invention discloses a multi-temporal infrared image super-resolution reconstruction method based on multiband adaptive weighting and an attention mechanism, and belongs to the field of infrared remote sensing image processing. Comprising the following steps: acquiring a medium-short wave infrared image data set, preprocessing the data set, constructing a multi-band multi-time feature fusion deep learning model, training the deep learning model by using the preprocessed data set, and inputting a low-resolution infrared remote sensing image to be reconstructed into a trained super-resolution reconstruction model; according to the method, model training and super-resolution reconstruction are carried out by using the data set containing the medium-short wave infrared band and 16 time periods, the characteristics of data of different bands and time periods are fully utilized, richer feature information can be obtained, and the detail retention capability and authenticity of the reconstructed image are improved; according to the constructed multi-band multi-time feature fusion deep learning model, deep features are extracted through the feature extraction layer, and effective fusion of different features is achieved through the multi-band multi-time feature fusion layer.

Description

Technical Field

[0001] This invention relates to the field of infrared remote sensing image processing technology, and in particular to a method for super-resolution reconstruction of multi-temporal infrared images based on multi-band adaptive weighting and attention mechanisms. Background Technology

[0002] Infrared remote sensing technology has advantages such as being unaffected by lighting conditions and being able to penetrate clouds and fog, and has been widely used in environmental monitoring, military reconnaissance, and resource exploration. However, due to the hardware limitations of infrared sensors, the acquired infrared remote sensing images often have low resolution, affecting subsequent image analysis and applications.

[0003] Super-resolution reconstruction technology aims to recover high-resolution images from low-resolution images using algorithms, and is an effective means of improving image quality. In recent years, deep learning technology has made significant progress in the field of super-resolution reconstruction, and deep learning-based super-resolution reconstruction methods have better reconstruction results than traditional methods.

[0004] However, most current multi-temporal infrared image super-resolution reconstruction methods based on multi-band adaptive weighting and attention mechanisms only process images in a single band or a short time period, failing to fully utilize the characteristics of mid- and short-wave infrared bands and multi-time period data. As a result, the reconstructed images still have shortcomings in terms of detail preservation and realism, and the generalization ability of the model needs to be improved.

[0005] Therefore, a multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanism is needed to solve the problems mentioned above. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanisms, which solves the problems mentioned in the background technology.

[0007] Technical Solution: To solve the above-mentioned technical problems, according to one aspect of the present invention, more specifically, a multi-temporal infrared image super-resolution reconstruction method based on a multi-band adaptive weighting and attention mechanism, comprising the following steps:

[0008] S1: Obtain a mid-wave and short-wave infrared image dataset. This dataset contains mid-wave infrared and short-wave infrared images, covering 16 time periods formed by the intersection of the four seasons, morning, noon, evening, and midnight. (The mid-wave and short-wave infrared image dataset contains 50,000 pairs of high / low resolution images, with 25,000 pairs each for the mid-wave and short-wave infrared bands; the high-resolution image size is 1024×1024 pixels, and the low-resolution image size is 256×256 pixels (generated through bicubic downsampling, downsampling due to…). (Sub-4); The data comes from data taken by the Fengyun-4 satellite. The image content is a cloud map from a global perspective. The disk map is formed by stitching regional data based on the Fengyun satellite remote sensing data network. The acquisition device is the Fengyun-4 satellite (equipped with a mid- and short-wave infrared imager. Specific parameters: the original resolution of the high-resolution image is adapted to the satellite's global observation accuracy. After processing, it is uniformly 1024×1024 pixels. The spectral range is 3-5μm for mid-wave and 1-3μm for short-wave. The observation mode is global disk observation. The data is acquired from the Fengyun satellite remote sensing data network).

[0009] S2: Preprocess the dataset, including band alignment, time period labeling, and image cropping and normalization;

[0010] S3: Construct a deep learning model for multi-band and multi-temporal feature fusion, the model including an input layer, a feature extraction layer, a multi-band and multi-temporal feature fusion layer, a super-resolution reconstruction layer, and an output layer;

[0011] The input to the super-resolution reconstruction layer is a 256×256 feature map after feature extraction and fusion. Resolution enhancement is achieved by alternately connecting two sub-pixel convolutional layers (scaling factor 2) with convolutional layers.

[0012] The first subpixel convolutional layer converts the 256×256 feature map into a 512×512, followed by a 3×3 convolutional layer to optimize the features.

[0013] The second sub-pixel convolutional layer converts the 512×512 feature map into 1024×1024, followed by a 3×3 convolutional layer to optimize the features.

[0014] S4: Train the deep learning model using the preprocessed dataset to obtain a trained super-resolution reconstruction model;

[0015] Perceptual loss: Input the reconstructed image and the real high-resolution image into the pre-trained VGG16 network, take the feature output of the conv3_3 layer, and calculate the L2 norm of both as the perceptual loss.

[0016] Mean squared error loss: Calculates the average of the squared differences between the corresponding pixel values ​​of the reconstructed image and the real high-resolution image;

[0017] Total loss: Perception loss × 0.5 + Mean squared error loss × 0.5;

[0018] S5: Input the low-resolution infrared remote sensing image to be reconstructed into the trained super-resolution reconstruction model, and output a high-resolution infrared remote sensing image.

[0019] Furthermore, in step S2, the band alignment adopts a feature point matching method to spatially align the mid-wave infrared band image and the short-wave infrared band image to ensure the accuracy of subsequent feature fusion.

[0020] Furthermore, in step S3, the feature extraction layer uses an improved residual network, and an attention mechanism is introduced into each residual block. The attention mechanism can highlight important feature information and suppress irrelevant information, thereby more effectively extracting deep features of images from different bands and time periods.

[0021] Furthermore, in step S3, the multi-band, multi-time feature fusion layer employs an adaptive weight fusion strategy, assigning different weights based on the feature importance of images from different bands and time periods for fusion. Specifically, by training a weight prediction network, the weights of each feature are automatically predicted based on the input features, achieving adaptive fusion. This fully utilizes the complementary information from data across different bands and time periods, enhancing the feature representation capability.

[0022] In the adaptive weight fusion strategy, the fusion weights of each feature are calculated using the following formula:

[0023] Let F be the i-th feature (from a certain band or time period). i Its fusion weight w i The formula for calculation is:

[0024]

[0025] in:

[0026] ||F i ||2 is a feature F i The L2 norm is used to measure the activation strength of a feature itself;

[0027] For time-dependent factors, Δt i α represents the time difference between the time period corresponding to the current feature and the target time period, where α is the time sensitivity coefficient.

[0028] -σ(β·c i ) represents the band correlation factor, c i Indicates the band type (1 for mid-wave infrared, 0 for short-wave infrared), β is the band sensitivity coefficient, σ(x) = 1 / (1+e^(-1 / ... -x() is the sigmoid function;

[0029] - The denominator is the sum of the weights of all features, achieving weight normalization, and satisfying the following conditions:

[0030]

[0031] The fused features are the weighted sum of each feature and its corresponding weight:

[0032]

[0033] Furthermore, in step S3, the super-resolution reconstruction layer employs a structure of alternating sub-pixel convolutional layers and convolutional layers to gradually improve image resolution. Sub-pixel convolutional layers can transform low-resolution feature maps into high-resolution feature maps through pixel rearrangement, while convolutional layers are used for further feature extraction and optimization. The alternating connections effectively preserve image details while improving resolution.

[0034] Furthermore, in step S4, the loss function used in the training process is a weighted sum of perceptual loss and mean squared error loss. Perceptual loss can constrain the difference between the reconstructed image and the real image from the perspective of high-level semantic information, while mean squared error loss can ensure the pixel-level accuracy of the reconstructed image. The combination of the two can improve the visual effect and realism of the reconstructed image.

[0035] Furthermore, in step S4, a dynamic learning rate adjustment strategy is adopted during model training. The learning rate is gradually reduced as the number of training rounds increases. This allows the model to converge quickly in the early stages of training and finely adjust the parameters in the later stages of training, thereby improving the training effect of the model.

[0036] Furthermore, in step S5, the low-resolution infrared remote sensing image to be reconstructed is a mid-wave infrared band image or a short-wave infrared band image, and contains corresponding time period information, so that the model can perform accurate super-resolution reconstruction based on the corresponding band and time period characteristics.

[0037] The beneficial effects of the multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanism of the present invention are as follows:

[0038] (1) This invention uses a dataset containing mid- and short-wave infrared bands and 16 time periods for model training and super-resolution reconstruction. It makes full use of the characteristics of data in different bands and time periods, and can obtain richer feature information, thereby improving the detail preservation and realism of the reconstructed image.

[0039] (2) The deep learning model for multi-band and multi-temporal feature fusion is constructed. Deep features are extracted through the feature extraction layer, and the multi-band and multi-temporal feature fusion layer realizes the effective fusion of different features. The super-resolution reconstruction layer gradually improves the image resolution. The layers work together to significantly improve the effect of super-resolution reconstruction.

[0040] (3) The adaptive weight fusion strategy is adopted for feature fusion, which can automatically allocate weights according to the importance of features, give full play to the complementary role of data in different bands and time periods, and improve the model's adaptability and generalization ability to complex scenarios.

[0041] (4) During the training process, the weighted sum of perceptual loss and mean squared error loss is used as the loss function, and combined with the dynamic adjustment strategy of learning rate, the training effect and reconstruction accuracy of the model are further improved. Attached Figure Description

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific implementation methods.

[0043] Figure 1 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0044] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0045] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Reference Figure 1 A multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanism includes the following implementation steps:

[0047] S1: Obtain a mid-wave and short-wave infrared image dataset. This dataset contains mid-wave and short-wave infrared images, covering 16 time periods formed by the intersection of the four seasons, morning, noon, evening, and midnight. The four seasons are spring, summer, autumn, and winter; morning, noon, evening, and midnight are divided into morning (6:00-8:00), forenoon (8:00-12:00), noon (12:00-14:00), afternoon (14:00-18:00), evening (18:00-22:00), midnight (22:00-24:00), and early morning (0:00-6:00), which are combined to form 16 time periods. The dataset is derived from data captured by the Fengyun-4 satellite. The images are global cloud images, which are obtained from cloud image data of different regions through the Fengyun satellite remote sensing data network and then stitched together to form a disk image.

[0048] S2: Preprocessing the dataset:

[0049] Band alignment: A SIFT-based feature point matching method is used to detect and match feature points in mid-wave infrared and short-wave infrared images, and then spatial alignment is achieved through perspective transformation.

[0050] Time period labeling: Based on the time information obtained from the images, label each image with the corresponding time period.

[0051] Image cropping and normalization: The aligned image is cropped into fixed-size image blocks, such as 256×256 pixels, and then the image blocks are normalized to normalize the pixel values ​​to the range of [0,1].

[0052] S3: Constructing a deep learning model that fuses multi-band and multi-temporal features:

[0053] Input layer: Receives preprocessed mid-wave infrared band images, short-wave infrared band images, and corresponding time period labels.

[0054] Feature extraction layer: An improved residual network is employed, consisting of 16 residual blocks. Each residual block comprises two convolutional layers, a batch normalization layer, and a ReLU activation function, with a channel attention mechanism introduced in each residual block. The channel attention mechanism obtains channel attention weights by performing global average pooling on the features of each channel, followed by a fully connected layer and a sigmoid activation function. These weights are then multiplied by the input features to achieve channel-dimensional attention weighting.

[0055] Multi-band, multi-temporal feature fusion layer: This includes band feature fusion and temporal feature fusion. First, the features obtained from the feature extraction layers of the mid-wave infrared and short-wave infrared images are fused within each band using an adaptive weight fusion strategy. A weight prediction network consisting of two fully connected layers predicts the weights of each feature, and then the weighted sum is used to obtain the band-fused feature. Next, temporal feature fusion is performed on the band-fused features from different time periods, again using an adaptive weight fusion strategy, to obtain the multi-band, multi-temporal fused feature.

[0056] Super-resolution reconstruction layer: It consists of 4 sub-pixel convolutional layers and 4 convolutional layers connected alternately. The scaling factor of the sub-pixel convolutional layers is 2. Each sub-pixel convolutional layer is followed by a convolutional layer. The convolutional layers use 3×3 convolutional kernels to optimize features.

[0057] Output layer: A 1×1 convolutional layer is used to map the features output by the super-resolution reconstruction layer to a high-resolution image.

[0058] S4: Train the deep learning model using the preprocessed dataset:

[0059] The preprocessed dataset is divided into a training set, a validation set, and a test set in a ratio of 7:2:1.

[0060] The model's initial learning rate was set to 0.001, using the Adam optimizer. The loss function was a weighted sum of the perceptual loss and the mean squared error loss, with both the perceptual loss and mean squared error loss having a weight of 0.5. The perceptual loss was calculated from the output of a layer in the pre-trained VGG16 network, and the mean squared error loss was the average of the squared differences between the reconstructed image and the real high-resolution image.

[0061] The model training employs a dynamic learning rate adjustment strategy, with the learning rate decreasing to 0.5% every 10 training epochs. The total training epochs are 100. Training is stopped and the trained super-resolution reconstruction model is saved when the loss on the validation set no longer decreases for 10 consecutive epochs.

[0062] S5: Input the low-resolution infrared remote sensing image to be reconstructed into the trained super-resolution reconstruction model, and output a high-resolution infrared remote sensing image. The low-resolution infrared remote sensing image to be reconstructed must contain mid-wave or short-wave infrared band information and corresponding time period information. After being input into the model, the corresponding high-resolution image is obtained through the model's feature extraction, fusion, and reconstruction processes.

[0063] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for super-resolution reconstruction of multi-temporal infrared images based on multi-band adaptive weighting and attention mechanisms, characterized in that, Includes the following steps: S1. Obtain a mid-wave and short-wave infrared image dataset, which includes mid-wave infrared band images and short-wave infrared band images, and covers 16 time periods formed by the intersection of the four seasons, morning, noon, evening and midnight; S2. Preprocess the dataset, including band alignment, time period labeling, and image cropping and normalization; S3. Construct a deep learning model for multi-band and multi-temporal feature fusion, the model including an input layer, a feature extraction layer, a multi-band and multi-temporal feature fusion layer, a super-resolution reconstruction layer, and an output layer; S4. Train the deep learning model using the preprocessed dataset to obtain a trained super-resolution reconstruction model. S5. Input the low-resolution infrared remote sensing image to be reconstructed into the trained super-resolution reconstruction model, and output a high-resolution infrared remote sensing image.

2. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S2, the band alignment adopts a feature point matching method to spatially align the mid-wave infrared band image and the short-wave infrared band image.

3. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S3, the feature extraction layer uses an improved residual network. An attention mechanism is introduced into each residual block to extract deep features of images from different bands and time periods. The improved residual network contains 16 residual blocks. Each residual block consists of two 3×3 convolutional layers (stride of 1, padding of 1), a batch normalization layer, and a ReLU activation function. The input features and output features are directly superimposed through skip connections. The attention mechanism is a channel attention mechanism, which performs global average pooling on each channel feature, generates channel weights through two fully connected layers (the hidden layer dimension is 1 / 4 of the number of channels) and a sigmoid activation function, with 3 input channels (including the band identifier channel) and 64 output channels.

4. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S3, the multi-band multi-time feature fusion layer adopts an adaptive weight fusion strategy, which assigns different weights to the feature importance of images in different bands and time periods for fusion.

5. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S3, the super-resolution reconstruction layer adopts a structure of alternating subpixel convolutional layers and convolutional layers to gradually improve the image resolution. The scaling factor of the subpixel convolutional layer is 4. Through the alternating connection of 4 subpixel convolutional and convolutional layers, the input feature map size is increased from 256×256 to 512×512 and 1024×1024. The convolutional layer connected after each subpixel convolutional layer adopts a 3×3 convolutional kernel (stride of 1, padding of 1), and the number of output channels is 64.

6. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S4, the loss function used in the training process is a weighted sum of perceptual loss and mean squared error loss; the perceptual loss is calculated based on the output of the conv3_3 layer of the pre-trained VGG16 network, and the weight ratio of perceptual loss to mean squared error loss is 1:

1.

7. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S4, a dynamic learning rate adjustment strategy is adopted during model training, gradually reducing the learning rate as the number of training rounds increases.

8. The method for multi-temporal infrared image super-resolution reconstruction based on multi-band adaptive weighting and attention mechanism according to claim 1, characterized in that, In step S5, the low-resolution infrared remote sensing image to be reconstructed is a mid-wave infrared band image or a short-wave infrared band image, and contains corresponding time period information.

9. The multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanism according to claim 4, characterized in that, In the adaptive weight fusion strategy, the fusion weights of each feature are calculated using the following formula: Let F be the i-th feature (from a certain band or time period). i Its fusion weight w i The formula for calculation is: in: ||F i ||2 is a feature F i The L2 norm is used to measure the activation strength of a feature itself; For time-dependent factors, Δt i α represents the time difference between the time period corresponding to the current feature and the target time period, where α is the time sensitivity coefficient. -σ(β·c i ) represents the band correlation factor, c i Indicates the band type (1 for mid-wave infrared, 0 for short-wave infrared), β is the band sensitivity coefficient, σ(x) = 1 / (1+e^(-1 / ... -x () is the sigmoid function; - The denominator is the sum of the weights of all features, achieving weight normalization, and satisfying the following conditions: The fused features are the weighted sum of each feature and its corresponding weight:

10. A multi-temporal infrared image super-resolution reconstruction method based on multi-band adaptive weighting and attention mechanism according to claim 9, characterized in that, The initial value range of the time sensitivity coefficient α is [0.1, 1.0], and the initial value range of the band sensitivity coefficient β is [0.5, 2.0]. Both are optimized through model training. The time difference Δt_i is calculated in hours (e.g., if the target time period is 8:00 AM and the time period corresponding to a certain feature is 4:00 AM, then Δt_i = 4).

Images