A super-resolution method based on remote sensing image degradation
By establishing a remote sensing image degradation model and training set, and combining spectral-spatial feature extraction blocks and channel attention modules, the generalization problem of remote sensing image super-resolution methods is solved, and better image reconstruction results are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHWEST UNIVERSITY FOR NATIONALITIES
- Filing Date
- 2023-02-13
- Publication Date
- 2026-07-03
AI Technical Summary
Existing remote sensing image super-resolution methods based on deep convolutional neural networks are mostly trained on a single degradation, resulting in poor network generalization and an inability to effectively handle real and complex remote sensing image degradation.
A degradation model for remote sensing images is established, training and testing sets are constructed, and the super-resolution model is trained using the Adam optimizer. The model is trained by combining L1 loss and InfoNCE loss with spectral-spatial feature extraction blocks and channel attention modules to generate high-resolution remote sensing images.
This improves the generalization ability of the remote sensing image super-resolution method, enabling it to better handle real and complex remote sensing image degradation and improve image reconstruction quality.
Smart Images
Figure CN115984116B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of remote sensing image processing, and in particular to a super-resolution method based on remote sensing image degradation. Background Technology
[0002] In recent years, deep convolutional neural networks have made significant progress in the field of super-resolution. Single-image super-resolution is a typical inverse problem, and its super-resolution performance is highly correlated with the degradation of the image itself. However, most existing deep convolutional neural network-based super-resolution methods are trained under the condition that the degradation is known. They typically assume that the low-resolution image is obtained by downsampling the high-resolution image using bicubic interpolation. This leads to a situation where many methods achieve good experimental results in research, but their performance degrades significantly in real-world scenarios. Therefore, blind super-resolution is an important research area. Summary of the Invention
[0003] The purpose of this invention is to address the shortcomings of current remote sensing image super-resolution techniques, which are mostly based on single degradation training, resulting in networks that cannot cope with real and complex degraded images and have poor generalization.
[0004] To address the aforementioned problems, one technical solution adopted by this invention is to propose a super-resolution method based on remote sensing image degradation, the super-resolution method comprising the following steps:
[0005] S1: Establishing a degradation model for remote sensing images: Compared to ordinary images, remote sensing images have interference factors such as geometric distortion, blurring effect, and noise interference. Based on these interference factors, a degradation model D for remote sensing images is established;
[0006] S2: Constructing Training and Test Sets: The training and test sets use the degradation model D from S1 to synthesize low-resolution images. Part of the blur kernel in D is modeled using isotropic and anisotropic Gaussian distributions, generating kernels with random parameters within a preset range; the other part is extracted from the images using KernelGAN. Noise in D is synthesized by extracting noise from the images.
[0007] S3: Constructing the super-resolution model m_1: The constructed super-resolution model consists of a degenerate representation learning network m_d and a super-resolution network m_s; the super-resolution network m_s consists of multiple sub-networks m_sub, each of which contains a feature extraction branch and a reconstruction branch. N spectral-spatial feature extraction blocks (SSFBs) are concatenated in the feature extraction branch, and the degenerate representation is fed into the SSFB to generate convolutional kernels. and channel ratio The deep feature map is obtained and then input into the upsampling module to obtain an enlarged feature map. This enlarged feature map is then added to the feature map obtained by direct enlargement through deconvolution in the reconstruction branch, serving as the input for the next level of the pyramid. In the final level of the pyramid, the result of adding the feature extraction branch and the reconstruction branch serves as the model's output.
[0008] S4: Model Training: Input the training set synthesized in S2 into the super-resolution model m_1 constructed in S3 for training. The Adam optimizer is used for training. The optimal network weights are obtained by minimizing the loss function, and finally the trained super-resolution model m_2 is obtained.
[0009] S5: Image Reconstruction: Input the low-resolution hyperspectral remote sensing image into the trained super-resolution model m_2, and the output of the model is the corresponding high-resolution hyperspectral remote sensing image.
[0010] The degradation model D in step S1 is given by the following formula:
[0011] (1);
[0012] in This indicates geometric distortion caused by the imaging device. The composite fuzzy kernel representing various fuzzy effects. This represents the downsampling process, where s represents the downsampling factor and N represents various types of noise.
[0013] The spectral-spatial feature extraction block SSFB in step S3 consists of two parts: the first part is a cascaded degenerate feature extraction module, which uses information in the degenerate representation to extract features; the second part is channel attention, which is used to extract spectral features.
[0014] The degradation feature extraction module consists of two branches, both of which receive degradation representations. The upper branch consists of two fully connected layers and one reshaping layer, used to generate the convolutional kernel. The lower branch represents the features and degradation. Input the cross-attention module to generate features with rich contextual information.
[0015] The cross-attention module mainly involves two steps: first, it performs a degenerate representation of the input. After 2 Convolutional layers, respectively generating , C, H, and W represent the number of channels, height, and width of the feature, respectively. Let be denoted as each position in the Q-space dimension. The vector at that location, Let's denote the extraction of the position from K. From feature vectors in the same row or column, obtain attention feature maps through the following operations. ,
[0016] (2);
[0017] in for and Relevance, Relevance After applying softmax, we obtain the attention feature map A. Then, we calculate the final features using the aggregation operation, which is defined as follows:
[0018] (3);
[0019] in To extract features from the input spectral-spatial feature extraction block SSFB, yes The feature vector at position u These are the scalar values at channel i and position u in A. yes After a The result of the convolutional layer, yes The scalar value at position u in channel i.
[0020] Channel attention is given by the following formula:
[0021] (4);
[0022] (5);
[0023] In formula (4) For input, This is the global average pooling function. and These are the sigmoid function and function, and For a convolutional layer, the channels are scaled by a ratio of r; in equation (5) and The first The scaling factor and feature map of each channel are adaptively scaled through channel attention for different bands.
[0024] The upsampling module described in step S3 consists of subpixel convolutions, which obtain high-resolution feature maps through convolution and recombination between multiple channels.
[0025] The loss function described in step S4 is based on the L1 loss, and incorporates the total spatial spectral variable loss and the InfoNCE loss for training.
[0026] (6);
[0027] (7);
[0028] (8);
[0029] (9);
[0030] In equation (6) , and These represent L1 loss, total spatial spectral variable loss, and InfoNCE loss, respectively; in equation (7) and Representing the Ground Truth image and the super-resolution result image respectively; in Equation (8) , , It is a calculation Horizontal, vertical, and spectral gradients; It is a predefined balance coefficient. In equation (9), B represents B low-resolution images, that is, B different degradations. This represents the number of samples in the queue. Let j represent the j-th negative sample.
[0031] The beneficial effects of this invention are: the designed network establishes a degradation model of remote sensing images based on the characteristics of remote sensing images, and uses a contrastive learning method to learn the degradation representation in remote sensing images. The learned degradation representation is then reshaped into convolutional kernels and weights in the super-resolution network as prior information for the super-resolution network. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating the method of the present invention.
[0033] Figure 2 This is a schematic diagram of the network model of the present invention.
[0034] Figure 3 Structural diagram of the spectral-space module
[0035] Figure 4 Structure diagram for comparative learning
[0036] Figure 5 Structure diagram of the degradation feature extraction module
[0037] Figure 6 This is a structural diagram of the channel attention module. Detailed Implementation
[0038] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following examples.
[0039] like Figure 1 As shown, this invention provides a super-resolution method based on remote sensing image degradation, the specific implementation process of which is as follows:
[0040] S1. Establish a degradation model for remote sensing images.
[0041] Compared with ordinary images, remote sensing images have interference factors such as geometric distortion, blurring effect, and noise interference. Based on these interference factors, a degradation model D of remote sensing images is established.
[0042] S101. Degeneration Model D
[0043] The model can be given by the following formula:
[0044] (10);
[0045] in This indicates geometric distortion caused by the imaging device. The composite fuzzy kernel representing various fuzzy effects. This represents the downsampling process, where s represents the downsampling factor and N represents various types of noise.
[0046] S2. Construct training and test sets
[0047] The training and test sets use the degradation model D in S1 to synthesize low-resolution images. The blur kernel in D is modeled using isotropic and anisotropic Gaussian distributions and generated with random parameters within a preset range. The noise in D is synthesized by extracting noise from the images.
[0048] S3. Constructing a super-resolution network model
[0049] like Figure 2As shown, this invention constructs a super-resolution network model m_1 for remote sensing images, capable of obtaining high-resolution images through the network model. The constructed super-resolution model consists of a degenerate representation learning network m_d and a super-resolution network m_s. The super-resolution network m_s consists of multiple sub-networks m_sub, each containing a feature extraction branch and a reconstruction branch. In the feature extraction branch, N spectral-spatial feature extraction blocks (SSFBs) are concatenated. The degenerate representation is fed into the SSFB to generate convolutional kernels and channel ratios, obtaining a deep feature map. This deep feature map is then input into an upsampling module to obtain an enlarged feature map, which is then added to the feature map obtained by deconvolution in the reconstruction branch as the input to the next level of the pyramid. In the final level of the pyramid, the result of adding the feature extraction branch and the reconstruction branch is used as the model's output.
[0050] S301. Degenerate Representation Learning
[0051] The degradation representation learning m_d is implemented using contrastive learning, randomly cropping two image patches from each image. These image patches are then encoded using an encoder. , It is the encoding of the first image block in the i-th image. For the i-th image, and These are referred to as the index and positive samples. Finally, the InfoNCE loss is used to narrow the distance between the positive samples and the index block, and to widen the distance between the negative samples and the index block. The InfoNCE loss is expressed by the formula:
[0052] (11);
[0053] Where B represents B low-resolution images, i.e. B distinct degradations. This represents the number of samples in the queue. Let j represent the j-th negative sample.
[0054] S302. Spectral-spatial feature extraction module.
[0055] The module consists of two parts: the first part is a cascaded degradation feature extraction module, which uses information from the degradation representation to extract features; the second part is channel attention, which is used to extract spectral features.
[0056] The degradation feature extraction module consists of two branches, both of which receive degradation representations. The upper branch consists of two fully connected layers and one reshaping layer, used to generate the convolutional kernel. The lower branch represents the features and degradation. Input the cross-attention module to generate features with rich contextual information.
[0057] The cross-attention module mainly involves two steps: first, it performs a degenerate representation of the input. After 2 Convolutional layers, respectively generating , C, H, and W represent the number of channels, height, and width of the feature, respectively. Let be denoted as each position in the Q-space dimension. The vector at that location, Let's denote the extraction of the position from K. From feature vectors in the same row or column, obtain attention feature maps through the following operations. ,
[0058] (12);
[0059] in for and Relevance, Relevance After applying softmax, we obtain the attention feature map A. Then, we calculate the final features using the aggregation operation, which is defined as follows:
[0060] (13);
[0061] in To extract features from the input spectral-spatial feature extraction block SSFB, yes The feature vector at position u These are the scalar values at channel i and position u in A. yes After a The result of the convolutional layer, yes The scalar value at position u in channel i.
[0062] Channel attention is given by the following formula:
[0063] (14);
[0064] (15);
[0065] In equation (14) For input, This is the global average pooling function. and These are the sigmoid function and function, and For a convolutional layer, the channels are scaled by a factor of r. In equation (25) and The first The scaling factor and feature map of each channel are adaptively scaled through channel attention for different bands.
[0066] S4. Model Training
[0067] The training set processed in S1 is input into the model m_1 constructed in S2 for training. The Adam optimizer is used for training, and the optimal network weights are obtained by minimizing the loss function, and finally the trained super-resolution network model m_2 is obtained.
[0068] S301. Loss Function
[0069] The loss function is based on L1 loss, and incorporates the total spatial spectrum variable loss and InfoNCE loss for training.
[0070] (16);
[0071] (17);
[0072] (18);
[0073] In equation (16) , and These represent the L1 loss, the total spatial spectral variable loss, and the InfoNCE loss, respectively. In Equation (17) and Representing the Ground Truth image and the super-resolution result image respectively; in Equation (18) , , It is a calculation Horizontal, vertical, and spectral gradients; It is a predefined balance coefficient.
[0074] S5. Image Reconstruction
[0075] Image reconstruction: The low-resolution hyperspectral remote sensing image is input into the trained super-resolution network model m_2, and the output of the model is the corresponding high-resolution hyperspectral remote sensing image.
[0076] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.
Claims
1. A method for super-resolution based on remote sensing image degradation, characterized in that, The super-resolution method includes the following steps: S1: Establish a degradation model for remote sensing images: Compared with ordinary images, remote sensing images have interference factors such as geometric distortion, blurring effect, and noise interference. Based on these interference factors, a degradation model D for remote sensing images is established. S2: Constructing the training and test sets: The training and test sets use the degradation model D from S1 to synthesize low-resolution images; part of the blur kernel in D is modeled using isotropic and anisotropic Gaussian distributions and generates kernels with random parameters within a preset range, while the other part is extracted from the image using KernelGAN; the noise in D is synthesized by extracting noise from the image. S3: Constructing the super-resolution model m_1: The constructed super-resolution model consists of a degenerate representation learning network m_d and a super-resolution network m_s; the super-resolution network m_s consists of multiple sub-networks m_sub, each of which contains a feature extraction branch and a reconstruction branch. N spectral-spatial feature extraction blocks (SSFBs) are concatenated in the feature extraction branch, and the degenerate representation is fed into the SSFB to generate convolutional kernels. and channel ratio The deep feature map is obtained and then input into the upsampling module to obtain the magnified feature map. Then, it is added to the feature map obtained by deconvolution in the reconstruction branch as the input of the next level pyramid. In the last level pyramid, the result of the feature extraction branch and the reconstruction branch is used as the output of the model. S4: Model Training: Input the training set synthesized in S2 into the super-resolution model m_1 constructed in S3 for training. The Adam optimizer is used for training. The optimal network weights are obtained by minimizing the loss function, and finally the trained super-resolution model m_2 is obtained. S5: Image Reconstruction: Input the low-resolution hyperspectral remote sensing image into the trained super-resolution model m_2, and the output of the model is the corresponding high-resolution hyperspectral remote sensing image; The degradation model D in step S1 is given by the following formula: (1)； in This indicates geometric distortion caused by the imaging device. The composite fuzzy kernel representing various fuzzy effects. This represents the downsampling process, where s represents the downsampling factor and N represents various types of noise. The degenerate representation learning m_d in step S3 is implemented by contrastive learning, which randomly crops two image patches from each image; then an encoder is used to encode these image patches into... , It is the encoding of the first image block in the i-th image. For the i-th image, and These are referred to as the index and positive samples. Finally, the InfoNCE loss is used to narrow the distance between the positive samples and the index block, and to widen the distance between the negative samples and the index block. The InfoNCE loss is expressed by the formula: (2)； Where B represents B low-resolution images, i.e. B distinct degradations. This represents the number of samples in the queue. This represents the j-th negative sample; The spectral-spatial feature extraction block (SSFB) in step S3 consists of two parts: the first part is a cascaded degenerate feature extraction module that uses information from the degenerate representation to extract features; the second part is channel attention, which is used to extract spectral features. The degradation feature extraction module consists of two branches, both of which receive degradation representations. The upper branch consists of two fully connected layers and one reshaping layer, used to generate the convolutional kernel. The lower branch represents the features and degradation. Input the cross-attention module to generate features with rich contextual information; The cross-attention module mainly involves two steps: first, it performs a degenerate representation of the input. After 2 Convolutional layers, respectively generating , C, H, and W represent the number of channels, height, and width of the feature, respectively. Let be denoted as each position in the Q-space dimension. The vector at that location, Let's denote the extraction of the position from K. From feature vectors in the same row or column, obtain attention feature maps through the following operations. , (3)； in for and Relevance, Relevance After applying softmax, we obtain the attention feature map A. Then, we calculate the final features using the aggregation operation, which is defined as follows: (4)； in To input the spectral-spatial feature extraction block SSFB, yes The feature vector at position u These are the scalar values at channel i and position u in A. yes After a The result of the convolutional layer, yes The scalar value at position u in channel i; Channel attention is given by the following formula: (5)； (6)； In formula (5) For input, This is the global average pooling function. and These are the sigmoid function and function, and For a convolutional layer, the channels are scaled by a ratio of r; in equation (6) and The first The scaling factor and feature map of each channel are adaptively scaled through channel attention for different bands; The upsampling module described in step S3 consists of subpixel convolutions, which obtain high-resolution feature maps through convolution and recombination between multiple channels; The loss function described in step S4 is based on the L1 loss, and incorporates the total spatial spectral variable loss and the InfoNCE loss for training. (7)； (8)； (9)； In equation (7) , and These represent L1 loss, total spatial spectral variable loss, and InfoNCE loss, respectively; in equation (8) and Representing the Ground Truth image and the super-resolution result image respectively; in equation (9) , , It is a calculation Horizontal, vertical, and spectral gradients; It is a predefined balance coefficient.