A remote sensing image super-resolution method and system for a classification task
By constructing a classification-super-resolution integrated deep learning network, and combining a pre-trained classification model with spectral attention blocks, the problem of poor performance of remote sensing image super-resolution methods in downstream tasks with hyperspectral data is solved, achieving high-precision super-resolution image reconstruction and improved classification accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2023-11-01
- Publication Date
- 2026-07-21
Smart Images
Figure CN117391962B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, specifically relating to a method and system for super-resolution of remote sensing images for classification tasks. Background Technology
[0002] Super-resolution (SR) aims to generate corresponding high-resolution (HR) images with rich details from existing low-resolution (LR) images. Specifically, in remote sensing image applications, because the devices used to acquire multispectral (MSI) and hyperspectral (HSI) images make trade-offs between spectral dimensions and spatial resolution, HSI and MSI images with the same spatial resolution are more expensive than RGB images. In recent years, many excellent super-resolution methods for hyperspectral images have emerged, demonstrating superior performance across various evaluation metrics. However, limited by traditional evaluation metrics, the reconstruction results of these methods perform poorly in common downstream tasks (DT) of hyperspectral data (such as image classification and anomaly detection). Therefore, the practical value of these super-resolution methods in production activities is far lower than their research value. Summary of the Invention
[0003] The purpose of this invention is to provide a method and system for super-resolution of remote sensing images for classification tasks, aiming to solve the technical problem that super-resolution methods in the prior art are disconnected from the downstream tasks they should serve.
[0004] To achieve the above objectives, the present invention provides the following solution:
[0005] A remote sensing image super-resolution method for classification tasks includes the following steps:
[0006] A classification task implementation model is constructed, which is used to extract the fusion features of the hyperspectral dataset and classify remote sensing images.
[0007] The model is pre-trained to perform the classification task.
[0008] Constructing a spectral branch-global hyperspectral super-resolution model;
[0009] Based on the pre-trained classification task implementation model and the spectral branch-global hyperspectral super-resolution model, a classification-super-resolution integrated deep learning network architecture is constructed.
[0010] Based on the aforementioned classification-super-resolution integrated deep learning network architecture, super-resolution hyperspectral image data is reconstructed.
[0011] Preferably, the classification task implementation model is implemented by a Unet; the Unet includes an encoder and a decoder, which are composed of nine 3x3 convolutional layers and four 2x2 max pooling layers or 2x2 upsampling convolutional layers, and include four residual connections;
[0012] The classification task implementation model uses an encoder to combine the hyperspectral dataset into a fused feature segment, extracts the deep feature sequence of the image, and then combines it with a decoder to achieve accurate pixel-by-pixel classification.
[0013] Preferably, the method for pre-training the classification task implementation model includes:
[0014] The PyTorch framework was used, with 100 epochs, the Adam optimizer, a learning rate of 1e-4, and cross-entropy used to calculate the classification loss.
[0015] Preferably, the spectral branch-global hyperspectral super-resolution model includes: two SSPN blocks and seven 3x3 grouped convolutions, wherein the SSPN block consists of three SAB blocks, and the operation formula is expressed as:
[0016]
[0017]
[0018] Where F SAB and F input These represent the input and output of the spectral attention extraction block, respectively, and Q, K, and V represent the query domain, key domain, and value domain of the attention mechanism. This represents three 1x1 convolutions connected in series. This represents two 3x3 convolutions connected in series, where α represents a learnable scaling parameter.
[0019] Preferably, the method for reconstructing super-resolution hyperspectral image data based on the aforementioned classification-super-resolution integrated deep learning network architecture includes:
[0020] Low-resolution sample images are pre-classified and fused with local spectral features to obtain fused spectral data.
[0021] Deep feature extraction is performed on the fused spectral data, and after the first upsampling, the multi-branch deep feature information is integrated to obtain complete hyperspectral data;
[0022] Feature extraction is performed on the complete hyperspectral data, and a second upsampling is performed to obtain a super-resolution hyperspectral image.
[0023] The present invention also provides a remote sensing image super-resolution system for classification tasks, comprising: a classification model building module, a pre-training module, a super-resolution model building module, a network architecture building module, and a reconstruction module;
[0024] The classification model building module is used to build a classification task implementation model, which is used to extract the fusion features of the hyperspectral dataset and classify remote sensing images.
[0025] The pre-training module is used to pre-train the classification task implementation model;
[0026] The super-resolution model construction module is used to construct a spectral branch-global hyperspectral super-resolution model;
[0027] The network architecture building module is used to construct a classification-super-resolution integrated deep learning network architecture based on the pre-trained classification task implementation model and the spectral branch-global hyperspectral super-resolution model.
[0028] The reconstruction module is used to reconstruct super-resolution hyperspectral image data based on the classification-super-resolution integrated deep learning network architecture.
[0029] Preferably, the classification task implementation model is implemented by a Unet; the Unet includes an encoder and a decoder, which are composed of nine 3x3 convolutional layers and four 2x2 max pooling layers or 2x2 upsampling convolutional layers, and include four residual connections;
[0030] The classification task implementation model uses an encoder to combine the hyperspectral dataset into a fused feature segment, extracts the deep feature sequence of the image, and then combines it with a decoder to achieve accurate pixel-by-pixel classification.
[0031] Preferably, the process of pre-training the classification task model includes:
[0032] The PyTorch framework was used, with 100 epochs, the Adam optimizer, a learning rate of 1e-4, and cross-entropy used to calculate the classification loss.
[0033] Preferably, the spectral branch-global hyperspectral super-resolution model includes: two SSPN blocks and seven 3x3 grouped convolutions, wherein the SSPN block consists of three SAB blocks, and the operation formula is expressed as:
[0034]
[0035]
[0036] Where F SAB and Finput These represent the input and output of the spectral attention extraction block, respectively, and Q, K, and V represent the query domain, key domain, and value domain of the attention mechanism. This represents three 1x1 convolutions connected in series. This represents two 3x3 convolutions connected in series, where α represents a learnable scaling parameter.
[0037] Preferably, the process of reconstructing super-resolution hyperspectral image data based on the aforementioned classification-super-resolution integrated deep learning network architecture includes:
[0038] Low-resolution sample images are pre-classified and fused with local spectral features to obtain fused spectral data.
[0039] Deep feature extraction is performed on the fused spectral data, and after the first upsampling, the multi-branch deep feature information is integrated to obtain complete hyperspectral data;
[0040] Feature extraction is performed on the complete hyperspectral data, and a second upsampling is performed to obtain a super-resolution hyperspectral image.
[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0042] This invention discloses a method and system for super-resolution of remote sensing images for classification tasks. It pre-classifies low-resolution sample images using a pre-trained classification model and fuses the classification with local spectral features. A branched spectral attention extraction block performs deep feature extraction, followed by an initial upsampling and integration of multi-branch deep feature information. Finally, a global spectral attention extraction block extracts features from the complete hyperspectral data and performs a second upsampling to obtain a super-resolution hyperspectral image. During the training phase, classification feedback guides the fusion of classification information and branched spectral feature information. This invention solves the technical problem of existing super-resolution methods being disconnected from the downstream tasks they should serve. Attached Figure Description
[0043] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 This is a schematic flowchart of a remote sensing image super-resolution method for classification tasks according to an embodiment of the present invention.
[0045] Figure 2 This is a network architecture diagram of a specific embodiment of the present invention;
[0046] Figure 3 This is a schematic diagram illustrating a specific implementation of the super-resolution model according to an embodiment of the present invention;
[0047] Figure 4 This is a schematic diagram of the SSPN block according to an embodiment of the present invention;
[0048] Figure 5 This is a schematic diagram of the spectral attention block (SSB) according to an embodiment of the present invention;
[0049] Figure 6 This is a schematic diagram of the Feedback Classification Model (FCM) according to an embodiment of the present invention. Detailed Implementation
[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0051] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the following description is provided in conjunction with the accompanying drawings and...
[0052] Example 1
[0053] like Figure 1 , Figure 2 As shown, this invention provides a remote sensing image super-resolution method for classification tasks, comprising the following steps:
[0054] Construct a model to implement the classification task;
[0055] Pre-trained classification task implementation model;
[0056] Constructing a spectral branch-global hyperspectral super-resolution model;
[0057] Based on the pre-trained classification task implementation model and the spectral branch-global hyperspectral super-resolution model, a classification-super-resolution integrated deep learning network architecture is constructed.
[0058] Based on a classification-super-resolution integrated deep learning network architecture, hyperspectral image data with high classification accuracy is reconstructed.
[0059] In this embodiment, the specific implementation of the classification task model (specific classification task classification module) can adopt different model structures depending on the different tasks (classification task or anomaly detection task). This embodiment is implemented by a Unet, which is trained using a full-fit method to accurately fit the specified distribution; the Unet includes an encoder and a decoder, which are composed of nine 3x3 convolutional layers and four 2x2 max-pooling layers or 2x2 upsampling convolutional layers, and include four residual connections.
[0060] The classification task implementation model uses an encoder to combine the hyperspectral dataset into a fused feature segment, extracts the deep feature sequence of the image, and then combines it with a decoder to achieve accurate pixel-by-pixel classification.
[0061] Specifically, with Figure 6 The network shown is an example. In this step, a deep learning network channel with the same input and output dimensions is built to extract the implicit global category information from the hyperspectral data. See also Figure 6 As shown, after inputting hyperspectral data, it first passes through two 7x7 convolutional layers to extract shallow features with a large field of view, then passes through a 2x2 max pooling layer for downsampling, and then passes through convolutional layers and pooling layers to reach a fully connected layer to extract deep global category features. Then, it passes through a deconvolutional layer for upsampling to restore the data, and then passes through convolutional layers and deconvolutional layers to restore it to the input spatial size. Finally, the fully connected layer merges the sorted features into classification information.
[0062] In this embodiment, the pre-trained classification task implementation model is trained with a complete dataset before testing the super-resolution model to ensure that the classification task implementation model provides accurate global classification information. The method for pre-training the classification task implementation model includes:
[0063] The PyTorch framework was used, with 100 epochs, the Adam optimizer, a learning rate of 1e-4, and cross-entropy used to calculate the classification loss.
[0064] In this embodiment, the spectral branch-global hyperspectral super-resolution model (local spectral attention feature extraction block and global spectral attention feature extraction block) includes: two SSPN blocks and seven 3x3 grouped convolutions, wherein the SSPN block consists of three SAB blocks, and its operation formula is expressed as:
[0065]
[0066]
[0067] Where F SAB and F inputThese represent the input and output of the spectral attention extraction block, respectively, and Q, K, and V represent the query domain, key domain, and value domain of the attention mechanism. This represents three 1x1 convolutions connected in series. This represents two 3x3 convolutions connected in series, where α represents a learnable scaling parameter.
[0068] In this embodiment, the method for reconstructing super-resolution hyperspectral image data based on the classification-super-resolution integrated deep learning network architecture includes:
[0069] By using a pre-trained classification model to pre-classify low-resolution sample images and fusing them with local spectral features, deep feature extraction is performed by a branch spectral attention extraction block. After the first upsampling, the multi-branch deep feature information is integrated. Finally, the global spectral attention extraction block extracts features from the complete hyperspectral data and performs a second upsampling to obtain a super-resolution hyperspectral image.
[0070] During the training phase, classification feedback guides the fusion of classification information and branch spectral feature information: local shallow features of the super-resolution hyperspectral data to be treated are combined with global classification data to obtain super-resolution results with implicit classification information, and the training process of the super-resolution model is guided by the feedback classification model.
[0071] Specifically, construct local spectral attention extraction blocks and global spectral attention extraction blocks, see [link to relevant documentation]. Figure 3 As shown, the local spectral attention extraction block first receives the original low-resolution hyperspectral data in blocks and performs shallow feature extraction. Then, it integrates the block-level shallow features and global classification features and performs a second shallow feature extraction. Finally, the data is handed over to the deep spectral feature extraction block for deep feature extraction. For details, see [link to implementation details]. Figure 4 It consists of three spectral attention blocks and one residual connection. For the specific implementation of the spectral attention block SSB, please refer to [link to implementation details]. Figure 5 It consists of two 3x3 convolutions, four 1x1 convolutions, and an attention block.
[0072] Furthermore, ablation experiments were conducted on embodiments of the present invention. When testing the performance of a module, the basic modules and network structures used in different experimental projects within the same group were always kept identical, and the complete super-resolution functionality of different experimental projects was ensured to stabilize non-experimental variables. The image reconstruction magnification selected for the experiment was 4x, and the performance of the training strategy and additional modules of the present invention were tested respectively. The ablation experiments were performed on the Houston dataset.
[0073] Table 1
[0074]
[0075] The ablation experiment data in Table 1 show that this embodiment makes the reconstruction results perform close to the performance of the original image on the classification model, and also shows that the performance improvement is gradual as our method is added.
[0076] This embodiment was compared with other advanced super-resolution methods on three public datasets. The comparison results are shown in Table 2. While the traditional performance is on par with advanced methods, this embodiment has a much higher classification accuracy than other methods and is close to the performance of the original image on the classification model.
[0077] It is evident that the hyperspectral image super-resolution method proposed in this invention for classification tasks can significantly improve the classification accuracy of super-resolution results in classification models, while maintaining first-class peak signal-to-noise ratio and structural similarity.
[0078] Table 2
[0079]
[0080] Example 2
[0081] The present invention also provides a remote sensing image super-resolution system for classification tasks, comprising: a classification model building module, a pre-training module, a super-resolution model building module, a network architecture building module, and a reconstruction module;
[0082] The classification model building module is used to build models for classification tasks;
[0083] The pre-training module is used to pre-train the model for classification tasks;
[0084] The super-resolution model building module is used to build spectral branch-global hyperspectral super-resolution models;
[0085] The network architecture building module is used to implement models and spectral branch-global hyperspectral super-resolution models based on pre-trained classification tasks, and to build a classification-super-resolution integrated deep learning network architecture.
[0086] The reconstruction module is used to reconstruct hyperspectral image data with high classification accuracy based on a classification-super-resolution integrated deep learning network architecture.
[0087] In this embodiment, the classification task implementation model is implemented by a Unet; the Unet includes an encoder and a decoder, which are composed of nine 3x3 convolutional layers and four 2x2 max pooling layers or 2x2 upsampling convolutional layers, and include four residual connections.
[0088] The classification task implementation model uses an encoder to combine the hyperspectral dataset into a fused feature segment, extracts the deep feature sequence of the image, and then combines it with a decoder to achieve accurate pixel-by-pixel classification.
[0089] In this embodiment, the process of pre-training the classification task to implement the model includes:
[0090] The PyTorch framework was used, with 100 epochs, the Adam optimizer, a learning rate of 1e-4, and cross-entropy used to calculate the classification loss.
[0091] In this embodiment, the spectral branch-global hyperspectral super-resolution model includes: two SSPN blocks and seven 3x3 grouped convolutions, wherein the SSPN block consists of three SAB blocks, and its operation formula is expressed as:
[0092]
[0093]
[0094] Where F SAB and F input These represent the input and output of the spectral attention extraction block, respectively, and Q, K, and V represent the query domain, key domain, and value domain of the attention mechanism. This represents three 1x1 convolutions connected in series. This represents two 3x3 convolutions connected in series, where α represents a learnable scaling parameter.
[0095] In this embodiment, the process of reconstructing super-resolution hyperspectral image data based on a classification-super-resolution integrated deep learning network architecture includes:
[0096] Low-resolution sample images are pre-classified and fused with local spectral features to obtain fused spectral data.
[0097] Deep feature extraction is performed on the fused spectral data, and after the first upsampling, the multi-branch deep feature information is integrated to obtain complete hyperspectral data;
[0098] Feature extraction is performed on the complete hyperspectral data, and a second upsampling is performed to obtain a super-resolution hyperspectral image.
[0099] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A remote sensing image super-resolution method for classification tasks, characterized in that, Includes the following steps: A classification task implementation model is constructed, which is used to extract the fusion features of the hyperspectral dataset and classify remote sensing images. The model is pre-trained to perform the classification task. Constructing a spectral branch-global hyperspectral super-resolution model; Based on the pre-trained classification task implementation model and the spectral branch-global hyperspectral super-resolution model, a classification-super-resolution integrated deep learning network architecture is constructed. Based on the aforementioned classification-super-resolution integrated deep learning network architecture, super-resolution hyperspectral remote sensing image data is reconstructed; The classification task implementation model is implemented by a Unet; the Unet includes an encoder and a decoder, which are composed of nine 3x3 convolutional layers and four 2x2 max pooling layers or 2x2 upsampling convolutional layers, and include four residual connections. The classification task implementation model uses an encoder to combine the hyperspectral dataset into a fused feature segment, extracts the deep feature sequence of the image, and then combines it with a decoder to achieve accurate pixel-by-pixel classification. Based on the aforementioned classification-super-resolution integrated deep learning network architecture, the method for reconstructing super-resolution hyperspectral remote sensing image data includes: By using a pre-trained classification model to pre-classify low-resolution sample images and fuse them with local spectral features, deep feature extraction is performed by a branch spectral attention extraction block, and after the first upsampling, multi-branch deep feature information is integrated. Finally, a global spectral attention extraction block is used to extract features from the complete hyperspectral data and perform a second upsampling to obtain a super-resolution hyperspectral image. During the training phase, classification feedback guides the fusion of classification information and branch spectral feature information: local shallow features of the super-resolution hyperspectral data to be treated are combined with global classification data to obtain super-resolution results with implicit classification information, and the training process of the super-resolution model is guided by the feedback classification model.
2. The remote sensing image super-resolution method for classification tasks according to claim 1, characterized in that, Methods for pre-training the classification task model include: The PyTorch framework was used, with 100 epochs, the Adam optimizer, a learning rate of 1e-4, and cross-entropy used to calculate the classification loss.
3. The remote sensing image super-resolution method for classification tasks according to claim 1, characterized in that, The spectral branch-global hyperspectral super-resolution model includes: two SSPN blocks and seven 3x3 grouped convolutions, wherein the SSPN block consists of three SAB blocks, and the operation formula is expressed as follows: ； ； in FSAB and Finput These represent the input and output of the spectral attention extraction block, respectively. Q , K , V The query domain, key domain, and value domain represent the attention mechanism. This represents three 1x1 convolutions connected in series. This represents two concatenated 3x3 convolutions. This represents a learnable scaling parameter.
4. A remote sensing image super-resolution system for classification tasks, characterized in that, include: The system includes a classification model building module, a pre-training module, a super-resolution model building module, a network architecture building module, and a reconstruction module. The classification model building module is used to build a classification task implementation model, which is used to extract the fusion features of the hyperspectral dataset and classify remote sensing images. The pre-training module is used to pre-train the classification task implementation model; The super-resolution model construction module is used to construct a spectral branch-global hyperspectral super-resolution model; The network architecture building module is used to construct a classification-super-resolution integrated deep learning network architecture based on the pre-trained classification task implementation model and the spectral branch-global hyperspectral super-resolution model. The reconstruction module is used to reconstruct super-resolution hyperspectral remote sensing image data based on the classification-super-resolution integrated deep learning network architecture. The classification task implementation model is implemented by a Unet; the Unet includes an encoder and a decoder, which are composed of nine 3x3 convolutional layers and four 2x2 max pooling layers or 2x2 upsampling convolutional layers, and include four residual connections. The classification task implementation model uses an encoder to combine the hyperspectral dataset into a fused feature segment, extracts the deep feature sequence of the image, and then combines it with a decoder to achieve accurate pixel-by-pixel classification. Based on the aforementioned classification-super-resolution integrated deep learning network architecture, the method for reconstructing super-resolution hyperspectral remote sensing image data includes: By using a pre-trained classification model to pre-classify low-resolution sample images and fuse them with local spectral features, deep feature extraction is performed by a branch spectral attention extraction block, and after the first upsampling, multi-branch deep feature information is integrated. Finally, a global spectral attention extraction block is used to extract features from the complete hyperspectral data and perform a second upsampling to obtain a super-resolution hyperspectral image. During the training phase, classification feedback guides the fusion of classification information and branch spectral feature information: local shallow features of the super-resolution hyperspectral data to be treated are combined with global classification data to obtain super-resolution results with implicit classification information, and the training process of the super-resolution model is guided by the feedback classification model.
5. The remote sensing image super-resolution system for classification tasks according to claim 4, characterized in that, The process of pre-training the model for the classification task includes: The PyTorch framework was used, with 100 epochs, the Adam optimizer, a learning rate of 1e-4, and cross-entropy used to calculate the classification loss.
6. The remote sensing image super-resolution system for classification tasks according to claim 4, characterized in that, The spectral branch-global hyperspectral super-resolution model includes: two SSPN blocks and seven 3x3 grouped convolutions, wherein the SSPN block consists of three SAB blocks, and the operation formula is expressed as follows: ； ； in FSAB and Finput These represent the input and output of the spectral attention extraction block, respectively. Q , K , V The query domain, key domain, and value domain represent the attention mechanism. This represents three 1x1 convolutions connected in series. This represents two concatenated 3x3 convolutions. This represents a learnable scaling parameter.