Panchromatic sharpening method based on auxiliary feature extraction for super-resolution reconstruction task

Abstract

The present invention relates to a panchromatic sharpening method based on auxiliary feature extraction of super-resolution reconstruction tasks, and belongs to the field of remote sensing image processing. The present invention comprises the steps of: constructing a multi-stage PAN feature extraction module and a multi-stage LRMS super-resolution module, and inputting two kinds of image data after preprocessing into corresponding modules respectively; constructing a multi-stage feature fusion module, fusing image features from the PAN feature extraction module and the LRMS super-resolution module, and constructing a feature interaction relationship between modules of different images; constructing two reconstruction modules with the same structure but no shared parameters, and reconstructing and restoring the output features of the panchromatic sharpening task and the super-resolution reconstruction task respectively, and generating a super-resolution MS image and a high-resolution multi-spectral HRMS image. The present invention provides additional constraints for the panchromatic sharpening task by introducing a super-resolution task, encourages the network module to have a stronger feature extraction capability, and improves the performance of the panchromatic sharpening task.

Description

Technical Field

[0001] The invention relates to a full color sharpening method based on super-resolution reconstruction task auxiliary feature extraction, and belongs to the field of remote sensing image processing. Background Art

[0002] With the successful launch of a large number of artificial satellites and the development of remote sensing imaging technology, the application of remote sensing images in production and life is becoming more and more important. Due to the limitations of various physical factors, remote sensing satellites can usually only obtain single-modality images, namely, panchromatic PAN (Panchromatic) images and low-resolution multispectral LRMS (Low-resolution Multispectral) images. PAN images contain rich spatial information, while LRMS images contain rich spectral information. In order to obtain high-resolution multispectral HRMS (High-resolution Multispectral) images that contain both types of information, the relevant fusion algorithm is designed to fuse PAN and LRMS images to integrate the advantages of both. This technology is called panchromatic sharpening. At present, panchromatic sharpening technology based on deep learning methods has become mainstream. However, most of the existing panchromatic sharpening methods are mainly designed based on single-task drive, which leads to the limitations of the features extracted by the network module and limits the performance improvement. To address this problem, a panchromatic sharpening method based on super-resolution reconstruction task-assisted feature extraction is proposed. Summary of the invention

[0003] In order to address the shortcomings of the existing methods, the present invention aims at the problem that the features extracted by the network module are limited, and proposes a full-color sharpening method based on super-resolution reconstruction task auxiliary feature extraction, thereby improving the performance of the full-color sharpening task.

[0004] The technical solution of the present invention is: a full color sharpening method based on super-resolution reconstruction task auxiliary feature extraction, and the specific steps of the method are as follows:

[0005] Step 1: Obtain panchromatic PAN and low-resolution multispectral LRMS image data and preprocess the images;

[0006] Step 2: Construct a multi-stage PAN feature extraction module and a multi-stage LRMS super-resolution module, and input the two types of image data after preprocessing into the corresponding modules respectively;

[0007] Step 3: Construct a multi-stage feature fusion module to fuse the image features from the PAN feature extraction module and the LRMS super-resolution module, and build a feature interaction relationship between modules of different images;

[0008] Step 4: Construct two reconstruction modules with the same structure but no shared parameters, respectively reconstruct and restore the output features of the full color sharpening task and the super-resolution reconstruction task, and generate corresponding experimental results, that is, generate super-resolution MS images and high-resolution multispectral HRMS (High-resolution Multispectral) images;

[0009] Step 5: Train the PAN feature extraction module, LRMS super-resolution module, feature fusion module and reconstruction module;

[0010] Step 6: All trained modules can be directly input into PAN and LRMS images for testing to generate corresponding experimental results.

[0011] The Step 5 specifically includes:

[0012] Step 51, use Adam optimizer to optimize the parameters in PAN feature extraction module, LRMS super-resolution module, feature fusion module and reconstruction module;

[0013] Step 52, use L1 loss function to optimize all modules, assuming that the full color sharpening result HRMS image output by the reconstruction module is represented by Y j , the super-resolution reconstruction result high-definition MS image is expressed as The label image is G j . Then the loss functions of the two tasks are:

[0014]

[0015] in, and They represent the loss functions of the full-color sharpening task and the super-resolution reconstruction task respectively. Since the main task of the network is full-color sharpening, the total loss function of the network is expressed as:

[0016]

[0017] Among them, λ is the balance coefficient, and its value is 0.5.

[0018] Furthermore, in the Step 1, the PAN image is cropped to a spatial size of 64×64, and the LRMS image is cropped to a spatial size of 16×16. They are normalized before being input into the network, and the LRMS image is upsampled to a spatial size of 64×64 in the preprocessing stage. Finally, the shallow features of the two images are extracted through a 3×3 convolution operation.

[0019] Furthermore, there are 4 PAN feature extraction modules in the Step 2, which extract multi-scale image spatial features through convolution operations of different sizes; there are 4 LRMS super-resolution modules, which realize image super-resolution through cross-attention and use Swin Transformer to extract global image features.

[0020] Furthermore, the specific operation process of the PAN feature extraction module and the LRMS super-resolution module in Step 2 is as follows:

[0021] Assume that the PAN image features output in the preprocessing stage are The input of the i-th (i=1, 2, 3, 4) PAN feature extraction module is The output is The PAN feature extraction module first extracts multi-scale features through 3×3, 5×5 and 7×7 convolution operations, then concatenates the three sets of features, performs dimensionality reduction through a 1×1 convolution, and finally combines them with the input features. Add element by element to get the output features

[0022] Assume that the LRMS image features output in the preprocessing stage are The input of the i-th (i=1, 2, 3, 4) LRMS super-resolution module is The output is LRMS super-resolution module Compared with the feature extraction module from PAN As input. First, use depth-wise separable convolution and 1×1 convolution operations to use Generate Q i , Generate K i With V i . Then the super-resolution process based on cross attention is expressed as:

[0023]

[0024] in, is the super-resolution feature, a is the control parameter, T represents the transposition operation, Q i , K i 、V i Respectively represent the query, key, and value used in the cross-attention operation of the super-resolution process. Then two consecutive Swin Transformers are used to extract global features and finally Add element by element to get the output features of the LRMS super-resolution module

[0025] Furthermore, the specific operation steps of the multi-stage feature fusion module in Step 3 are as follows:

[0026] The features from the PAN feature extraction module and the LRMS super-resolution module are and The two sets of features are concatenated and input into the i-th (i=1, 2, 3, 4) feature fusion module. The output of this module is This module performs two self-attention calculations in parallel. The image feature dimension of the channel dimension is B×C×H×W, and the attention calculation generates a C×C attention weight map to enhance the channel feature. The process is expressed as:

[0027]

[0028] in, represents the image features enhanced in channel dimension, [.] represents the feature concatenation operation, They represent the query, key, and value for the self-attention operation in the channel dimension respectively. The dimension of the image feature in the spatial dimension is rearranged from B×C×H×W to B×H×W×C, and restored to B×C×H×W after the attention calculation is completed. The attention calculation generates an HW×HW attention weight map to enhance the spatial feature. The process is expressed as:

[0029]

[0030] in, Represents the image features with enhanced spatial dimension, Respectively represent the query, key, and value for the spatial dimension self-attention operation. The two sets of features after enhancement are concatenated and input into a gated convolution operation for feature regulation to reduce redundant information. The output feature of this module is Finally, the output features of multiple feature fusion modules are concatenated and the final fusion features are obtained by 1×1 convolution dimensionality reduction.

[0031] The beneficial effects of the present invention are:

[0032] 1. Since the features extracted by the network design driven by the single task of panchromatic sharpening have limitations, the present invention adds an auxiliary super-resolution reconstruction network to the panchromatic sharpening task. There is feature interaction between the super-resolution reconstruction network and the panchromatic sharpening network, that is, the features output by the PAN feature extraction module must participate in the LRMS super-resolution module, and the features output by the LRMS super-resolution module must participate in the feature fusion module. Under the constraints of the loss functions of both parties, the features output by the PAN feature extraction module and the LRMS super-resolution module are required to meet both tasks at the same time, forming a mutually reinforcing relationship between them. This approach improves the comprehensiveness of the features extracted by the modules in each task, which is critical to improving task performance.

[0033] 2. Most existing full-color sharpening methods directly splice two image features to complete the fusion in the feature fusion stage, without further exploring the fusion features, which affects the reconstruction quality. The present invention proposes a feature fusion module, which applies different calculation methods to the channel dimension and spatial dimension of the fusion feature in a parallel manner, fully explores the potential information of the fusion feature in different dimensions, and can further improve the reconstruction quality.

[0034] 3. In the network optimization stage, the present invention applies loss functions to the full color sharpening and super-resolution reconstruction results respectively, and balances the importance of the two tasks by means of a balance coefficient. While highlighting the main task, the auxiliary task is not ignored, thereby ensuring the stable performance of the two tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the process structure of the method of the present invention;

[0036] Figure 2 It is a feature fusion structure diagram of the method of the present invention;

[0037] Figure 3 The comparison results of the method of the present invention with different methods on the low-resolution test set;

[0038] Figure 4 This is the comparison result between the proposed method and different methods on the full-resolution test set. DETAILED DESCRIPTION

[0039] Example 1: Figure 1-Figure 2 As shown, a full color sharpening method based on super-resolution reconstruction task auxiliary feature extraction, the specific steps of the method are as follows:

[0040] Step 1: Obtain PAN and LRMS image data and preprocess the images: crop the PAN image to a spatial size of 64×64 and the LRMS image to a spatial size of 16×16, normalize them before entering the network, and upsample the LRMS image to a spatial size of 64×64 in the preprocessing stage. Finally, extract the shallow features of the two images through a 3×3 convolution operation.

[0041] Step 2: Construct a multi-stage PAN feature extraction module and a multi-stage LRMS super-resolution module, and input the two types of image data after preprocessing into the corresponding modules respectively.

[0042] There are 4 PAN feature extraction modules in the Step 2, which extract multi-scale image spatial features through convolution operations of different sizes; there are 4 LRMS super-resolution modules, which realize image super-resolution through cross-attention and use SwinTransformer to extract global image features.

[0043] The specific operation process of the PAN feature extraction module and the LRMS super-resolution module in Step 2 is as follows:

[0044] Assume that the PAN image features output in the preprocessing stage are The input of the i-th (i=1, 2, 3, 4) PAN feature extraction module is The output is The PAN feature extraction module first extracts multi-scale features through 3×3, 5×5 and 7×7 convolution operations, then concatenates the three sets of features, performs dimensionality reduction through a 1×1 convolution, and finally combines them with the input features. Add element by element to get the output features

[0045] Assume that the LRMS image features output in the preprocessing stage are The input of the i-th (i=1, 2, 3, 4) LRMS super-resolution module is The output is LRMS super-resolution module Compared with the feature extraction module from PAN As input. First, use depth-wise separable convolution and 1×1 convolution operations to use Generate Q i , Generate K i With V i . Then the super-resolution process based on cross attention is expressed as:

[0046]

[0047] in, is the super-resolution feature, a is the control parameter, T represents the transposition operation, Q i , K i 、V i Respectively represent the query, key, and value used in the cross-attention operation of the super-resolution process. Then two consecutive Swin Transformers are used to extract global features and finally Add element by element to get the output features of the LRMS super-resolution module

[0048] Step 3: Construct a multi-stage feature fusion module to fuse the image features from the PAN feature extraction module and the LRMS super-resolution module, and build a feature interaction relationship between modules of different images.

[0049] The specific operation steps of the multi-stage feature fusion module in Step 3 are as follows:

[0050] The features from the PAN feature extraction module and the LRMS super-resolution module are and The two sets of features are concatenated and input into the i-th (i=1, 2, 3, 4) feature fusion module. The output of this module is This module performs two self-attention calculations in parallel. The image feature dimension of the channel dimension is B×C×H×W, and the attention calculation generates a C×C attention weight map to enhance the channel feature. The process is expressed as:

[0051]

[0052] in, represents the image features with enhanced channel dimension, and [.] represents the feature concatenation operation. They represent the query, key, and value for the self-attention operation in the channel dimension respectively. The dimension of the image feature in the spatial dimension is rearranged from B×C×H×W to B×H×W×C, and restored to B×C×H×W after the attention calculation is completed. The attention calculation generates an HW×HW attention weight map to enhance the spatial feature. The process is expressed as:

[0053]

[0054] in, Represents the image features with enhanced spatial dimension, Respectively represent the query, key, and value for the spatial dimension self-attention operation. The two sets of features after enhancement are concatenated and input into a gated convolution operation for feature regulation to reduce redundant information. The output feature of this module is Finally, the output features of multiple feature fusion modules are concatenated and the final fusion features are obtained by 1×1 convolution dimensionality reduction.

[0055] Step 4: Construct two reconstruction modules with the same structure but non-shared parameters, reconstruct and restore the output features of the full color sharpening task and the super-resolution reconstruction task respectively, and generate corresponding experimental results.

[0056] The specific operation steps of the reconstruction module in Step 4 are as follows:

[0057] The reconstruction module respectively outputs the final features of the LRMS super-resolution module And the final fusion features output by multiple feature fusion modules The main structure of the reconstruction module is 5 cascaded residual blocks, and finally a 1×1 convolution is used to reduce the dimension and output the reconstructed image, namely the super-resolution reconstruction result super-resolution MS image and the full color sharpening result HRMS image.

[0058] Step 5: Train the PAN feature extraction module, LRMS super-resolution module, feature fusion module and reconstruction module. Step 5 specifically includes:

[0059] Step 51, use Adam optimizer to optimize the parameters in PAN feature extraction module, LRMS super-resolution module, feature fusion module and reconstruction module;

[0060] Step 52, use L1 loss function to optimize all modules, assuming that the full color sharpening result HRMS image output by the reconstruction module is represented by Y j , the super-resolution reconstruction result high-definition MS image is expressed as The label image is G j . Then the loss functions of the two tasks are:

[0061]

[0062] in, and They represent the loss functions of the full-color sharpening task and the super-resolution reconstruction task respectively. Since the main task of the network is full-color sharpening, the total loss function of the network is expressed as:

[0063]

[0064] Among them, λ is the balance coefficient, and its value is 0.5.

[0065] Step 6: All trained modules can be directly input into PAN and LRMS images for testing to generate corresponding experimental results.

[0066] Furthermore, to verify the effectiveness of the method of the present invention, the performance of the proposed method was evaluated on the GaoFen-2 (GF-2) satellite dataset. The dataset contains 22,010 PAN and LRMS image pairs. Data simulation is performed according to the Wald protocol, and then according to the dataset partitioning rule, the dataset can be divided into a training set, a validation set, and a test set. The algorithm of the present invention was developed under the Pytorch framework and trained on an NVIDIA GTX3090 graphics card. During training, the Adam optimizer was used to optimize the model parameters. During this process, the batch size was set to 16, the learning rate was fixed to 0.0002, and a total of 200 rounds of training were performed.

[0067] Furthermore, the present invention uses SAM (spectral angle mapper), ERGAS (erreur relative global adimensionnellede syntheses), SCC (spatial correlation coefficient) and Q2n (multi-band extension of the universal image quality index) as evaluation indicators to measure the performance of different methods on a low-resolution test set. λ And the spatial distortion coefficient D s It is used as an evaluation metric to measure the performance of different methods on the full-resolution test set.

[0068] Furthermore, on the GF-2 dataset, the performance of the proposed method is compared with that of the panchromatic sharpening methods such as PNN, PanNet, MSDCNN, BDPN, FusionNet and S2DBPN, and the experimental results are shown in Table 1. It can be seen that the proposed method achieves 0.6998, 0.6133, 0.9924 and 0.9821 in the low-resolution indicators SAM, ERGAS, SCC and Q4, respectively, and achieves 0.6998, 0.6133, 0.9924 and 0.9821 in the full-resolution indicators QNR, D λ and D s This reaches 0.9435, 0.0143 and 0.0429 respectively, which proves the effectiveness of the method of the present invention in the full color sharpening task.

[0069] Table 1 shows the objective evaluation comparison of different methods on the GF-2 dataset

[0070]

[0071] Further, Figure 3 and Figure 4 The visual effect comparison between the proposed method and different methods on two scales of GF-2 test sets is shown. It can be seen that compared with other comparison methods, the proposed method can generate more accurate spatial details on two scales of test sets and retain rich spectral information, which has stronger superiority.

[0072] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A full color sharpening method based on super-resolution reconstruction task-assisted feature extraction, characterized by: The specific steps of the method are as follows: Step 1: Obtain PAN and LRMS image data and preprocess the images; Step 2: Construct a multi-stage PAN feature extraction module and a multi-stage LRMS super-resolution module, and input the two types of image data after preprocessing into the corresponding modules respectively; Step 3: Construct a multi-stage feature fusion module to fuse the image features from the PAN feature extraction module and the LRMS super-resolution module, and build a feature interaction relationship between modules of different images; Step 4: Construct two reconstruction modules with the same structure but no shared parameters, respectively reconstruct and restore the output features of the full color sharpening task and the super-resolution reconstruction task, and generate corresponding experimental results; Step 5: Train the PAN feature extraction module, LRMS super-resolution module, feature fusion module and reconstruction module; Step 6: All trained modules are directly input into PAN and LRMS images for testing to generate corresponding experimental results; The specific operation process of the PAN feature extraction module and the LRMS super-resolution module in Step 2 is as follows: Assume that the PAN image features output in the preprocessing stage are The input of the i-th PAN feature extraction module is The output is Where i = 1, 2, 3, 4, The PAN feature extraction module first extracts multi-scale features through 3×3, 5×5 and 7×7 convolution operations, then concatenates the three sets of features, performs dimensionality reduction through a 1×1 convolution, and finally combines them with the input features. Add element by element to get the output features Assume that the LRMS image features output in the preprocessing stage are The input of the i-th LRMS super-resolution module is The output is Where i = 1, 2, 3, 4, LRMS super-resolution module Compared with the feature extraction module from PAN As input; first use depthwise separable convolution and 1×1 convolution operations to utilize Generate Q i , Generate K i With V i , then the super-resolution process based on cross attention is expressed as: in, is the super-resolution feature, a is the control parameter, T represents the transposition operation, Q i , K i 、V i Represent the query, key and value of the cross-attention operation used in the super-resolution process, respectively. Then two consecutive SwinTransformers are used to extract global features, and finally Add element by element to get the output features of the LRMS super-resolution module The specific operation steps of the multi-stage feature fusion module in Step 3 are as follows: The features from the PAN feature extraction module and the LRMS super-resolution module are and The two sets of features are concatenated and input into the i-th feature fusion module, where i = 1, 2, 3, 4. The output of this module is This module performs two self-attention calculations in parallel. The image feature dimension of the channel dimension is B×C×H×W. The attention calculation generates a C×C attention weight map to enhance the channel feature. The process is expressed as: in, represents the image features enhanced in channel dimension, [.] represents the feature concatenation operation, They represent the query, key, and value for the self-attention operation in the channel dimension respectively. The dimension of the image feature in the spatial dimension is rearranged from B×C×H×W to B×H×W×C, and restored to B×C×H×W after the attention calculation is completed. The attention calculation generates an HW×HW attention weight map to enhance the spatial feature. The process is expressed as: in, Represents the image features with enhanced spatial dimension, They represent the query, key, and value for the self-attention operation in the spatial dimension. The two sets of features after enhancement are concatenated and input into a gated convolution operation for feature regulation to reduce redundant information. The output features of this module are: Finally, the output features of multiple feature fusion modules are concatenated and the final fusion features are obtained by 1×1 convolution dimensionality reduction. The specific operation steps of the reconstruction module in Step 4 are as follows: The reconstruction module respectively outputs the final features of the LRMS super-resolution module And the final fusion features output by multiple feature fusion modules Reconstruction and restoration are performed; the structure of the reconstruction module includes 5 cascaded residual blocks, and finally a 1×1 convolution is used to reduce the dimension and output the reconstructed image, namely the super-resolution reconstruction result super-resolution MS image and the full-color sharpening result HRMS image.

2. The method for panchromatic sharpening based on super-resolution reconstruction task-assisted feature extraction according to claim 1, characterized in that: In the Step 1, the PAN image is cropped to a spatial size of 64×64, and the LRMS image is cropped to a spatial size of 16×16. They are normalized before being input into the network, and the LRMS image is upsampled to a spatial size of 64×64 in the preprocessing stage. Finally, the shallow features of the two images are extracted through a 3×3 convolution operation.

3. The method for panchromatic sharpening based on super-resolution reconstruction task-assisted feature extraction according to claim 1, characterized in that: There are 4 PAN feature extraction modules in the Step 2, which extract multi-scale image spatial features through convolution operations of different sizes; there are 4 LRMS super-resolution modules, which realize image super-resolution through cross-attention and use SwinTransformer to extract global image features.

4. The method for panchromatic sharpening based on super-resolution reconstruction task-assisted feature extraction according to claim 1, characterized in that: The Step 5 specifically includes: Step 51, use Adam optimizer to optimize the parameters in PAN feature extraction module, LRMS super-resolution module, feature fusion module and reconstruction module; Step 52, use L1 loss function to optimize all modules, assuming that the full color sharpening result HRMS image output by the reconstruction module is represented by Y j , the super-resolution reconstruction result high-definition MS image is expressed as The label image is G j ; Then the loss letters of the two tasks are: in, and They represent the loss functions of the full-color sharpening task and the super-resolution reconstruction task respectively. Since the task of the network is full-color sharpening, the total loss function of the network is expressed as: Among them, λ is the balance coefficient, and its value is 0.5.

Images