Remote sensing image panchromatic sharpening method and system based on spatial spectrum transformer

Abstract

The invention discloses a remote sensing image panchromatic sharpening method and system based on a spatial spectrum transformer, and belongs to the field of remote sensing image panchromatic sharpening. The invention provides a novel image sharpening model for panchromatic sharpening of a remote sensing image, the image sharpening model is of an encoder and decoder structure, and the encoder and the decoder both comprise a spatial spectrum transformer with guidance. Each spatial spectrum transformer with guidance comprises a spatial self-attention part, a spectrum self-attention part with guidance and a gating multilayer perceptron, and the spatial self-attention part and the spectrum self-attention part with guidance are respectively used for capturing spatial similarity and spectrum correlation of an input image; and the fusion quality of the spectrum and the spatial information is effectively improved. Compared with the prior art, the method has stronger advantages in the aspects of spectrum information retention and space detail recovery, and can be widely applied to panchromatic sharpening tasks in the fields of remote sensing monitoring, geological exploration, environment monitoring, precision agriculture, national defense safety and the like.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image panchromatic sharpening, and in particular to a method and system for fusing low spatial resolution multispectral images and high spatial resolution panchromatic images. Background Art

[0002] Remote sensing satellites acquire ground-based information using multispectral and panchromatic sensors. However, due to limitations such as signal-to-noise ratio and the diffraction limit, a single sensor struggles to achieve both high spatial and high spectral resolution. Panchromatic sharpening technology fuses high-resolution panchromatic imagery with low-resolution multispectral imagery to generate high-spatial-resolution multispectral imagery that combines the advantages of both, effectively overcoming the limitations of sensor performance. This technology leverages information from different sensors and has widespread applications in remote sensing, geological exploration, environmental monitoring, precision agriculture, and other fields. With the deployment of high-resolution satellites, its data support value is becoming increasingly prominent.

[0003] Currently, pan-sharpening methods are mainly divided into four categories: component substitution, multiresolution analysis, variational optimization, and machine learning. However, existing methods still have limitations in modeling the spatial non-local similarity and spectral correlation of the input images, which can lead to significant structural loss and spectral distortion in the fused image. Therefore, the development of efficient and stable pan-sharpening methods is of great significance for improving the quality of remote sensing images and promoting the development of remote sensing technology. Summary of the Invention

[0004] The purpose of the present invention is to adopt new technical means to propose a remote sensing image full color sharpening method based on spatial spectral transformer to solve the problems of structural loss and spectral distortion easily generated in the existing technology.

[0005] The above technical problems of the present invention are mainly solved by the following technical solutions: a remote sensing image full color sharpening method based on spatial spectral transformer, the specific steps are as follows: Step 1: Acquire low spatial resolution multispectral images and high spatial resolution panchromatic images; Step 2: upsample the low spatial resolution multispectral image to obtain an intermediate multispectral image of the same size as the high spatial resolution panchromatic image. Step 3: The intermediate multispectral image and the high spatial resolution panchromatic image are spliced in the spectral dimension to obtain input features; Step 4: Build an image sharpening model. The image sharpening model consists of an encoder and decoder structure. Both the encoder and decoder include guided spatial-spectral transformers. Each guided spatial-spectral transformer includes three parts: spatial self-attention, guided spectral self-attention, and a gated multi-layer perceptron. The output of the image sharpening model is added to the intermediate multispectral image to output a high spatial resolution multispectral image. Step 5: Construct a spatial spectral loss function and train an image sharpening model, and use the trained model to achieve full color sharpening.

[0006] Furthermore, in step 1, the low spatial resolution multispectral image is obtained by downsampling the original spatial resolution multispectral image, i.e., the ground truth image, and the high spatial resolution panchromatic image is obtained by pooling the original spatial resolution multispectral image in the spectral dimension; In step 2, the nearest neighbor interpolation is used to upsample the low spatial resolution multispectral image to obtain the intermediate multispectral image.

[0007] Furthermore, the processing process of the image sharpening model is as follows: first, a single convolution layer is used to extract the input features Obtain shallow features , then passes through a three-level hierarchical encoder and decoder structure composed of guided spatial spectral transformers. Each level of the encoder will undergo downsampling operations, and each level of the decoder will undergo upsampling operations and convolution operations. A jump connection is added between the encoder and decoder, and the processed features are finally output. , After adding it to the intermediate multispectral image, a high spatial resolution multispectral image is output.

[0008] Furthermore, the processing of spatial self-attention is as follows: First, the input features are transformed into Divide into several non-overlapping patches ,in I Represents the number of patches; Each patch is then projected as a query ,key Sum , modeled as:

[0009] in , and is the linear layer weight; further , and Divide into multiple heads and apply them separately in each head and The product of is used to generate the non-local spatial attention map , modeled as:

[0010] in For query Dimensions, is the relative deviation, Softmax is the normalization function; the obtained non-local spatial attention map is the weight mapping of each position to all position relationships, which is used to aggregate global context information to enhance feature expression capabilities. The output feature of spatial self-attention is recorded as .

[0011] Furthermore, the processing of guided spectral attention is as follows: First, the output features of spatial self-attention Transpose and reshape into , and then the same linear projection is used as the query ,key Sum , the projection formula is:

[0012] in , and is the linear layer weight; for , building a learnable global spectral guidance library , used to store representative universal spectral features. The initial parameters of the global spectral guidance library are randomly initialized and gradually learned through end-to-end optimization during training; Then Global average pooling AvgPool and linear projection are performed in sequence to obtain a weighted vector, which is then combined with the global spectral guidance library Interactively obtain spectral feature guidance , and finally Acts on Get guidance on optimizing queries :

[0013] in is the linear layer weight; then the same and Apply the product to obtain the global spectral attention map , modeled as:

[0014] in is the dimension; finally, use the obtained global spectral attention map The features are weighted and the output features are recorded as .

[0015] Furthermore, the processing of the gated multilayer perceptron is as follows: Given input features , the calculation process of the gated multilayer perceptron is as follows:

[0016] in is the weight matrix of the first layer, used for channel transformation; is a nonlinear activation function; is the weight matrix of the gating branch, used to generate the gating weight; Represents element-wise multiplication, which is a gated operation; is the intermediate feature after gating; is the second layer weight matrix, used to map the features back to the original channel dimension; is the output of the gated multilayer perceptron.

[0017] Furthermore, the spatial spectral loss function includes spectral angle consistency loss, guided gradient difference loss and Laplace structure consistency loss. The spatial spectral loss function is optimized by weighted combination of these three loss terms.

[0018] Furthermore, the spectral angle consistency loss , the similarity of the spectral vector of each pixel is measured by the angle difference, and the calculation formula is as follows:

[0019] in Output image i The spectral vector of pixels, The true value image i The spectral vector of pixels, N is the number of pixels in the image.

[0020] Furthermore, the guided gradient difference loss is used to measure and minimize the consistency error between the gradient response of each channel of the multispectral image and the gradient of the panchromatic image. The calculation formula is as follows:

[0021] in is the gradient operator, is the bth spectral channel of the output image, B is the number of multispectral image bands, is the preset channel adaptive weight, It is a high spatial resolution panchromatic image; The Laplace structure consistency loss guides the model to more accurately restore the edge and structure information in the image by comparing the response consistency of the high-resolution panchromatic image and the reconstructed high-spatial-resolution multispectral image at the Laplace pyramid level. The calculation formula is as follows:

[0022] in, is the grayscale image of the output image, is the grayscale image of the true value image, is the Laplace-Gaussian filter, is the L1 norm.

[0023] The present invention also provides a remote sensing image full color sharpening system based on spatial spectral transformer, which includes the following modules: Data acquisition module, used to acquire low spatial resolution multispectral images and high spatial resolution panchromatic images; The upsampling module is used to upsample the low spatial resolution multispectral image to obtain an intermediate multispectral image of the same size as the high spatial resolution panchromatic image; The input feature acquisition module is used to stitch the intermediate multispectral image with the high spatial resolution panchromatic image in the spectral dimension to obtain the input features; The model construction module is used to build an image sharpening model. The image sharpening model has an encoder and decoder structure. Both the encoder and decoder include guided spatial-spectral transformers. Each guided spatial-spectral transformer contains three parts: spatial self-attention, guided spectral self-attention, and a gated multi-layer perceptron. The output of the image sharpening model is added to the intermediate multispectral image to output a high-spatial-resolution multispectral image. The model training and panchromatic sharpening module is used to construct a spatial spectral loss function and perform image sharpening model training, and use the trained model to achieve panchromatic sharpening.

[0024] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: The remote sensing image full-color sharpening method of the present invention has stronger advantages in spectral information preservation and spatial detail recovery compared with the existing technology, and can be widely used in full-color sharpening tasks in remote sensing monitoring, geological exploration, environmental monitoring, precision agriculture, national defense and security and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Flowchart of the present invention.

[0026] Figure 2 2 is a structural diagram of an image sharpening model in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to provide a better understanding and recognition of the technical solution of the present invention, the following is further described with reference to preferred embodiments and accompanying drawings: Due to the limitations of imaging technology, there is a trade-off between spatial resolution and spectral resolution in satellite images. Pan-sharpening is used to fuse low spatial resolution multispectral images with high spatial resolution pan-chromatic images to generate high spatial resolution multispectral images. Existing traditional pan-sharpening methods are prone to spectral distortion, and although pan-sharpening methods based on deep learning improve the ability to learn spatial features, they are still limited in preserving spectral information. Figure 1 As shown, a remote sensing image pan-sharpening method provided by an embodiment of the present invention includes the following steps: In the first step, low spatial resolution multispectral images and high spatial resolution panchromatic images are simulated for training.

[0028] (1) Obtaining multispectral images with low spatial resolution: In this embodiment, the image acquired by the satellite is selected as the original spatial resolution multispectral image, that is, the true value image, whose width is 512, length is 512, and number of bands is 4. In order to obtain simulated training data, the low spatial resolution multispectral image From the ground truth image The downsampling factor is 4, and the downsampling formula is:

[0029] Low spatial resolution multispectral images obtained The size is , Indicates 4x downsampling; (2) Obtaining high spatial resolution panchromatic images: High spatial resolution panchromatic imaging From the ground truth image Pooling is performed in the spectral dimension, and the number of bands is reduced from 4 to 1. The pooling formula is:

[0030] in B Represents the number of bands, resulting in a high spatial resolution panchromatic image The size is .

[0031] In the second step, the nearest neighbor interpolation is used to simulate the Upsampling is performed to obtain , its purpose is to obtain the same size as the simulated full-color image to facilitate the splicing of the spectral dimension, assuming that the low spatial resolution image Need to zoom to new coordinates , the interpolation formula of the nearest neighbor interpolation is:

[0032] Indicates rounding to the nearest integer coordinate, where the interpolation multiple is 4.

[0033] The third step is to upsample the multispectral image With panchromatic image Splicing in the spectral dimension to obtain the input features of the model :

[0034] The fourth step is to build an image sharpening model, which is an encoder and decoder structure. Both the encoder and decoder include guided spatial spectral transformers: First, a single convolutional layer is used to extract the input features Obtain shallow features , C is the number of channels of shallow features, which is set to 32 here, and then passes through a 3-level hierarchical encoder and decoder structure composed of guided spatial spectral transformer (SSTG), as shown in Figure 2 As shown, by downsampling, upsampling and convolution operations, the channel capacity is increased while reducing the spatial resolution to obtain the processed features. Each guided spatial-spectral transformer consists of three parts: spatial self-attention (SSA), guided spectral self-attention (SSAG), and multi-layer perceptron (GMLP), which will be introduced in detail below: The input features first pass through the spatial self-attention, which calculates the attention matrix in the spatial dimension. In order to improve the computational efficiency while retaining the ability to model long-distance dependencies, we use a window size of Window operation. Input features First divided into non-overlapping patches ,in I Represents the number of patches.

[0035] Each patch is then projected as a query ,key Sum , modeled as:

[0036] The linear layer weight , and The size is . Further , and Divide into multiple heads and apply them separately in each head and The product of is used to generate the non-local spatial attention map , modeled as:

[0037] in For query The dimension of is 32, is the relative deviation, and Softmax is the normalization function. The obtained non-local spatial attention map is a weight mapping of each position to all position relationships, which is used to aggregate global context information to enhance feature expression capabilities. The output feature of spatial self-attention is recorded as .

[0038] For the features processed by spatial self-attention, guided spectral attention is used to capture the global dependencies between input feature channels. First, the output features of spatial self-attention are Transpose and reshape into , and then the same linear projection is used as the query ,key Sum , the projection formula is:

[0039] The linear layer weight , and The size is .for Considering that different objects have different spectral curves, we constructed a learnable global spectral guidance library , which is used to store representative universal spectral features, where 64 represents the number of spectral curves. The initial parameters of the boot library are randomly initialized and gradually learned through end-to-end optimization during the training process, thereby providing effective spectral priors for different images. Global average pooling AvgPool and linear projection are performed in sequence to obtain a weighted vector, which is then combined with the global spectral guidance library Interactively obtain spectral feature guidance , and finally Acts on Get guidance on optimizing queries ,

[0040] The linear layer weight The sizes are The spectral characteristics of any ground object can be expressed as a linear combination of these universal spectral curves, which greatly improves the robustness and generalization of the model. and Apply the product to obtain the global spectral attention map , modeled as:

[0041] Finally, use the obtained global spectral attention map The features are weighted and adjusted to highlight important channels and suppress redundant information. The output features of the guided spectral self-attention are recorded as , which will be input into a multi-layer perceptron (GMLP) with a gated mechanism.

[0042] The gated multilayer perceptron mainly introduces a gating mechanism in the spatial or channel dimension based on the traditional multilayer perceptron to control the flow of feature information and improve the nonlinear expression ability of the network. Its core idea is to perform weighted control on the output results to achieve stronger feature modeling capabilities. Given an input feature , the calculation process of the gated multilayer perceptron is as follows:

[0043] in is the weight matrix of the first layer, used for channel transformation; It is a nonlinear activation function, GELU is used here; is the weight matrix of the gating branch, used to generate the gating weight; Represents element-wise multiplication (Hadamard product), which is a gated operation; It is the intermediate feature after gating, containing selective information flow; is the second layer weight matrix, used to map the features back to the original channel dimension; It is the output of the gated multilayer perceptron, which is consistent with the input dimension and will be input into the next guided spatial spectral transformer block.

[0044] Finally, the model output and upsampled multispectral images After adding and fusion, the output restored image is obtained :

[0045] Notably, we also added skip connections between the encoder and decoder to mitigate information loss and enhance the ability to restore high-resolution details. Specifically, skip connections pass features from the corresponding layer in the encoder directly to the decoder, helping to fuse low-level spatial information with high-level semantic information, thereby improving the accuracy and detail restoration of the output image.

[0046] The fifth step is to construct the spatial spectral loss function and perform model training: (1) Constructing spatial spectral loss function In the pan-sharpening task of remote sensing images, our goal is to generate a high-resolution multispectral image using a low-resolution multispectral image and a high-resolution panchromatic image. To achieve this goal, we want to maintain spectral consistency while preserving spatial details.

[0047] Therefore, we propose a spatial-spectral loss function that integrates three main considerations to ensure the spectral, spatial, and structural consistency of the image. The design adopts the following three loss terms: Spectral consistency is one of the key factors in the pan-sharpening task of remote sensing images. We hope that the high-resolution image output by the model is as consistent as possible with the ground-truth image in each band. Therefore, the first is the spectral angle consistency loss. , the similarity of the spectral vector of each pixel is measured by the angle difference:

[0048] in is the output image No. i The spectral vector of each pixel (the set of values of the pixel in different bands), is the true value image No. i The spectral vector of pixels, N is the number of pixels in the image.

[0049] Next, in order to improve the spatial resolution and detail expression ability of multispectral images, we introduced a guided gradient consistency mechanism. Specifically, we use the gradient information in the high-resolution panchromatic image as a priori to guide the detail enhancement of the multispectral image in each spectral channel. This mechanism is based on the following assumption: the spatial structural details in the image (such as edges and textures) appear clearer and more stable in the high-resolution panchromatic image, and can therefore be used as a structural reference for the reconstruction of the multispectral image at the detail level. To this end, we designed a guided gradient difference loss , to measure and minimize the consistency error between the gradient response of each channel of the multispectral image and the gradient of the panchromatic image. This loss term calculates the local gradient of the image in the horizontal and vertical directions respectively and accumulates it in the multi-channel image to ensure that each spectral channel of the output image is effectively guided by the gradient of the panchromatic image in terms of edge structure:

[0050] in is the gradient operator, The output image is b spectral channels, B is the number of multispectral image bands, is the preset channel adaptive weight.

[0051] To further improve the structural fidelity of the reconstructed image, we introduced the Laplacian structural consistency loss. The core idea is to guide the model to more accurately recover the edge and structural information in the image by comparing the response consistency of the high-resolution panchromatic image and the reconstructed multispectral image at the Laplacian pyramid level:

[0052] in, is the grayscale image of the output image (the average result of multi-spectral channels), is the grayscale image of the true value image, It is a Laplace-Gaussian filter. The Laplace-Gaussian filter is an edge detection and structure extraction tool that combines the advantages of Gaussian smoothing and the Laplace operator. By removing noise first and then calculating the edge, it obtains more stable and more structured edge information:

[0053] in is a standard two-dimensional Gaussian function, Control the degree of smoothness (i.e., the size of the receptive field), is the coordinate of the pixel in the image, represents the two-dimensional Laplace operator (second-order derivative).

[0054] Finally, our spatial spectral loss function is optimized by weighted combination of these three loss terms:

[0055] in , and is a hyperparameter that weights each loss term and is used to balance the importance of different loss terms. Through the comprehensive optimization of these three loss terms, the spatial-spectral loss function can balance the requirements of different tasks in terms of spectrum, space, and structure, thereby achieving high-quality image restoration and pan-sharpening effects.

[0056] (2) Model training: After constructing the spatial-spectral transformer network and image sharpening model, the simulated image obtained in step S1 is used for training using the Adam optimizer and the constructed spatial-spectral loss function. The trained model can be used to fuse real low-spatial-resolution multispectral images and high-spatial-resolution panchromatic images.

[0057] We conducted down-resolution simulation experiments and full-resolution real experiments on two public datasets, WV3 and GF2, respectively. The results are shown in Tables 1 and 2. In the down-resolution simulation experiments, we used three indicators, spectral angle mapping (SAM), dimensionless global relative error (ERGAS), and multispectral structural similarity index (Q8), to measure the effectiveness of the method. SAM is used to measure the spectral angle difference between the restored image and the reference image at each pixel, ERGAS is used to measure the overall reconstruction quality, and Q8 measures the consistency of structure, brightness, and contrast between the reconstructed image and the reference image on multiple channels. In the full-resolution real experiments, we used the no-reference image quality index (HQNR), spatial distortion index ( ) and the spectral distortion index ( ) are used to measure the effect of the method, among which HQNR is a comprehensive indicator combining spectrum and spatial distortion, which is suitable for the case where there is no high-resolution reference image in real experiments. Used to measure the fidelity of the image in terms of spatial structure. Used to measure the consistency of spectral information between the fused image and the original image.

[0058] Table 1. Simulation experiments on 20 low-resolution samples, covering the WV3 and GF2 datasets

[0059] The lower the value, the better the effect. The lower the value, the better the effect. The best results are highlighted in bold, and the suboptimal results are underlined.

[0060] Table 2. Real experiments on 20 full-resolution samples, covering both WV3 and GF2 datasets

[0061] From the experimental results, the present invention shows strong advantages on different data sets. In the reduced-resolution simulation experiment, the method outperforms other comparison methods in SAM, ERGAS and Q8 indicators. In the full-resolution real experiment, the present invention also shows excellent performance, with HQNR, and All indicators have reached the optimal or near-optimal level, indicating that it has a stronger ability in fusing spatial and spectral information.

[0062] The EXP in the listed methods comes from the literature Aiazzi B, Alparone L, Baronti S, et al. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis[J]. IEEE Transactions on geoscience and remote sensing, 2002, 40(10): 2300-2312; MTF-GLP-FS is derived from the literature Vivone G, Restaino R, Chanussot J. Full scale regression-based injection coefficients for panchromatic sharpening[J]. IEEE Transactions on Image Processing, 2018, 27(7): 3418-3431; TV comes from the literature Palsson F, Sveinsson JR, Ulfarsson M O. A newpansharpening algorithm based on total variation[J]. IEEE Geoscience andRemote Sensing Letters, 2013, 11(1): 318-322; BDSD-PC comes from the literature Vivone G. Robust band-dependent spatial-detail approaches for panchromatic sharpening[J]. IEEE transactions on Geoscienceand Remote Sensing, 2019, 57(9): 6421-6433; LRTCFan comes from the literature Wu ZC, Huang TZ, Deng LJ, et al. LRTCFan: Low-rank tensor completion based framework for pansharpening[J]. IEEETransactions on Image Processing, 2023, 32: 1640-1655; PNN comes from the literature Masi G, Cozzolino D, Verdoliva L, et al. Pansharpeningby convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594; PanNet is derived from Yang J, Fu X, Hu Y, et al. PanNet: A deep network architecture for pan-sharpening[C] / / Proceedings of the IEEE international conference on computer vision. 2017: 5449-5457; DiCNN comes from the literature He L, Rao Y, Li J, et al. Pansharpening via detailinjection based convolutional neural networks[J]. IEEE Journal of SelectedTopics in Applied Earth Observations and Remote Sensing, 2019, 12(4): 1188-1204; FusionNet comes from the literature Deng LJ, Vivone G, Jin C, et al. Detailinjection-based deep convolutional neural networks for pansharpening[J]. IEEETransactions on Geoscience and Remote Sensing, 2020, 59(8): 6995-7010; DCFNet comes from the literature Wu X, Huang TZ, Deng LJ, et al. Dynamic crossfeature fusion for remote sensing pansharpening[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 14687-14696; LAGConv comes from the literature Jin ZR, Zhang TJ, Jiang TX, et al. LAGConv: Local-context adaptive convolution kernels with global harmonic bias forpansharpening[C] / / Proceedings of the AAAI conference on artificialintelligence. 2022, 36(1): 1113-1121; HMPNet comes from the literature Tian X, Li K, Zhang W, et al. Interpretable model-driven deep network for hyperspectral, multispectral, and panchromatic imagefusion[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023; CANConv comes from the literature Duan Y, Wu On the other hand, an embodiment of the present invention further provides a remote sensing image pan-sharpening system based on a spatial spectral transformer, comprising the following modules: Data acquisition module, used to acquire low spatial resolution multispectral images and high spatial resolution panchromatic images; The upsampling module is used to upsample the low spatial resolution multispectral image to obtain an intermediate multispectral image of the same size as the high spatial resolution panchromatic image; The input feature acquisition module is used to stitch the intermediate multispectral image with the high spatial resolution panchromatic image in the spectral dimension to obtain the input features; The model construction module is used to build an image sharpening model. The image sharpening model has an encoder and decoder structure. Both the encoder and decoder include guided spatial-spectral transformers. Each guided spatial-spectral transformer contains three parts: spatial self-attention, guided spectral self-attention, and a gated multi-layer perceptron. The output of the image sharpening model is added to the intermediate multispectral image to output a high-spatial-resolution multispectral image. The model training and panchromatic sharpening module is used to construct a spatial spectral loss function and perform image sharpening model training, and use the trained model to achieve panchromatic sharpening.

[0063] The specific implementation method of each module is the same as that of each step and will not be described in detail in this invention.

[0064] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions merely illustrate the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the invention as claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A remote sensing image pan-sharpening method based on spatial spectral transformer, characterized in that: The steps include: Step 1: Acquire low spatial resolution multispectral images and high spatial resolution panchromatic images; Step 2: upsample the low spatial resolution multispectral image to obtain an intermediate multispectral image of the same size as the high spatial resolution panchromatic image. Step 3: The intermediate multispectral image and the high spatial resolution panchromatic image are spliced in the spectral dimension to obtain input features; Step 4: Build an image sharpening model. The image sharpening model consists of an encoder and decoder structure. Both the encoder and decoder include guided spatial-spectral transformers. Each guided spatial-spectral transformer includes three parts: spatial self-attention, guided spectral self-attention, and a gated multi-layer perceptron. The output of the image sharpening model is added to the intermediate multispectral image to output a high spatial resolution multispectral image. Step 5: Construct a spatial spectral loss function and train an image sharpening model, and use the trained model to achieve full color sharpening.

2. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 1, characterized in that: In step 1, the low spatial resolution multispectral image is obtained by downsampling the original spatial resolution multispectral image, i.e., the ground truth image, and the high spatial resolution panchromatic image is obtained by pooling the original spatial resolution multispectral image in the spectral dimension; In step 2, the nearest neighbor interpolation is used to upsample the low spatial resolution multispectral image to obtain the intermediate multispectral image.

3. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 1, wherein: The processing process of the image sharpening model is as follows: first, a single convolution layer is used to extract the input features Obtain shallow features , then passes through a three-level hierarchical encoder and decoder structure composed of guided spatial spectral transformers. Each level of the encoder will undergo downsampling operations, and each level of the decoder will undergo upsampling operations and convolution operations. A jump connection is added between the encoder and decoder, and the processed features are finally output. , After adding it to the intermediate multispectral image, a high spatial resolution multispectral image is output.

4. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 1, wherein: The processing process of spatial self-attention is as follows: First, the input features are transformed into Divide into several non-overlapping patches ,in I Represents the number of patches; Each patch is then projected as a query ,key Sum , modeled as: in , and is the linear layer weight; further , and Divide into multiple heads and apply them separately in each head and The product of is used to generate the non-local spatial attention map , modeled as: in For query Dimensions, is the relative deviation, Softmax is the normalization function; the obtained non-local spatial attention map is the weight mapping of each position to all position relationships, which is used to aggregate global context information to enhance feature expression capabilities. The output feature of spatial self-attention is recorded as .

5. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 1, wherein: The processing of guided spectral attention is as follows: First, the output features of spatial self-attention Transpose and reshape into , and then the same linear projection is used as the query ,key Sum , the projection formula is: in , and is the linear layer weight; for , building a learnable global spectral guidance library , used to store representative universal spectral features. The initial parameters of the global spectral guidance library are randomly initialized and gradually learned through end-to-end optimization during training; Then Global average pooling AvgPool and linear projection are performed in sequence to obtain a weighted vector, which is then combined with the global spectral guidance library Interactively obtain spectral feature guidance , and finally Acts on Get guidance on optimizing queries : in is the linear layer weight; then the same and Apply the product to obtain the global spectral attention map , modeled as: in is the dimension; finally, use the obtained global spectral attention map The features are weighted and the output features are recorded as .

6. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 1, wherein: The processing of the gated multilayer perceptron is as follows: Given input features , the calculation process of the gated multilayer perceptron is as follows: in is the weight matrix of the first layer, used for channel transformation; is a nonlinear activation function; is the weight matrix of the gating branch, used to generate the gating weight; Represents element-wise multiplication, which is a gated operation; is the intermediate feature after gating; is the second layer weight matrix, used to map the features back to the original channel dimension; is the output of the gated multilayer perceptron.

7. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 1, wherein: The spatial spectral loss function includes spectral angle consistency loss, guided gradient difference loss and Laplace structure consistency loss. The spatial spectral loss function is optimized by weighted combination of these three loss terms.

8. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 7, characterized in that: Spectral angle consistency loss , the similarity of the spectral vector of each pixel is measured by the angle difference, and the calculation formula is as follows: in Output image i The spectral vector of pixels, The true value image i The spectral vector of pixels, N is the number of pixels in the image.

9. The method for pan-sharpening remote sensing images based on a spatial spectral transformer according to claim 7, wherein: The guided gradient difference loss is used to measure and minimize the consistency error between the gradient response of each channel of the multispectral image and the gradient of the panchromatic image. The calculation formula is as follows: in is the gradient operator, is the bth spectral channel of the output image, B is the number of multispectral image bands, is the preset channel adaptive weight, It is a high spatial resolution panchromatic image; The Laplace structure consistency loss guides the model to more accurately restore the edge and structure information in the image by comparing the response consistency of the high-resolution panchromatic image and the reconstructed high-spatial-resolution multispectral image at the Laplace pyramid level. The calculation formula is as follows: in, is the grayscale image of the output image, is the grayscale image of the true value image, is the Laplace-Gaussian filter, is the L1 norm.

10. A remote sensing image full color sharpening system based on spatial spectral transformer, characterized in that: Includes the following modules: Data acquisition module, used to acquire low spatial resolution multispectral images and high spatial resolution panchromatic images; The upsampling module is used to upsample the low spatial resolution multispectral image to obtain an intermediate multispectral image of the same size as the high spatial resolution panchromatic image; The input feature acquisition module is used to obtain input features by splicing the intermediate multispectral image and the high spatial resolution panchromatic image in the spectral dimension; The model construction module is used to build an image sharpening model. The image sharpening model has an encoder and decoder structure. Both the encoder and decoder include guided spatial-spectral transformers. Each guided spatial-spectral transformer contains three parts: spatial self-attention, guided spectral self-attention, and a gated multi-layer perceptron. The output of the image sharpening model is added to the intermediate multispectral image to output a high-spatial-resolution multispectral image. The model training and panchromatic sharpening module is used to construct a spatial spectral loss function and perform image sharpening model training, and use the trained model to achieve panchromatic sharpening.

Images