Panchromatic sharpening method based on space-spectrum Token learning network

Abstract

The invention discloses a panchromatic sharpening method (SSTL) based on a space-spectrum Token learning network, and provides a neural network framework capable of reconstructing and fusing space and spectrum features for a panchromatic sharpening task, namely the space-spectrum Token learning network which comprises three novel modules. The Fourier-guided Token reconstruction module extracts the interaction characteristics of Fourier domain Token from the panchromatic image and the low-resolution multispectral image by applying a complex characteristic interaction rule to realize the reconstruction of spectral characteristics; the Token selection module is used for forcing the space-spectrum Token learning network to reconstruct the spatial characteristics of the high-resolution multispectral image from the most relevant spatial region of the panchromatic image; and the adaptive Token interaction module uses the weight learned from the Token selection module in the channel interaction process of the spectral characteristics, so that seamless integration of the space-spectral characteristics is realized. In addition, an experiment on a public data set proves the effectiveness of the method disclosed by the invention.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning and computer vision, and in particular relates to a full color sharpening method based on a spatial-spectral Token learning network. Background Art

[0002] With the increasing demand for earth observation and monitoring, existing optical satellites are able to simultaneously record low-resolution multispectral (LR-MS) images and corresponding high-resolution panchromatic (PAN) images from the same scene. Due to the physical limitations of existing satellite sensors, the recorded LR-MS images usually contain rich spectral features but relatively sparse spatial features, while their corresponding PAN images contain rich spatial features but sparse spectral features. However, in practical applications, such as agriculture and forestry, meteorology, ecological environment assessment, urban planning and construction, military reconnaissance and monitoring, remote sensing images with multiple spectral bands and high spatial resolution, namely high-resolution multispectral (HR-MS) images, play a more important role.

[0003] To solve this problem, researchers proposed a pan-sharpening method, which obtains HR-MS images by fusing the spatial and spectral features of PAN and LR-MS images.

[0004] Early panchromatic sharpening methods mainly include component replacement (CS) methods, multi-resolution analysis (MRA) methods, and variational optimization (VO)-based methods. Specifically, the CS method uses the spatial details in the PAN image to replace the corresponding details in the LR-MS image. Although this method is simple and has high fidelity, it often introduces spectral distortion during the fusion process; the MRA method injects the spatial structure in the PAN image into the LR-MS image, better preserving the spectral characteristics, but causing spatial distortion. The VO-based method constructs a regularization function based on physical priors to optimize the model. Although it has achieved better results, the optimization effect is limited.

[0005] In recent years, pan-sharpening methods based on convolutional neural networks have been widely used due to their powerful ability to learn task-specific representations. These methods mainly adopt two fusion strategies to exploit the spatial-spectral characteristics of PAN images and LR-MS images: image-level fusion and feature-level fusion. The image-level fusion strategy directly concatenates the LR-MS images and PAN images along the channel dimension before inputting the features into the network, which may cause the model to confuse spectral and spatial information during feature extraction, thus distorting it. Meanwhile, the feature-level fusion strategy usually utilizes a two-stream network to extract spatial features from PAN images and spectral features from LR-MS images respectively, and then combines these features in the latent feature space. However, this strategy lacks the modal interaction between PAN images and LR-MS images in the feature extraction stage, so the learned model cannot effectively exploit the cross-feature spatial dependency between LR-MS images and PAN images.

[0006] Unlike CNN, Transformer-style networks are able to effectively capture long-range dependencies between different sequential tokens in the input data based on their multi-head attention mechanism. Therefore, many recent studies have extended Transformer to the field of panchromatic sharpening. However, these Transformer-based methods still fail to consider the relationship between PAN, LR-MS, and target HR-MS images from the perspective of spatial-spectral characteristics. Summary of the invention

[0007] The purpose of the present invention is to solve the problems existing in the prior art, and use deep learning and computer vision technologies to extract and fuse PAN and LR-MS images with different spectral and spatial features, thereby providing a full-color sharpening method based on a spatial-spectral Token learning network.

[0008] In order to achieve the above-mentioned invention object, the present invention specifically adopts the following technical solutions:

[0009] A panchromatic sharpening method based on a spatial-spectral Token learning network comprises the following steps: a group of PAN images and LR-MS images to be panchromatic sharpened are input into a trained spatial-spectral Token learning network, the PAN images and the LR-MS images are respectively subjected to feature extraction by a first convolutional layer, the extracted implicit features of the PAN images and the implicit features of the LR-MS images are input into a feature learning module, a group of Tokens describing the reconstructed spectral feature attributes and a group of Tokens describing the reconstructed spatial attributes output by the feature learning module are input into a feature fusion module to obtain aggregated features that fuse spatial features and spectral features, and the aggregated features are processed by a second convolutional layer to obtain a target HR-MS image;

[0010] The spatial-spectral Token learning network has a U-shaped structural feature. The feature learning module includes four feature learning sub-modules, the first three feature learning sub-modules all include a Token reconstruction module, a Token selection module and a down-sampling module, and the last feature learning sub-module only includes a Token reconstruction module and a Token selection module; the feature fusion module includes four feature fusion sub-modules, the first three feature fusion sub-modules all include an adaptive Token interaction module and an up-sampling module, and the last feature fusion sub-module only includes an adaptive Token interaction module, and the first three feature fusion sub-modules each perform feature fusion by means of jump connection before entering the next adjacent feature fusion sub-module to reduce the information loss caused by the down-sampling operation.

[0011] Based on the above scheme, each step can be implemented in the following preferred specific manner.

[0012] Preferably, the specific processing flow in the spatial-spectral Token learning network is as follows: after the PAN image passes through the first convolution layer, the implicit features of the PAN image are obtained; after the LR-MS image passes through the first convolution layer, the implicit features of the LR-MS image are obtained; the two implicit features are used as inputs of the first feature learning submodule; the implicit features of the PAN image are first processed by the Token selection module in the first feature learning submodule to obtain the first intermediate features; and then the implicit features of the PAN image and the implicit features of the LR-MS image are input together into the Token reconstruction module in the first feature learning submodule to obtain the second intermediate features;

[0013] The first intermediate feature and the second intermediate feature are respectively downsampled as inputs of the second feature learning submodule, the Token selection module in the second feature learning submodule first processes the downsampled first intermediate feature to obtain a third intermediate feature, and then the downsampled first intermediate feature and the downsampled second intermediate feature are input together to the Token reconstruction module in the second feature learning submodule to obtain a fourth intermediate feature, the third intermediate feature and the fourth intermediate feature are respectively downsampled as inputs of the third feature learning submodule, the Token selection module in the third feature learning submodule first processes the downsampled third intermediate feature to obtain a fifth intermediate feature, and then the downsampled third intermediate feature and the downsampled fourth intermediate feature are input together to the Token reconstruction module in the third feature learning submodule to obtain a sixth intermediate feature;

[0014] The fifth intermediate feature and the sixth intermediate feature are respectively downsampled as inputs of the fourth feature learning submodule, and the Token selection module in the fourth feature learning submodule first processes the downsampled fifth intermediate feature to obtain a group of Tokens describing the attributes of the reconstructed space, and then the downsampled fifth intermediate feature and the downsampled sixth intermediate feature are input together into the Token reconstruction module in the fourth feature learning submodule to obtain a group of Tokens describing the attributes of the reconstructed spectral features;

[0015] A group of Tokens describing the reconstructed spectral feature attributes and a group of Tokens describing the reconstructed spatial attributes are input together into the adaptive Token interaction module in the first feature fusion submodule to obtain a first interaction feature, the upsampled first interaction feature is concatenated with the fifth intermediate feature to obtain a first concatenated feature, the first concatenated feature is processed by the third convolutional layer to obtain a seventh intermediate feature, the upsampled first interaction feature is concatenated with the sixth intermediate feature to obtain a second concatenated feature, the second concatenated feature is processed by the third convolutional layer to obtain an eighth intermediate feature;

[0016] The seventh intermediate feature and the eighth intermediate feature are input together into the adaptive Token interaction module in the second feature fusion submodule to obtain the second interaction feature, the upsampled second interaction feature is concatenated with the third intermediate feature to obtain the third concatenated feature, the third concatenated feature is processed by the fourth convolutional layer to obtain the ninth intermediate feature, the upsampled second interaction feature is concatenated with the fourth intermediate feature to obtain the fourth concatenated feature, the fourth concatenated feature is processed by the fourth convolutional layer to obtain the tenth intermediate feature;

[0017] The ninth intermediate feature and the tenth intermediate feature are input together into the adaptive Token interaction module in the third feature fusion submodule to obtain the third interaction feature, the upsampled third interaction feature is concatenated with the first intermediate feature to obtain the fifth concatenated feature, the fifth concatenated feature is processed by the fifth convolutional layer to obtain the eleventh intermediate feature, the upsampled third interaction feature is concatenated with the second intermediate feature to obtain the sixth concatenated feature, the sixth concatenated feature is processed by the fifth convolutional layer to obtain the twelfth intermediate feature;

[0018] The eleventh intermediate feature and the twelfth intermediate feature are input together into the adaptive Token interaction module in the fourth feature fusion submodule to obtain the fourth interaction feature and use it as an aggregate feature that fuses the spatial characteristics and the spectral characteristics. Finally, the aggregate feature is processed by the second convolutional layer to obtain the target HR-MS image.

[0019] Preferably, the specific processing flow in the Token reconstruction module is as follows:

[0020] First, the two features input to the Token reconstruction module are normalized and discrete Fourier transformed to obtain the Fourier domain Token of each input feature. Then, the Fourier domain Token of each feature is approximated in the form of complex numbers to facilitate implicit modeling. The specific calculation formula is as follows:

[0021]

[0022] Among them, r x The real mapping of Fourier domain Token representing the LR-MS image; r y The real mapping of Fourier domain Token representing the PAN image; p x represents the virtual mapping of the Fourier domain Token of the LR-MS image; p y The virtual mapping of Fourier domain Token representing the PAN image; They all represent the complex form of the Fourier domain Token of each feature; j represents the imaginary unit.

[0023] Then, the real mapping of the Fourier domain Token of the LR-MS image and the real mapping of the Fourier domain Token of the PAN image are connected to obtain a fused real mapping, and the virtual mapping of the Fourier domain Token of the LR-MS image and the virtual mapping of the Fourier domain Token of the PAN image are connected to obtain a fused virtual mapping. The fused real mapping and the fused virtual mapping are input into a Fourier domain MLP network formed by cascading L Fourier domain MLP layers, and the implicit feature interaction result in the Fourier domain is output and an inverse discrete Fourier transform operation is performed on it to obtain the output of the Token reconstruction module;

[0024] Among them, the specific processing process of the lth Fourier domain MLP layer is as follows:

[0025]

[0026] In the formula, y l represents the implicit feature interaction result output by the lth Fourier domain MLP layer; r s l-1 ,p s l-1 denote the real and imaginary parts of the input to the lth Fourier domain MLP layer, respectively; σ denotes the activation function; denote the real and imaginary parts of the complex weight matrix of the lth Fourier domain MLP layer, respectively; denote the real and imaginary parts of the complex bias of the lth Fourier domain MLP layer, respectively.

[0027] Preferably, the specific processing flow in the Token selection module is as follows: first, the features input to the Token selection module are normalized by layers to obtain first normalized features, and the first normalized features are respectively passed through three convolution branches, the first convolution branch outputs a query matrix, the second convolution branch outputs a key matrix K, and the third convolution branch outputs a value matrix V. The attention matrix S is calculated by the query matrix Q and the reshaped key matrix:

[0028]

[0029] Where d represents the dimension of the key matrix; T represents the matrix transpose;

[0030] Then, the elements in the attention matrix are masked to obtain the mask matrix, and the mask matrix is ​​normalized to obtain the activation attention matrix with a dense rate. Finally, the activation attention matrix is ​​multiplied by the value matrix to obtain the output of the Token selection module.

[0031] As a preferred method, in the Token selection module, the specific process of the mask operation is as follows: the kth largest element value t in the fth row of the attention matrix is f As the relevance threshold, if the element value of the attention matrix row f and column g is greater than or equal to t f , then the element value of the attention matrix located in the fth row and gth column is retained, otherwise the element value of the attention matrix located in the fth row and gth column is reset to negative infinity until each row of the attention matrix is ​​judged and a complete mask matrix is ​​obtained.

[0032] Preferably, in the first feature fusion submodule, the specific processing flow of the adaptive Token interaction module is as follows: a group of Tokens describing the characteristic attributes of the reconstructed spectrum are normalized by layers to obtain a second normalized feature, and the second normalized feature is passed through three new convolution branches respectively, and the query projection is output by the first new convolution branch. The output value of the third new convolution branch is projected, and the output of the second new convolution branch is reshaped to obtain the key projection. The query projection and the reshaped key projection are dot-producted to calculate the attention map A:

[0033]

[0034] in, represents the result of key projection after reshaping; α is a learnable parameter for adapting the attention map; Represents the dot product operation;

[0035] A set of Tokens describing the attributes of the reconstructed space are multiplied by the value projection to obtain a weighted importance weight matrix. The weighted importance weight matrix is ​​reshaped and then dot-producted with the normalized attention map to obtain a weighted attention map. After the weighted attention map is reshaped, the output of the adaptive Token interaction module in the first feature fusion submodule is obtained.

[0036] Preferably, in the second, third and fourth feature fusion sub-modules, the eighth, tenth and twelfth intermediate features are used for layer normalization and processed by convolution branches, and the seventh, ninth and eleventh intermediate features are used to multiply the value projections generated by the corresponding feature fusion sub-modules to obtain the weighted importance weight matrix of the corresponding feature fusion sub-module.

[0037] Preferably, each convolution branch is formed by sequentially cascading a standard convolution layer and a deep convolution layer, wherein the convolution kernel size of the standard convolution layer is 1×1, and the convolution kernel size of the deep convolution layer is 3×3.

[0038] Preferably, the loss function used in the training of the spatial-spectral Token learning network is:

[0039]

[0040] Among them, τ is the balance factor; represents reconstruction losses; Indicates auxiliary loss; X out is the HR-MS image output by the spatial-spectral Token learning network, X gt is the true value of the HR-MS image; A out is the amplitude spectrum of the output feature of the Token reconstruction module in the last feature learning submodule; A gt The amplitude spectrum is obtained by fast Fourier transforming the true value of the HR-MS image.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] The present invention takes into account that in the panchromatic sharpening problem, PAN images and LR-MS images have different spatial and spectral features, and the implicit interaction between the extracted Fourier domain features can reconstruct the amplitude of the target Fourier domain features. Based on this, when reconstructing the target spectral characteristics, a Token reconstruction module is designed to model the spectral characteristics of the target HR-MS image; the present invention takes into account the characteristics of large-area earth observation scenes, such as the existence of redundant features, and thus designs a Token selection module when dealing with problems such as feature redundancy, so that the spatial-spectral Token learning network can capture the most relevant spatial features on the target HR-MS image while maintaining a moderate spatial selection complexity; the present invention takes into account that the extracted spectral features have position dependence, and thus designs an adaptive Token interaction module for dynamically adjusting the spectral features according to the spatial features when the spatial-spectral features are fused.

[0043] The present invention constructs a spatial-spectral Token learning network for panchromatic sharpening based on a Token reconstruction module, a Token selection module, and an adaptive Token interaction module. Aiming at the spatial-spectral characteristics in the panchromatic sharpening problem, the present invention designs a panchromatic sharpening method SSTL based on a spatial-spectral Token learning network, which effectively extracts and fuses spatial and spectral features. Experiments are conducted on WorldView-3, QuickBird, and GaoFen-2 datasets. The experimental results on various datasets show that compared with the most advanced methods in the past, the method of the present invention has achieved better results and better performance after introducing the above modules, making the spatial-spectral Token learning network of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the steps of the method of the present invention;

[0045] Figure 2 This is a schematic diagram of the overall structure of the spatial-spectral Token learning network of the present invention;

[0046] Figure 3 This is a schematic diagram of the structure of the Token reconstruction module in the spatial-spectral Token learning network;

[0047] Figure 4 This is a schematic diagram of the Token selection module structure in the spatial-spectral Token learning network;

[0048] Figure 5 This is a schematic diagram of the structure of the adaptive Token interaction module in the spatial-spectral Token learning network;

[0049] Figure 6A training and testing flow chart of a spatial-spectral Token learning network in an embodiment of the present invention;

[0050] Figure 7 This is a reconstruction result diagram of the WorldView-3 case in the low-resolution experiment of an embodiment of the present invention;

[0051] Figure 8 This is a reconstruction result diagram of the QuickBird case in the low-resolution experiment of an embodiment of the present invention;

[0052] Fig. 9 This is a reconstruction result diagram of the GaoFen-2 case in the low-resolution experiment of an embodiment of the present invention;

[0053] Fig.10 This is a reconstruction result diagram of the WorldView-3 case in the full-resolution experiment of an embodiment of the present invention;

[0054] Fig.11 This is a reconstruction result diagram of the QuickBird case in the full-resolution experiment of an embodiment of the present invention;

[0055] Fig.12 This is a reconstruction result diagram of the GaoFen-2 case in the full-resolution experiment of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0057] In the description of the present invention, it should be understood that the terms "first" and "second" are only used for the purpose of distinguishing descriptions, and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features.

[0058] The problem of panchromatic sharpening is to break through the limitation of technology and obtain remote sensing images with rich spatial features and rich spectral features at the same time. Its input is a PAN image containing rich spatial features but relatively sparse spectral features and an LR-MS image containing rich spectral features but relatively sparse spatial features. The output is a HR-MS image with both rich spectral features and spatial features. How to effectively fuse spectral features and spatial features is a key issue that needs to be considered in the panchromatic sharpening task. Existing technologies for panchromatic sharpening tasks have not been able to consider the relationship between image spatial features and spectral features well. Therefore, the present invention conducts a comprehensive study on the relationship between the target HR-MS image and its corresponding input PAN and LR-MS images in the spectral domain and the spatial domain. During the research process, we first found that the spatial distribution of PAN images is very similar to the spatial distribution of each spectrum of their corresponding HR-MS images, from which we concluded that the spatial properties of the target HR-MS image can be reconstructed through the feature interaction within the Tokens of the PAN image; secondly, we found that the intra-spectral relationship of the LR-MS image is very different from the intra-spectral relationship of the corresponding HR-MS image, so we believe that using only LR-MS images to reconstruct the intra-spectral relationship of the target HR-MS image is not the optimal method, that is, it is necessary to achieve effective feature interaction between the Tokens of the PAN image and the LR-MS image; in addition, it is found that the reconstruction of spectral characteristics can be achieved by reconstructing the amplitude in the Fourier domain.

[0059] Based on the above findings, the present invention adopts a new approach to the panchromatic sharpening task: (1) implicitly interacting the Fourier domain tokens extracted from the PAN image and the LR-MS image to reconstruct the spectral characteristics of the target HR-MS image; (2) only selecting the tokens corresponding to the top k highest attention values ​​in the PAN image to capture the spatial characteristics provided by the PAN image that are most relevant to the reconstruction of the spatial characteristics of the target HR-MS image. Therefore, the present invention proposes a spatial-spectral Token learning network, and based on this network, designs a panchromatic sharpening method based on the spatial-spectral Token learning network.

[0060] like Figure 1As shown, in a preferred implementation of the present invention, the above-mentioned full-color sharpening method based on the space-spectral Token learning network has the following specific steps: a group of PAN images (P) and LR-MS images (L) to be full-color sharpened are input into the trained space-spectral Token learning network, the PAN images and the LR-MS images are respectively subjected to feature extraction by the first convolutional layer, the extracted implicit features y0 of the PAN image and the implicit features x0 of the LR-MS image are input into the feature learning module, a group of Tokens describing the reconstructed spectral feature attributes and a group of Tokens describing the reconstructed spatial attributes output by the feature learning module are input into the feature fusion module to obtain aggregated features that fuse spatial characteristics and spectral characteristics, and the aggregated features are processed by the second convolutional layer to obtain the target HR-MS image;

[0061] The spatial-spectral Token learning network has a U-shaped structural feature. The feature learning module includes four feature learning sub-modules, the first three feature learning sub-modules all include a Token reconstruction module (FGTR), a Token selection module (TKTS) and a down-sampling module, and the last feature learning sub-module only includes a Token reconstruction module and a Token selection module; the feature fusion module includes four feature fusion sub-modules, the first three feature fusion sub-modules all include an adaptive Token interaction module (ATI) and an up-sampling module, and the last feature fusion sub-module only includes an adaptive Token interaction module, and the first three feature fusion sub-modules each perform feature fusion by skip connection before entering the next adjacent feature fusion sub-module to reduce the information loss caused by the down-sampling operation.

[0062] It should be noted that in the present invention, the structure of the spatial-spectral Token learning network for panchromatic sharpening is as follows: Figure 2 The following will introduce the specific processing flow in the spatial-spectral Token learning network to facilitate the understanding of those skilled in the art.

[0063] exist Figure 2 In the example, the PAN image (P) is subjected to the first convolutional layer to obtain the latent feature y0 of the PAN image, and the LR-MS image is subjected to the first convolutional layer to obtain the latent feature x0 of the LR-MS image. The two latent features are used as the input of the first feature learning submodule. The latent feature y0 of the PAN image is first processed by the Token selection module in the first feature learning submodule to obtain the first intermediate feature. The latent feature y0 of the PAN image and the latent feature x0 of the LR-MS image are then input together into the Token reconstruction module in the first feature learning submodule to obtain the second intermediate feature.

[0064] The first intermediate feature and the second intermediate feature are respectively downsampled as inputs of the second feature learning submodule, the Token selection module in the second feature learning submodule first processes the downsampled first intermediate feature to obtain a third intermediate feature, and then the downsampled first intermediate feature and the downsampled second intermediate feature are input together to the Token reconstruction module in the second feature learning submodule to obtain a fourth intermediate feature, the third intermediate feature and the fourth intermediate feature are respectively downsampled as inputs of the third feature learning submodule, the Token selection module in the third feature learning submodule first processes the downsampled third intermediate feature to obtain a fifth intermediate feature, and then the downsampled third intermediate feature and the downsampled fourth intermediate feature are input together to the Token reconstruction module in the third feature learning submodule to obtain a sixth intermediate feature;

[0065] The fifth intermediate feature and the sixth intermediate feature are respectively downsampled as the input of the fourth feature learning submodule, and the Token selection module in the fourth feature learning submodule first processes the downsampled fifth intermediate feature to obtain a group of Tokens describing the reconstructed spatial attributes; then the downsampled fifth intermediate feature and the downsampled sixth intermediate feature are input together into the Token reconstruction module in the fourth feature learning submodule to obtain a group of Tokens describing the reconstructed spectral feature attributes;

[0066] A group of Tokens describing the reconstructed spectral feature attributes and a group of Tokens describing the reconstructed spatial attributes are input together into the adaptive Token interaction module in the first feature fusion submodule to obtain a first interaction feature, the upsampled first interaction feature is concatenated with the fifth intermediate feature to obtain a first concatenated feature, the first concatenated feature is processed by the third convolutional layer to obtain a seventh intermediate feature, the upsampled first interaction feature is concatenated with the sixth intermediate feature to obtain a second concatenated feature, the second concatenated feature is processed by the third convolutional layer to obtain an eighth intermediate feature;

[0067] The seventh intermediate feature and the eighth intermediate feature are input together into the adaptive Token interaction module in the second feature fusion submodule to obtain the second interaction feature, the upsampled second interaction feature is concatenated with the third intermediate feature to obtain the third concatenated feature, the third concatenated feature is processed by the fourth convolutional layer to obtain the ninth intermediate feature, the upsampled second interaction feature is concatenated with the fourth intermediate feature to obtain the fourth concatenated feature, the fourth concatenated feature is processed by the fourth convolutional layer to obtain the tenth intermediate feature;

[0068] The ninth intermediate feature and the tenth intermediate feature are input together into the adaptive Token interaction module in the third feature fusion submodule to obtain the third interaction feature, the upsampled third interaction feature is concatenated with the first intermediate feature to obtain the fifth concatenated feature, the fifth concatenated feature is processed by the fifth convolutional layer to obtain the eleventh intermediate feature, the upsampled third interaction feature is concatenated with the second intermediate feature to obtain the sixth concatenated feature, the sixth concatenated feature is processed by the fifth convolutional layer to obtain the twelfth intermediate feature;

[0069] The eleventh intermediate feature and the twelfth intermediate feature are input together into the adaptive Token interaction module in the fourth feature fusion submodule to obtain the fourth interaction feature and use it as an aggregate feature that fuses the spatial characteristics and the spectral characteristics. Finally, the aggregate feature is processed by the second convolutional layer to obtain the target HR-MS image.

[0070] Furthermore, in the spatial-spectral Token learning network, the convolution kernel sizes of the first and second convolution layers are both 3×3, and the convolution kernel sizes of the third, fourth, and fifth convolution layers are all 1×1.

[0071] It should be noted that in the process of feature extraction of spatial-spectral Token learning network, the Token reconstruction module is used to generate Tokens that can describe spectral features, and the Token selection module is used to select Tokens that can describe effective spatial features. In the feature fusion stage, the adaptive Token interaction module is used to aggregate spectral features and spatial features to finally obtain the target HR-MS image.

[0072] The following will introduce the principles and functions of the Token reconstruction module, the Token selection module, and the adaptive Token interaction module in the spatial-spectral Token learning network, as well as the specific processing flow within each module, so that technicians in this field can understand it.

[0073] 1. Token reconstruction module

[0074] In the present invention, the Token reconstruction module (FGTR) reconstructs the amplitude component of the target HR-MS image through the implicit interaction between the Fourier domain tokens of the PAN image and the LR-MS image. The module is used to generate a set of tokens that can describe the adjusted spectral features of the PAN image and the LR-MS image. The input of the first Token reconstruction module is the implicit features of the PAN image and the implicit features of the LR-MS image, and the input of each subsequent Token reconstruction module is the two features output by the previous feature learning submodule. Taking the i-th Token reconstruction module as an example, its processing process can be written as:

[0075]

[0076] Wherein, FGTR represents Token Reconstruction Module (FGTR); x i and i Represents the input features of the i-th Token reconstruction module; Represents the output of the i-th Token reconstruction module.

[0077] In this embodiment, if Figure 3 As shown, firstly, the two features input to the Token reconstruction module are normalized and discrete Fourier transformed to obtain the Fourier domain Token of each input feature:

[0078] F x =DFT(LN(x i ))

[0079] F y =DFT(LN(y i ))

[0080] Among them, F x Represents the input feature x i Fourier domain Token; F y Represents the input feature y i Fourier domain Token; DFT represents discrete Fourier transform; LN(·) represents the normalization layer.

[0081] The Fourier domain Token of each feature is approximated in the form of complex numbers to facilitate implicit modeling. The specific calculation formula is as follows:

[0082]

[0083] Among them, r x The real mapping of Fourier domain Token representing the LR-MS image; r y The real mapping of Fourier domain Token representing the PAN image; p x represents the virtual mapping of the Fourier domain Token of the LR-MS image; p y The virtual mapping of Fourier domain Token representing the PAN image; They all represent the complex form of the Fourier domain Token of each feature. Indicates F x plural form of; Indicates F y The plural form of ; j represents the imaginary unit.

[0084] Then, the real mapping of the Fourier domain Token of the LR-MS image and the real mapping of the Fourier domain Token of the PAN image are connected to obtain the fused real mapping r s, connect the virtual mapping of the Fourier domain Token of the LR-MS image and the virtual mapping of the Fourier domain Token of the PAN image to obtain the fused virtual mapping p s , the calculation formula is as follows:

[0085] r s =[r x ; r y ]

[0086] p s =[p x ;p y ]

[0087] Subsequently, the Fourier domain MLP layer realizes the implicit interaction between the tokens extracted from the PAN image and the LR-MS image to reconstruct the spectral characteristics required for the target HR-MS image. Specifically, the fused real map and the fused virtual map are input into the Fourier domain MLP network formed by the cascade of L Fourier domain MLP layers, and the implicit feature interaction results in the Fourier domain are output and the inverse discrete Fourier transform operation is performed on them to obtain the output of the Token reconstruction module. Among them, the implicit feature interaction results in the Fourier domain can effectively simulate the intra-spectral relationship required for the target HR-MS image.

[0088] Among them, the first Fourier domain MLP layer input is the fused real map r s and fused virtual mapping p s ,Right now:

[0089] y 0 =r s +jp s

[0090] The specific processing process of the lth Fourier domain MLP layer is as follows:

[0091]

[0092] According to the multiplication rules of complex numbers, the above expression can be expanded to:

[0093]

[0094] In the formula, y l represents the implicit feature interaction result output by the lth Fourier domain MLP layer; r s l-1 ,p s l-1 They represent the input y of the l-th Fourier domain MLP layer respectively. l-1 The real and imaginary parts of ; σ represents the activation function; Respectively represent the complex weight matrix of the lth Fourier domain MLP layer The real and imaginary parts of They represent the complex bias of the lth Fourier domain MLP layer The real and imaginary parts of .

[0095] 2. Token Selection Module

[0096] In the present invention, the Token Selection Module (TKTS) is used to extract Tokens that describe the properties of the reconstructed space. It captures the most critical information for restoring the spatial characteristics required for the target HR-MS image by selecting the k Tokens with the highest correlation with the query matrix of the PAN image. In this module, the features of the input corresponding PAN image are first projected into a query matrix, a key matrix, and a value matrix; then the attention matrix is ​​calculated to describe the spatial pixel relationship between the Tokens of the PAN image; then the elements with lower attention in the attention matrix are masked to obtain the activated attention matrix; finally, the value matrix is ​​weighted with the activated attention matrix to obtain the output of each Token selection module. The input of the first Token selection module is the implicit feature of the PAN image, and the input of each Token selection module is the feature corresponding to the PAN image in the two features output by the previous feature learning submodule. Taking the i-th Token selection module as an example, its processing process can be written as:

[0097]

[0098] Where TKTS stands for Token Selection Module (TKTS); i represents the input features of the i-th Token reconstruction module, that is, the input features of the i-th Token selection module; Represents the output of the i-th Token selection module.

[0099] In this embodiment, if Figure 4 As shown in , the features input by the ith Token selection module are first projected into a query matrix, a key matrix, and a value matrix. Specifically, the features y input by the Token selection module are first projected into i After layer normalization, the first normalized features are obtained. The first normalized features are passed through three independent convolution branches respectively. The first convolution branch outputs the query matrix Q, the second convolution branch outputs the key matrix K, and the third convolution branch outputs the value matrix V.

[0100] In this embodiment, each convolution branch is composed of a standard convolution layer and a deep convolution layer cascaded in sequence, where the convolution kernel size of the standard convolution layer is 1×1, and the convolution kernel size of the deep convolution layer is 3×3. Therefore, the specific forms of the query matrix Q, the value matrix V, and the key matrix K can be written as:

[0101]

[0102] in, Represent the weight matrices of the standard convolution layer and the depth convolution layer in the first convolution branch respectively; Represent the weight matrices of the standard convolution layer and the depth convolution layer in the second convolution branch respectively; Represent the weight matrices of the standard convolutional layer and the depthwise convolutional layer in the third convolutional branch respectively.

[0103] Then the attention matrix S is calculated by the query matrix Q and the reshaped key matrix to describe the spatial pixel relationship between the tokens of the PAN image. The calculation formula of the attention matrix is ​​as follows:

[0104]

[0105] Where d represents the dimension of the key matrix K; T represents the matrix transpose.

[0106] Then, the elements in the attention matrix S are masked (here, irrelevant elements in the attention matrix S, i.e., elements with lower attention, are selected) to obtain the mask matrix Normalize the mask matrix to obtain the activation attention matrix M with density k:

[0107]

[0108] in, Represents a mask operation.

[0109] In this embodiment, the mask operation is as follows: the kth largest element value t in the fth row of the attention matrix is f As the relevance threshold, if the element value of the attention matrix row f and column g is greater than or equal to t f , then the element value of the attention matrix located at the fth row and the gth column is retained, otherwise the element value of the attention matrix located at the fth row and the gth column is reset to negative infinity:

[0110]

[0111] Until every row of the attention matrix is ​​judged, a complete mask matrix is ​​obtained.

[0112] After the above masking operation, the element values ​​of each row in the attention matrix that are less than the correlation threshold of the corresponding row are assigned negative infinity. When these element values ​​are normalized, their values ​​are close to 0. In this way, the part that contributes most to the self-attention score is retained, and other irrelevant information is deleted, thereby obtaining the activation attention matrix. Finally, the activation attention matrix M is multiplied by the value matrix V to obtain the output of the i-th Token selection module

[0113]

[0114] 3. Adaptive Token Interaction Module

[0115] The adaptive token interaction module uses the activation attention matrix obtained from the token selection module (TKTS) to dynamically modulate the channels of the spectral features extracted from the token reconstruction module (FGTR), aggregate the features generated by the token reconstruction module (FGTR) and the features generated by the token selection module, thereby fusing the spatial features and spectral features to achieve spatial position-aware adaptive fusion of spatial-spectral characteristics. In the adaptive token interaction module, each channel feature map of the features extracted from the token reconstruction module is regarded as a token, and they are projected to obtain query projection, key projection and value projection; the key projection is reshaped and dot product operation is performed with the query projection to obtain the attention map; then, the activation attention matrix obtained from the token selection module (TKTS) is used to weight the value projection; finally, the final output result is obtained through matrix reshaping operation. Taking the i-th adaptive token interaction module as an example, its processing process can be written as:

[0116] o i =ATI(p i ,q i )

[0117] Among them, ATI represents the adaptive Token interaction module; i represents the output of the i-th adaptive Token interaction module; p i ,q i Represents the input of the i-th adaptive Token interaction module.

[0118] For the adaptive Token interaction module in the first feature fusion submodule, its input is a set of Tokens describing the reconstructed spatial attributes and a set of Tokens describing the reconstructed spectral feature attributes output by the fourth feature learning submodule; for the adaptive Token interaction modules in the remaining feature fusion submodules, their input is the features output by the previous feature fusion submodule.

[0119] In this embodiment, if Figure 5 As shown in the figure, taking the adaptive Token interaction module in the first feature fusion submodule as an example, it transforms a set of Tokens (i.e., p 1 ) is regarded as a Token, and the second normalized feature is obtained after layer normalization. The second normalized feature is passed through three new convolution branches respectively, and the query projection is output by the first new convolution branch. The output key is projected by the second new convolution branch Projected by the output value of the third new convolution branch

[0120]

[0121] in, Respectively represent the weight matrices of the standard convolution layer and the depth convolution layer in the first new convolution branch; Respectively represent the weight matrices of the standard convolution layer and the depth convolution layer in the second new convolution branch; Represent the weight matrices of the standard convolutional layer and the depthwise convolutional layer in the third convolutional branch respectively.

[0122] Projected by query and the reshaped key projection Perform dot product operation and calculate the attention map A:

[0123]

[0124] in, are learnable parameters that adapt the attention map.

[0125] Project a set of tokens and values ​​describing the attributes of the reconstruction space Multiply them to get the weighted importance weight matrix, reshape the weighted importance weight matrix and perform dot product operation with the normalized attention map to get the weighted attention map, reshape the weighted attention map to get the output o of the adaptive Token interaction module in the first feature fusion submodule 1 :

[0126]

[0127] For the second, third and fourth feature fusion sub-modules, the eighth, tenth and twelfth intermediate features replace a group of Tokens describing the reconstructed spectral feature attributes input into the first feature fusion sub-module, and after layer normalization and convolution branch processing, the query projection, value projection and key projection of the second, third and fourth feature fusion sub-modules are obtained respectively; the seventh, ninth and eleventh intermediate features replace a group of Tokens describing the reconstructed spatial attributes input into the first feature fusion sub-module, and are used to multiply the value projections generated by the corresponding feature fusion sub-modules to obtain the weighted importance weight matrix of the corresponding feature fusion sub-module. The specific process will not be repeated here.

[0128] It should be noted that the above-mentioned spatial-spectral Token learning network also includes a downsampling module and an upsampling module. The specific structure of the downsampling module and the upsampling module is not limited by the present invention, as long as it can cooperate with the spatial-spectral Token learning network of the present invention and realize the corresponding downsampling and upsampling processes. In this embodiment, the downsampling module is implemented by a 4×4 convolutional layer with a step size, which doubles the number of channels of the feature map while downsampling the feature map; the upsampling module is implemented by a 2×2 deconvolution layer with a step size, which halves the number of channels of the feature map while upsampling the feature map.

[0129] It should also be noted that when the spatial-spectral Token learning network of the present invention is trained, the reconstruction loss is introduced into the total loss. So that the spatial-spectral Token learning network can generate HR-MS images similar to the corresponding true values. At the same time, an auxiliary loss is introduced This helps the Token Reconstruction Module (FGTR) to correct the correlation within the spectrum according to the true value. The total loss function used is Specifically:

[0130]

[0131] Among them, τ is the balance factor; represents reconstruction losses; Indicates auxiliary loss; X out is the HR-MS image output by the spatial-spectral Token learning network, X gt is the true value of the HR-MS image; A out is the amplitude spectrum of the output feature of the Token reconstruction module in the last feature learning submodule; A gt The amplitude spectrum is obtained by fast Fourier transforming the true value of the HR-MS image.

[0132] The present invention will now use a specific example to demonstrate the application effect of the full color sharpening method based on the spatial-spectral Token learning network described in the above embodiment on a specific data set, so as to facilitate understanding of the essence of the present invention.

[0133] Example

[0134] The specific implementation process of the full color sharpening method based on the spatial-spectral Token learning network used in this embodiment is as described above and will not be repeated here.

[0135] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and HR-MS image generation. Figure 6 shown.

[0136] 1. Data preprocessing stage

[0137] This embodiment conducts extensive experiments on three image sets collected by QuickBird, GaoFen-2, and WorldView-3 satellites. In order to compare with other methods, this embodiment completes experiments under the Wald protocol and full-scale settings. According to the Wald protocol, both PAN and MS images are downsampled to a lower scale so that the original MS image can be used as the real image. This embodiment uses a Gaussian filter to blur the original PAN / LR-MS / true value image pair and then downsamples it with a scaling factor of 4. Since the size of remote sensing images is very large, it is difficult to input them into the spatial-spectral Token learning network, so this embodiment crops these images into small blocks to form training and test sets. On the QuickBird dataset, this embodiment crops the test image into 356 small blocks to construct the test set, while the training set consists of 12119 small blocks randomly cropped from other images. Then, this embodiment performs the same processing on the WorldView-3 dataset containing 3639 test blocks and 11856 training blocks. On the GaoFen-2 dataset, 35725 training blocks and 3370 test blocks were generated. During the training process, this embodiment provides the model with PAN images of size 128×128×1 and LR-MS images of size 32×32×4. During the test process, the sizes of PAN and LR-MS images are 128×128×1 and 32×32×4, respectively. In addition, 70% of the dataset is divided into a training set and 30% is divided into a test set.

[0138] 2. Model training

[0139] In the training phase, the Adam optimizer with β1 = 0.9 and β2 = 0.999 is used for optimization, and the batch size is set to 4 and the number of iterations is 30000. The initial learning rate is set to 1×10 -4 , and decayed by 0.85 after every 3000 iterations. In addition, all experiments were conducted on the PyTorch framework using four NVIDIA RTX A6000 GPUs.

[0140] 3. HR-MS image generation

[0141] The image of the test set is directly used as input, and finally a HR-MS image with rich spatial features and rich spectral features is generated through the trained spatial-spectral Token learning network SSTL.

[0142] In the low-resolution experiment of the embodiment of the present invention, the test results of the WorldView-3 case are as follows: Figure 7 The test results of the QuickBrid case are shown in Figure 8 As shown in Figure 2, the test results of GaoFen-2 are as follows: Fig. 9 shown.

[0143] Analysis of the WorldView-3 case shows that the deep learning-based method has higher fusion accuracy than the traditional method, which is reflected in the lower brightness of its error map. Among the deep learning-based methods, the difference between the proposed method SSTL and the real image is the smallest and the fusion performance is the best, followed by LGTEUN and MDCUN.

[0144] By analyzing the QuickBrid case, it can be found that, in terms of deep learning-based methods, the reconstruction performance of MSDCNN, DCFNet, and ARFNet is poor, and the brightness values ​​of their error maps are high, even similar to the effects of traditional methods. In addition, the method SSTL of the present invention has significant advantages in preserving spectral information, and the visual difference between its reconstruction result and the real image in color distribution is minimal.

[0145] By analyzing the GaoFen-2 case, it can be found that the SSTL method of the present invention shows excellent fusion performance. At the same time, it is found that, except for ARFNet, the reconstruction effect of the deep learning-based method is better than that of the traditional method.

[0146] In order to evaluate the performance of the model in real-world scenarios and its generalization ability, this embodiment further experiments on some full-resolution examples. Specifically, the pre-trained model obtained from the down-resolution data is directly applied to some full-resolution samples. In the full-resolution experiment, the test results of the WorldView-3 case are as follows: Fig.10 The test results of the QuickBrid case are shown in Fig.11 As shown in Figure 2, the test results of GaoFen-2 are as follows: Fig.12 shown. Fig.10 , Fig.11 and Fig.12 Visual comparisons of several methods on representative full-resolution test cases are shown. From these figures, it can be easily observed that the proposed spatial-spectral token learning network presents better spatial texture while maintaining pleasing spectral properties. In contrast, other competing methods either contain blurred spatial textures and distorted edges, or exhibit obvious spectral distortions.

[0147] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A panchromatic sharpening method based on a spatial-spectral Token learning network, characterized in that: The following steps are involved: A set of PAN images and LR-MS images to be pan-sharpened are input into the trained spatial-spectral Token learning network. The PAN images and LR-MS images are respectively subjected to feature extraction by the first convolutional layer. The extracted latent features of the PAN images and the latent features of the LR-MS images are input into the feature learning module. A set of Tokens describing the reconstructed spectral feature attributes and a set of Tokens describing the reconstructed spatial attributes output by the feature learning module are input into the feature fusion module to obtain aggregated features that fuse spatial characteristics and spectral characteristics. The aggregated features are processed by the second convolutional layer to obtain the target HR-MS image. The spatial-spectral Token learning network has a U-shaped structural feature. The feature learning module includes four feature learning sub-modules, the first three feature learning sub-modules all include a Token reconstruction module, a Token selection module and a down-sampling module, and the last feature learning sub-module only includes a Token reconstruction module and a Token selection module; the feature fusion module includes four feature fusion sub-modules, the first three feature fusion sub-modules all include an adaptive Token interaction module and an up-sampling module, and the last feature fusion sub-module only includes an adaptive Token interaction module, and the first three feature fusion sub-modules each perform feature fusion by means of jump connection before entering the next adjacent feature fusion sub-module to reduce the information loss caused by the down-sampling operation.

2. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 1, characterized in that: The specific processing flow in the spatial-spectral Token learning network is as follows: after the PAN image passes through the first convolution layer, the implicit features of the PAN image are obtained; after the LR-MS image passes through the first convolution layer, the implicit features of the LR-MS image are obtained; the two implicit features are used as inputs of the first feature learning submodule; the implicit features of the PAN image are first processed by the Token selection module in the first feature learning submodule to obtain the first intermediate features; the implicit features of the PAN image and the implicit features of the LR-MS image are then input together into the Token reconstruction module in the first feature learning submodule to obtain the second intermediate features; The first intermediate feature and the second intermediate feature are respectively downsampled as inputs of the second feature learning submodule, the Token selection module in the second feature learning submodule first processes the downsampled first intermediate feature to obtain a third intermediate feature, and then the downsampled first intermediate feature and the downsampled second intermediate feature are input together to the Token reconstruction module in the second feature learning submodule to obtain a fourth intermediate feature, the third intermediate feature and the fourth intermediate feature are respectively downsampled as inputs of the third feature learning submodule, the Token selection module in the third feature learning submodule first processes the downsampled third intermediate feature to obtain a fifth intermediate feature, and then the downsampled third intermediate feature and the downsampled fourth intermediate feature are input together to the Token reconstruction module in the third feature learning submodule to obtain a sixth intermediate feature; The fifth intermediate feature and the sixth intermediate feature are respectively downsampled as inputs of the fourth feature learning submodule, and the Token selection module in the fourth feature learning submodule first processes the downsampled fifth intermediate feature to obtain a group of Tokens describing the attributes of the reconstructed space, and then the downsampled fifth intermediate feature and the downsampled sixth intermediate feature are input together into the Token reconstruction module in the fourth feature learning submodule to obtain a group of Tokens describing the attributes of the reconstructed spectral features; A group of Tokens describing the reconstructed spectral feature attributes and a group of Tokens describing the reconstructed spatial attributes are input together into the adaptive Token interaction module in the first feature fusion submodule to obtain a first interaction feature, the upsampled first interaction feature is concatenated with the fifth intermediate feature to obtain a first concatenated feature, the first concatenated feature is processed by the third convolutional layer to obtain a seventh intermediate feature, the upsampled first interaction feature is concatenated with the sixth intermediate feature to obtain a second concatenated feature, the second concatenated feature is processed by the third convolutional layer to obtain an eighth intermediate feature; The seventh intermediate feature and the eighth intermediate feature are input together into the adaptive Token interaction module in the second feature fusion submodule to obtain the second interaction feature, the upsampled second interaction feature is concatenated with the third intermediate feature to obtain the third concatenated feature, the third concatenated feature is processed by the fourth convolutional layer to obtain the ninth intermediate feature, the upsampled second interaction feature is concatenated with the fourth intermediate feature to obtain the fourth concatenated feature, the fourth concatenated feature is processed by the fourth convolutional layer to obtain the tenth intermediate feature; The ninth intermediate feature and the tenth intermediate feature are input together into the adaptive Token interaction module in the third feature fusion submodule to obtain the third interaction feature, the upsampled third interaction feature is concatenated with the first intermediate feature to obtain the fifth concatenated feature, the fifth concatenated feature is processed by the fifth convolutional layer to obtain the eleventh intermediate feature, the upsampled third interaction feature is concatenated with the second intermediate feature to obtain the sixth concatenated feature, the sixth concatenated feature is processed by the fifth convolutional layer to obtain the twelfth intermediate feature; The eleventh intermediate feature and the twelfth intermediate feature are input together into the adaptive Token interaction module in the fourth feature fusion submodule to obtain the fourth interaction feature and use it as an aggregate feature that fuses the spatial characteristics and the spectral characteristics. Finally, the aggregate feature is processed by the second convolutional layer to obtain the target HR-MS image.

3. The panchromatic sharpening method based on spatial-spectral Token learning network according to claim 2, characterized in that: The specific processing flow in the Token reconstruction module is as follows: First, the two features input to the Token reconstruction module are normalized and discrete Fourier transformed to obtain the Fourier domain Token of each input feature. Then, the Fourier domain Token of each feature is approximated in the form of complex numbers to facilitate implicit modeling. The specific calculation formula is as follows: Among them, r x The real mapping of Fourier domain Token representing the LR-MS image; r y The real mapping of Fourier domain Token representing the PAN image; p x represents the virtual mapping of the Fourier domain Token of the LR-MS image; p y The virtual mapping of Fourier domain Token representing the PAN image; They all represent the complex form of the Fourier domain Token of each feature; j represents the imaginary unit. Then, the real mapping of the Fourier domain Token of the LR-MS image and the real mapping of the Fourier domain Token of the PAN image are connected to obtain a fused real mapping, and the virtual mapping of the Fourier domain Token of the LR-MS image and the virtual mapping of the Fourier domain Token of the PAN image are connected to obtain a fused virtual mapping. The fused real mapping and the fused virtual mapping are input into a Fourier domain MLP network formed by cascading L Fourier domain MLP layers. The implicit feature interaction result in the Fourier domain is output and the discrete Fourier inverse transform operation is performed on it to obtain the output of the Token reconstruction module.

4. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 3, characterized in that: The specific processing process of the lth Fourier domain MLP layer is as follows: In the formula, y l represents the implicit feature interaction result output by the lth Fourier domain MLP layer; r s l-1 ,p s l-1 denote the real and imaginary parts of the input to the lth Fourier domain MLP layer, respectively; σ denotes the activation function; denote the real and imaginary parts of the complex weight matrix of the lth Fourier domain MLP layer, respectively; denote the real and imaginary parts of the complex bias of the lth Fourier domain MLP layer, respectively.

5. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 2, characterized in that: The specific processing flow in the Token selection module is as follows: first, the features input to the Token selection module are normalized by layers to obtain the first normalized features, and the first normalized features are respectively passed through three convolution branches, the first convolution branch outputs the query matrix, the second convolution branch outputs the key matrix K, and the third convolution branch outputs the value matrix V. The attention matrix S is calculated by the query matrix Q and the reshaped key matrix: Where d represents the dimension of the key matrix; T represents the matrix transpose; Then, the elements in the attention matrix are masked to obtain the mask matrix, and the mask matrix is ​​normalized to obtain the activation attention matrix with a dense rate. Finally, the activation attention matrix is ​​multiplied by the value matrix to obtain the output of the Token selection module.

6. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 5, characterized in that: In the Token selection module, the specific process of the mask operation is as follows: the kth largest element value t in the fth row of the attention matrix f As the relevance threshold, if the element value of the attention matrix row f and column g is greater than or equal to t f , then the element value of the attention matrix located in the fth row and gth column is retained, otherwise the element value of the attention matrix located in the fth row and gth column is reset to negative infinity until each row of the attention matrix is ​​judged and a complete mask matrix is ​​obtained.

7. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 2, characterized in that: In the first feature fusion submodule, the specific processing flow of the adaptive Token interaction module is as follows: a set of Tokens describing the characteristic attributes of the reconstructed spectrum is normalized by layers to obtain the second normalized features, and the second normalized features are respectively passed through three new convolution branches, and the query projection is output by the first new convolution branch. The output value of the third new convolution branch is projected, and the output of the second new convolution branch is reshaped to obtain the key projection. The query projection and the reshaped key projection are dot-producted to calculate the attention map A: in, represents the result of key projection after reshaping; α is a learnable parameter for adapting the attention map; Represents the dot product operation; A set of Tokens describing the attributes of the reconstructed space are multiplied by the value projection to obtain a weighted importance weight matrix. The weighted importance weight matrix is ​​reshaped and then dot-producted with the normalized attention map to obtain a weighted attention map. After the weighted attention map is reshaped, the output of the adaptive Token interaction module in the first feature fusion submodule is obtained.

8. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 7, characterized in that: In the second, third and fourth feature fusion sub-modules, the eighth, tenth and twelfth intermediate features are used for layer normalization and processed by convolution branches, and the seventh, ninth and eleventh intermediate features are used to multiply the value projections generated by their respective corresponding feature fusion sub-modules to obtain the weighted importance weight matrix of the corresponding feature fusion sub-module.

9. The method for panchromatic sharpening based on a spatial-spectral Token learning network according to claim 5 or 7, characterized in that: Each convolution branch is composed of a standard convolution layer and a deep convolution layer cascaded in sequence, where the convolution kernel size of the standard convolution layer is 1×1, and the convolution kernel size of the deep convolution layer is 3×3.

10. The method for panchromatic sharpening based on spatial-spectral Token learning network according to claim 1, characterized in that: The loss function used by the spatial-spectral Token learning network during training is specifically: Among them, τ is the balance factor; represents reconstruction losses; Indicates auxiliary loss; X out is the HR-MS image output by the spatial-spectral Token learning network, X gt is the true value of the HR-MS image; A out is the amplitude spectrum of the output feature of the Token reconstruction module in the last feature learning submodule; A gt The amplitude spectrum is obtained by fast Fourier transforming the true value of the HR-MS image.

Images