Pan-sharpening method based on spatial-spectral Token learning network

Abstract

This invention discloses a panchromatic sharpening method (SSTL) based on a spatial-spectral token learning network. Specifically, it proposes a neural network framework for panchromatic sharpening tasks that can reconstruct and fuse spatial and spectral features: the spatial-spectral token learning network, which comprises three novel modules. A Fourier-guided token reconstruction module reconstructs spectral characteristics by applying complex feature interaction rules to the interaction features of Fourier domain tokens extracted from panchromatic and low-resolution multispectral images. A token selection module forces the spatial-spectral token learning network to reconstruct the spatial characteristics of the high-resolution multispectral image from the most relevant spatial regions of the panchromatic image. An adaptive token interaction module uses weights learned from the token selection module during the channel interaction of spectral features, thereby achieving seamless integration of spatial and spectral characteristics. Furthermore, experiments on publicly available datasets demonstrate the effectiveness of the proposed method.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and computer vision, and particularly relates to a panchromatic sharpening method based on a spatial-spectral token learning network. Background Technology

[0002] With the increasing demand for Earth observation and monitoring, existing optical satellites can simultaneously record low-resolution multispectral (LR-MS) images and corresponding high-resolution panchromatic (PAN) images of the same scene. Due to the physical limitations of existing satellite sensors, the recorded LR-MS images typically contain rich spectral features but relatively sparse spatial features, while their corresponding PAN images contain rich spatial features but sparse spectral features. However, in practical applications, such as agriculture and forestry, meteorology, ecological environment assessment, urban planning and construction, and military reconnaissance and monitoring, remote sensing images with multiple spectral bands and high spatial resolution, namely high-resolution multispectral (HR-MS) images, play a more important role.

[0003] To address this issue, researchers proposed a panchromatic sharpening method. It obtains an HR-MS image by fusing spatial and spectral features from PAN and LR-MS images.

[0004] Early panchromatic sharpening methods mainly included component replacement (CS), multi-resolution analysis (MRA), and variational optimization (VO) methods. Specifically, the CS method uses spatial details from the PAN image to replace corresponding details in the LR-MS image. Although this method is simple and has high fidelity, it often introduces spectral distortion during the fusion process. The MRA method injects the spatial structure from the PAN image into the LR-MS image, better preserving spectral characteristics, but causing spatial distortion. VO-based methods construct regularization functions based on physical priors to optimize the model; although they achieve better results, the optimization effect is limited.

[0005] In recent years, panchromatic sharpening methods based on convolutional neural networks have been widely used due to their powerful ability to learn task-specific representations. These methods mainly employ two fusion strategies to utilize the spatial-spectral properties of PAN and LR-MS images: image-level fusion and feature-level fusion. Image-level fusion directly concatenates the LR-MS and PAN images along the channel dimension before inputting features into the network, which may cause the model to confuse spectral and spatial information during feature extraction, thus distorting the model. Meanwhile, feature-level fusion typically uses a two-stream network to extract spatial features from the PAN image and spectral features from the LR-MS image separately, and then combines these features in the latent feature space. However, this strategy lacks modal interaction between the PAN and LR-MS images during the feature extraction stage, so the learned model cannot effectively utilize the cross-feature space dependencies between the LR-MS and PAN images.

[0006] Unlike CNNs, Transformer-style networks can effectively capture long-range dependencies between tokens of different orders in the input data based on their multi-head attention mechanism. Therefore, many recent studies have extended Transformers to the realm of panchromatic sharpening. However, these Transformer-based methods still fail to consider the relationships between PAN, LR-MS, and target HR-MS images from a spatial-spectral perspective. Summary of the Invention

[0007] The purpose of this invention is to solve the problems existing in the prior art and to use deep learning and computer vision technologies to extract and fuse features of PAN and LR-MS images with different spectral and spatial features, thereby providing a panchromatic sharpening method based on a spatial-spectral token learning network.

[0008] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0009] A panchromatic sharpening method based on a spatial-spectral token learning network includes the following steps: a set of PAN images and LR-MS images to be panchromatic sharpened are input into a trained spatial-spectral token learning network. The PAN images and LR-MS images are each subjected to feature extraction in the first convolutional layer. The extracted latent features of the PAN images and the LR-MS images are input into a feature learning module. A set of tokens describing the reconstructed spectral feature attributes and a set of tokens describing the reconstructed spatial attributes output by the feature learning module are input into a feature fusion module to obtain aggregated features that fuse spatial and spectral characteristics. After processing the aggregated features through a second convolutional layer, the target HR-MS image is obtained.

[0010] The spatial-spectral token learning network has a U-shaped structure. The feature learning module includes four feature learning sub-modules. The first three feature learning sub-modules each include a token reconstruction module, a token selection module, and a downsampling module. The last feature learning sub-module only includes a token reconstruction module and a token selection module. The feature fusion module includes four feature fusion sub-modules. The first three feature fusion sub-modules each include an adaptive token interaction module and an upsampling module. The last feature fusion sub-module only includes an adaptive token interaction module. Furthermore, the first three feature fusion sub-modules perform feature fusion through skip connections before entering the next adjacent feature fusion sub-module to reduce the information loss caused by the downsampling operation.

[0011] Based on the above scheme, each step can be implemented in the following preferred manner.

[0012] Preferably, the specific processing flow in the spatial-spectral token learning network is as follows: the PAN image passes through the first convolutional layer to obtain the latent features of the PAN image, and the LR-MS image passes through the first convolutional layer to obtain the latent features of the LR-MS image. The two latent features are used as inputs to the first feature learning submodule. The token selection module in the first feature learning submodule processes the latent features of the PAN image to obtain the first intermediate features. Then, the latent features of the PAN image and the latent features of the LR-MS image are input together into the token reconstruction module in the first feature learning submodule to obtain the second intermediate features.

[0013] The first and second intermediate features are each downsampled and used as input to the second feature learning submodule. The token selection module in the second feature learning submodule processes the downsampled first intermediate feature to obtain the third intermediate feature. The downsampled first and second intermediate features are then input together into the token reconstruction module in the second feature learning submodule to obtain the fourth intermediate feature. The third and fourth intermediate features are each downsampled and used as input to the third feature learning submodule. The token selection module in the third feature learning submodule processes the downsampled third intermediate feature to obtain the fifth intermediate feature. The downsampled third and fourth intermediate features are then input together into the token reconstruction module in the third feature learning submodule to obtain the sixth intermediate feature.

[0014] The fifth and sixth intermediate features are downsampled and used as input to the fourth feature learning submodule. The Token selection module in the fourth feature learning submodule processes the downsampled fifth intermediate feature to obtain a set of tokens describing the reconstructed spatial attributes. Then, the downsampled fifth and sixth intermediate features are input together into the Token reconstruction module in the fourth feature learning submodule to obtain a set of tokens describing the reconstructed spectral feature attributes.

[0015] A set of tokens describing the reconstructed spectral features and a set of tokens describing the reconstructed spatial features are input together into the adaptive token interaction module in the first feature fusion submodule to obtain the first interactive feature. The upsampled first interactive feature is concatenated with the fifth intermediate feature to obtain the first concatenated feature. After the first concatenated feature is processed by the third convolutional layer, the seventh intermediate feature is obtained. The upsampled first interactive feature is concatenated with the sixth intermediate feature to obtain the second concatenated feature. After the second concatenated feature is processed by the third convolutional layer, the eighth intermediate feature is obtained.

[0016] The seventh and eighth intermediate features are input together into the adaptive Token interaction module in the second feature fusion submodule to obtain the second interaction feature. The upsampled second interaction feature is concatenated with the third intermediate feature to obtain the third concatenated feature. The third concatenated feature is processed by the fourth convolutional layer to obtain the ninth intermediate feature. The upsampled second interaction feature is concatenated with the fourth intermediate feature to obtain the fourth concatenated feature. The fourth concatenated feature is processed by the fourth convolutional layer to obtain the tenth intermediate feature.

[0017] The ninth and tenth intermediate features are input together into the adaptive Token interaction module in the third feature fusion submodule to obtain the third interaction feature. The upsampled third interaction feature is concatenated with the first intermediate feature to obtain the fifth concatenated feature. The fifth concatenated feature is processed by the fifth convolutional layer to obtain the eleventh intermediate feature. The upsampled third interaction feature is concatenated with the second intermediate feature to obtain the sixth concatenated feature. The sixth concatenated feature is processed by the fifth convolutional layer to obtain the twelfth intermediate feature.

[0018] The eleventh and twelfth intermediate features are input together into the adaptive Token interaction module in the fourth feature fusion submodule to obtain the fourth interaction feature, which is used as the aggregated feature to fuse spatial and spectral characteristics. Finally, the aggregated feature is processed by the second convolutional layer to obtain the target HR-MS image.

[0019] Preferably, the specific processing flow in the Token reconstruction module is as follows:

[0020] First, the two features input to the Token reconstruction module are normalized and subjected to Discrete Fourier Transform to obtain the Fourier domain Token for each input feature. Then, the Fourier domain Token for each feature is approximated in complex form to facilitate implicit modeling. The specific calculation formula is as follows:

[0021]

[0022] Where, r x The real mapping representing the Fourier domain token of an LR-MS image; r y The real mapping of the Fourier domain token of the PAN image; p x The virtual mapping representing the Fourier domain token of an LR-MS image; p y The virtual mapping representing the Fourier domain token of the PAN image; Each represents the complex form of the Fourier domain Token for each feature; j represents the imaginary unit.

[0023] Then, the real mapping of the Fourier domain token of the LR-MS image and the real mapping of the Fourier domain token of the PAN image are connected to obtain the fused real mapping. The virtual mapping of the Fourier domain token of the LR-MS image and the virtual mapping of the Fourier domain token of the PAN image are connected to obtain the fused virtual mapping. The fused real mapping and the fused virtual mapping are input into a Fourier domain MLP network composed of L cascaded Fourier domain MLP layers. The implicit feature interaction result of the Fourier domain is output and subjected to discrete inverse Fourier transform operation to obtain the output of the token reconstruction module.

[0024] The specific processing procedure for the l-th Fourier domain MLP layer is as follows:

[0025]

[0026] In the formula, y l The implicit feature interaction result is represented by the output of the l-th Fourier domain MLP layer; r s l-1 ,p s l-1 Let represent the real and imaginary parts of the input to the l-th Fourier domain MLP layer, respectively; σ represents the activation function. Let represent the real and imaginary parts of the complex weight matrix of the l-th Fourier domain MLP layer, respectively; They represent the real and imaginary parts of the complex bias of the l-th Fourier domain MLP layer, respectively.

[0027] Preferably, the specific processing flow in the Token selection module is as follows: First, the features input to the Token selection module are normalized to obtain the first normalized features. The first normalized features are then processed through three convolutional branches. The first convolutional branch outputs the query matrix, the second convolutional branch outputs the key matrix K, and the third convolutional branch outputs the value matrix V. The attention matrix S is calculated from the query matrix Q and the key matrix after shape reshaping.

[0028]

[0029] Where d represents the dimension of the key matrix; T represents the matrix transpose;

[0030] Next, the elements in the attention matrix are masked to obtain a mask matrix. The mask matrix is ​​then normalized to obtain an activation attention matrix with a density ratio. Finally, the activation attention matrix is ​​multiplied by the value matrix to obtain the output of the token selection module.

[0031] As a preferred approach, the masking operation in the Token selection module is performed as follows: The k-th largest element t in the f-th row of the attention matrix is ​​selected... f As a relevance threshold, if the value of the element in the f-th row and g-th column of the attention matrix is ​​greater than or equal to t f If the value is positive, the element in row f and column g of the attention matrix is ​​retained; otherwise, the element in row f and column g of the attention matrix is ​​reset to negative infinity. This process continues until each row of the attention matrix has been evaluated, resulting in a complete mask matrix.

[0032] As a preferred embodiment, the specific processing flow of the adaptive Token interaction module in the first feature fusion submodule is as follows: A set of tokens describing the reconstructed spectral feature attributes are normalized through a layer to obtain the second normalized feature. The second normalized feature is then passed through three new convolutional branches, and the query projection is output by the first new convolutional branch. Projecting the output value of the third new convolutional branch onto the key projection after reshaping the output of the second new convolutional branch, we then perform a dot product operation between the query projection and the reshaped key projection to calculate attention map A.

[0033]

[0034] in, The shape is reshaped after key projection; α is a learnable parameter of the adaptive attention map. This represents the dot product operation;

[0035] A set of tokens describing the reconstructed spatial attributes are multiplied by the value projection to obtain a weighted importance matrix. After reshaping the weighted importance matrix, a dot product operation is performed with the normalized attention map to obtain a weighted attention map. After reshaping the weighted attention map, the output of the adaptive token interaction module in the first feature fusion submodule is obtained.

[0036] Preferably, in the second, third, and fourth feature fusion sub-modules, the eighth, tenth, and twelfth intermediate features are used for layer normalization and convolutional branching, and the seventh, ninth, and eleventh intermediate features are multiplied by the projection of the values ​​generated by their respective feature fusion sub-modules to obtain the weighted importance matrix of the corresponding feature fusion sub-module.

[0037] Preferably, each convolutional branch consists of a standard convolutional layer and a depth convolutional layer cascaded together, wherein the kernel size of the standard convolutional layer is 1×1 and the kernel size of the depth convolutional layer is 3×3.

[0038] Preferably, the loss function used during training of the spatial-spectral token learning network is as follows:

[0039]

[0040] Where τ is the balance factor; Indicates the losses incurred during reconstruction; Indicates auxiliary loss; X out It is the HR-MS image output by the spatial-spectral token learning network, X gt It is the ground truth of the HR-MS image; A out It is the amplitude spectrum of the features output by the Token reconstruction module in the last feature learning submodule; A gt The amplitude spectrum is obtained by performing a fast Fourier transform on the true value of the HR-MS image.

[0041] Compared with the prior art, the present invention has the following advantages:

[0042] This invention considers that in the panchromatic sharpening problem, PAN images and LR-MS images have different spatial and spectral features, and that the implicit interaction between the extracted Fourier domain features can reconstruct the amplitude of the target Fourier domain features. Based on this, a Token reconstruction module is designed to model the spectral characteristics of the target HR-MS image when reconstructing the target spectral characteristics. This invention also considers the characteristics of large-area Earth observation scenarios, such as the existence of redundant features. Therefore, when dealing with feature redundancy, a Token selection module is designed, enabling the spatial-spectral Token learning network to capture the most relevant spatial features on the target HR-MS image while maintaining appropriate spatial selection complexity. Finally, this invention considers the positional dependence of the extracted spectral features, and therefore designs an adaptive Token interaction module to dynamically adjust the spectral features according to the spatial features during spatial-spectral feature fusion.

[0043] This invention constructs a spatial-spectral token learning network for panchromatic sharpening based on a token reconstruction module, a token selection module, and an adaptive token interaction module. Addressing the spatial-spectral characteristics of panchromatic sharpening, this invention designs a panchromatic sharpening method, SSTL, based on the spatial-spectral token learning network, effectively extracting and fusing spatial and spectral features. Experiments on the WorldView-3, QuickBird, and GaoFen-2 datasets demonstrate that, compared to previous state-of-the-art methods, the introduction of the aforementioned modules results in superior performance of the spatial-spectral token learning network. Attached Figure Description

[0044] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0045] Figure 2 This is a schematic diagram of the overall structure of the spatial-spectral token learning network of the present invention;

[0046] Figure 3 This is a schematic diagram of the token reconstruction module in a spatial-spectral token learning network.

[0047] Figure 4 A schematic diagram of the token selection module structure in a spatial-spectral token learning network;

[0048] Figure 5 This is a schematic diagram of the adaptive token interaction module structure in a spatial-spectral token learning network.

[0049] Figure 6This is a flowchart illustrating the training and testing process of a spatial-spectral token learning network in an embodiment of the present invention.

[0050] Figure 7 This is a reconstruction result image of the WorldView-3 case in the low-resolution experiment of this invention embodiment;

[0051] Figure 8 This is a reconstruction result of the QuickBird case in the low-resolution experiment of this invention embodiment;

[0052] Figure 9 This is a reconstruction result of the GaoFen-2 case in the low-resolution experiment of this invention.

[0053] Figure 10 This is a reconstruction result image of the WorldView-3 case in the full-resolution experiment of this invention embodiment;

[0054] Figure 11 This is a reconstruction result of the QuickBird case in the full-resolution experiment of this invention.

[0055] Figure 12 This is a reconstruction result of the GaoFen-2 case in the full-resolution experiment of this invention. Detailed Implementation

[0056] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0057] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0058] The panchromatic sharpening problem aims to overcome technological limitations by acquiring remote sensing images with rich spatial and spectral features simultaneously. Its inputs are a PAN image (rich in spatial features but relatively sparse in spectral features) and an LR-MS image (rich in spectral features but relatively sparse in spatial features), and the output is an HR-MS image (rich in both spectral and spatial features). Effectively fusing spectral and spatial features is a key issue that panchromatic sharpening needs to address. Existing techniques for panchromatic sharpening have not adequately considered the relationship between spatial and spectral features. Therefore, this invention comprehensively studies the relationship between the target HR-MS image and its corresponding input PAN and LR-MS images in the spectral and spatial domains. During the research, it was first discovered that the spatial distribution of the PAN image is very similar to the spatial distribution of each spectrum of its corresponding HR-MS image. This led to the conclusion that the spatial properties of the target HR-MS image can be reconstructed through the feature interaction within the tokens of the PAN image. Secondly, it was found that the intraspectral relationship of the LR-MS image is very different from that of the corresponding HR-MS image. Therefore, it was concluded that using only the LR-MS image to reconstruct the intraspectral relationship of the target HR-MS image is not the optimal method. In other words, it is necessary to achieve effective feature interaction between the tokens of the PAN image and the LR-MS image. In addition, it was found that the spectral characteristics can be reconstructed by reconstructing the amplitude in the Fourier domain.

[0059] Based on the above findings, this invention adopts a novel approach to processing panchromatic sharpening tasks: (1) implicitly interacting Fourier domain tokens extracted from PAN and LR-MS images to reconstruct the spectral characteristics of the target HR-MS image; (2) selecting only the tokens corresponding to the top k highest attention values ​​in the PAN image to capture the spatial characteristics most relevant to the reconstruction of the spatial characteristics of the target HR-MS image provided by the PAN image. Therefore, this invention proposes a spatial-spectral token learning network, and based on this network, designs a panchromatic sharpening method.

[0060] like Figure 1As shown, in a preferred embodiment of the present invention, the specific steps of the above-mentioned pancolor sharpening method based on the spatial-spectral token learning network are as follows: A set of PAN images (P) and LR-MS images (L) to be pancolor sharpened are input into the trained spatial-spectral token learning network. The PAN images and LR-MS images are each subjected to feature extraction in the first convolutional layer. The latent features y0 of the extracted PAN images and the latent features x0 of the extracted LR-MS images are input into the feature learning module. A set of tokens describing the reconstructed spectral feature attributes and a set of tokens describing the reconstructed spatial attributes output by the feature learning module are input into the feature fusion module to obtain the aggregated features that fuse spatial and spectral characteristics. After the aggregated features are processed by the second convolutional layer, the target HR-MS image is obtained.

[0061] The spatial-spectral token learning network has a U-shaped structure. The feature learning module includes four feature learning sub-modules. The first three feature learning sub-modules each include a token reconstruction module (FGTR), a token selection module (TKTS), and a downsampling module. The last feature learning sub-module only includes a token reconstruction module and a token selection module. The feature fusion module includes four feature fusion sub-modules. The first three feature fusion sub-modules each include an adaptive token interaction module (ATI) and an upsampling module. The last feature fusion sub-module only includes an adaptive token interaction module. Furthermore, the first three feature fusion sub-modules perform feature fusion through skip connections before entering the next adjacent feature fusion sub-module to reduce the information loss caused by the downsampling operation.

[0062] It should be noted that, in this invention, the structure of the spatial-spectral token learning network used for pan-color sharpening is as follows: Figure 2 As shown below, the specific processing flow in the spatial-spectral token learning network will be introduced to facilitate understanding by those skilled in the art.

[0063] exist Figure 2 In the process, the PAN image (P) is processed through the first convolutional layer to obtain the latent feature y0 of the PAN image, and the LR-MS image is processed through the first convolutional layer to obtain the latent feature x0 of the LR-MS image. The two latent features are used as inputs to the first feature learning submodule. The latent feature y0 of the PAN image is first processed by the Token selection module in the first feature learning submodule to obtain the first intermediate feature. Then, the latent feature y0 of the PAN image and the latent feature x0 of the LR-MS image are input together into the Token reconstruction module in the first feature learning submodule to obtain the second intermediate feature.

[0064] The first and second intermediate features are each downsampled and used as input to the second feature learning submodule. The token selection module in the second feature learning submodule processes the downsampled first intermediate feature to obtain the third intermediate feature. The downsampled first and second intermediate features are then input together into the token reconstruction module in the second feature learning submodule to obtain the fourth intermediate feature. The third and fourth intermediate features are each downsampled and used as input to the third feature learning submodule. The token selection module in the third feature learning submodule processes the downsampled third intermediate feature to obtain the fifth intermediate feature. The downsampled third and fourth intermediate features are then input together into the token reconstruction module in the third feature learning submodule to obtain the sixth intermediate feature.

[0065] The fifth and sixth intermediate features are each downsampled and used as input to the fourth feature learning submodule. The Token selection module in the fourth feature learning submodule first processes the downsampled fifth intermediate feature to obtain a set of tokens describing the reconstructed spatial attributes. Then, the downsampled fifth and sixth intermediate features are input together into the Token reconstruction module in the fourth feature learning submodule to obtain a set of tokens describing the reconstructed spectral feature attributes.

[0066] A set of tokens describing the reconstructed spectral features and a set of tokens describing the reconstructed spatial features are input together into the adaptive token interaction module in the first feature fusion submodule to obtain the first interactive feature. The upsampled first interactive feature is concatenated with the fifth intermediate feature to obtain the first concatenated feature. After the first concatenated feature is processed by the third convolutional layer, the seventh intermediate feature is obtained. The upsampled first interactive feature is concatenated with the sixth intermediate feature to obtain the second concatenated feature. After the second concatenated feature is processed by the third convolutional layer, the eighth intermediate feature is obtained.

[0067] The seventh and eighth intermediate features are input together into the adaptive Token interaction module in the second feature fusion submodule to obtain the second interaction feature. The upsampled second interaction feature is concatenated with the third intermediate feature to obtain the third concatenated feature. The third concatenated feature is processed by the fourth convolutional layer to obtain the ninth intermediate feature. The upsampled second interaction feature is concatenated with the fourth intermediate feature to obtain the fourth concatenated feature. The fourth concatenated feature is processed by the fourth convolutional layer to obtain the tenth intermediate feature.

[0068] The ninth and tenth intermediate features are input together into the adaptive Token interaction module in the third feature fusion submodule to obtain the third interaction feature. The upsampled third interaction feature is concatenated with the first intermediate feature to obtain the fifth concatenated feature. The fifth concatenated feature is processed by the fifth convolutional layer to obtain the eleventh intermediate feature. The upsampled third interaction feature is concatenated with the second intermediate feature to obtain the sixth concatenated feature. The sixth concatenated feature is processed by the fifth convolutional layer to obtain the twelfth intermediate feature.

[0069] The eleventh and twelfth intermediate features are input together into the adaptive Token interaction module in the fourth feature fusion submodule to obtain the fourth interaction feature, which is used as the aggregated feature to fuse spatial and spectral characteristics. Finally, the aggregated feature is processed by the second convolutional layer to obtain the target HR-MS image.

[0070] Furthermore, in the spatial-spectral token learning network, the kernel size of the first and second convolutional layers is 3×3, and the kernel size of the third, fourth and fifth convolutional layers is 1×1.

[0071] It should be noted that during the feature extraction process of the spatial-spectral Token learning network, the Token reconstruction module generates Tokens that can describe spectral features, the Token selection module selects Tokens that can describe effective spatial features, and in the feature fusion stage, the adaptive Token interaction module aggregates spectral features and spatial features to finally obtain the target HR-MS image.

[0072] The following section will introduce the principles and functions of the Token Reconstruction Module, Token Selection Module, and Adaptive Token Interaction Module in the Spatial-Spectral Token Learning Network, as well as the specific processing flow within each module, to facilitate understanding by those skilled in the art.

[0073] I. Token Reconstruction Module

[0074] In this invention, the Token Reconstruction Module (FGTR) reconstructs the amplitude component of the target HR-MS image through the implicit interaction between the Fourier domain tokens of the PAN image and the LR-MS image. This module generates a set of tokens that can describe the adjusted spectral features of the PAN image and the LR-MS image. The input to the first Token Reconstruction Module is the implicit features of the PAN image and the implicit features of the LR-MS image. Subsequent Token Reconstruction Modules are input to the two features output by the previous feature learning submodule. Taking the i-th Token Reconstruction Module as an example, its processing can be written as follows:

[0075]

[0076] Where FGTR stands for Token Reconstruction Module (FGTR); x i and y i This represents the input features of the i-th Token reconstruction module; This represents the output of the i-th Token reconstruction module.

[0077] In this embodiment, as Figure 3 As shown, the two features input to the Token reconstruction module are first normalized and subjected to discrete Fourier transform to obtain the Fourier domain Token for each input feature:

[0078] F x =DFT(LN(x) i ))

[0079] F y =DFT(LN(y) i ))

[0080] Among them, F x Indicates input feature x i Fourier domain token; F y Indicates input feature y i The Fourier domain Token; DFT represents Discrete Fourier Transform; LN(·) represents the normalization layer.

[0081] Each feature's Fourier domain token is approximated in complex form to facilitate implicit modeling. The specific calculation formula is as follows:

[0082]

[0083] Where, r x The real mapping representing the Fourier domain token of an LR-MS image; r y The real mapping of the Fourier domain token of the PAN image; p x The virtual mapping representing the Fourier domain token of an LR-MS image; p y The virtual mapping representing the Fourier domain token of the PAN image; Each represents the complex form of the Fourier field token for each feature. F represents x The complex form; F represents y The complex form of ; j represents the imaginary unit.

[0084] Then, the real mappings of the Fourier domain tokens of the LR-MS image and the real mappings of the Fourier domain tokens of the PAN image are concatenated to obtain the fused real mapping r. sThe fused virtual map p is obtained by connecting the virtual maps of the Fourier domain tokens of the LR-MS image and the Fourier domain tokens of the PAN image. s The calculation formula is as follows:

[0085] r s =[r x ;r y ]

[0086] p s =[p x ;p y ]

[0087] Subsequently, an implicit interaction between tokens extracted from the PAN and LR-MS images is implemented by a Fourier domain MLP layer to reconstruct the spectral characteristics required for the target HR-MS image. Specifically, the fused real and fused imaginary maps are input into a Fourier domain MLP network consisting of L cascaded Fourier domain MLP layers. The implicit feature interaction results in the Fourier domain are output and subjected to an inverse Discrete Fourier transform to obtain the output of the token reconstruction module. The implicit feature interaction results in the Fourier domain can effectively simulate the intraspectral relationships required for the target HR-MS image.

[0088] The input to the first Fourier domain MLP layer is the fused real map r. s and fusion virtual mapping p s ,Right now:

[0089] y 0 =r s +jp s

[0090] The specific processing procedure for the l-th Fourier domain MLP layer is as follows:

[0091]

[0092] According to the rules of complex number multiplication, the above expression can be expanded as follows:

[0093]

[0094] In the formula, y l The implicit feature interaction result is represented by the output of the l-th Fourier domain MLP layer; r s l-1 ,p s l-1 Let y represent the input y of the l-th Fourier domain MLP layer respectively. l-1 The real and imaginary parts; σ represents the activation function; Let each represent the complex weight matrix of the l-th Fourier domain MLP layer. The real and imaginary parts; These represent the complex biases of the l-th Fourier domain MLP layer, respectively. The real and imaginary parts.

[0095] II. Token Selection Module

[0096] In this invention, the Token Selection Module (TKTS) is used to extract tokens describing the spatial attributes of the reconstruction. It captures the most critical information for restoring the spatial characteristics required for the target HR-MS image by selecting the k tokens with the highest relevance to the query matrix of the PAN image. In this module, the features of the input PAN image are first projected into a query matrix, a key matrix, and a value matrix; then, an attention matrix is ​​calculated to describe the spatial pixel relationships between the tokens in the PAN image; subsequently, elements with lower attention in the attention matrix are masked to obtain an activation attention matrix; finally, the activation attention matrix is ​​used to weight the value matrix to obtain the output of each token selection module. The input of the first token selection module is the latent features of the PAN image, and the input of each subsequent token selection module is the feature corresponding to the PAN image from the two features output by the previous feature learning submodule. Taking the i-th token selection module as an example, its processing can be written as follows:

[0097]

[0098] Where TKTS stands for Token Selection Module (TKTS); y i This represents the input features of the i-th Token reconstruction module, which is also the input features of the i-th Token selection module; This represents the output of the i-th Token selection module.

[0099] In this embodiment, as Figure 4 As shown, the features input to the i-th Token selection module are first projected into a query matrix, a key matrix, and a value matrix. Specifically, the features y input to the Token selection module are first projected... i After layer normalization, the first normalized feature is obtained. The first normalized feature is then passed through three independent convolutional branches. The query matrix Q is output by the first convolutional branch, the key matrix K is output by the second convolutional branch, and the value matrix V is output by the third convolutional branch.

[0100] In this embodiment, each convolutional branch consists of a standard convolutional layer and a depthwise convolutional layer cascaded together. The kernel size of the standard convolutional layer is 1×1, and the kernel size of the depthwise convolutional layer is 3×3. Therefore, the specific forms of the query matrix Q, the value matrix V, and the key matrix K can be written as follows:

[0101]

[0102] in, These represent the weight matrices of the standard convolutional layer and the deep convolutional layer in the first convolutional branch, respectively. These represent the weight matrices of the standard convolutional layer and the deep convolutional layer in the second convolutional branch, respectively. These represent the weight matrices of the standard convolutional layer and the deep convolutional layer in the third convolutional branch, respectively.

[0103] Then, the attention matrix S is calculated from the query matrix Q and the reshaped key matrix to describe the spatial pixel relationship between tokens in the PAN image. The formula for calculating the attention matrix is ​​as follows:

[0104]

[0105] Where d represents the dimension of the key matrix K; T represents the matrix transpose.

[0106] Next, a masking operation is performed on the elements of the attention matrix S (here, irrelevant elements in the attention matrix S, i.e., elements with low attention), to obtain the mask matrix. Normalizing the mask matrix yields the activation and attention matrix M with a density rate k:

[0107]

[0108] in, This indicates a masking operation.

[0109] In this embodiment, the masking operation is as follows: the k-th largest element t in the f-th row of the attention matrix is ​​selected. f As a relevance threshold, if the value of the element in the f-th row and g-th column of the attention matrix is ​​greater than or equal to t f If the condition is met, the element in the f-th row and g-th column of the attention matrix is ​​retained; otherwise, the element in the f-th row and g-th column of the attention matrix is ​​reset to negative infinity.

[0110]

[0111] The process continues until each row of the attention matrix has been evaluated, resulting in a complete mask matrix.

[0112] After the masking operation above, elements in the attention matrix whose values ​​are less than the relevance threshold of the corresponding row are assigned negative infinity. These values ​​are then normalized to approximately zero. This process preserves the portion that contributes most to the self-attention score while removing irrelevant information, resulting in the activation attention matrix. Finally, the activation attention matrix M is multiplied by the value matrix V to obtain the output of the i-th token selection module.

[0113]

[0114] III. Adaptive Token Interaction Module

[0115] The adaptive token interaction module utilizes the activation attention matrix obtained from the token selection module (TKTS) to dynamically modulate the channels of the spectral features extracted from the token reconstruction module (FGTR), aggregating the features generated by the FGTR and the token selection module, thereby fusing spatial and spectral features to achieve spatial location-aware adaptive fusion of spatial-spectral characteristics. In the adaptive token interaction module, each channel feature map extracted from the token reconstruction module is treated as a token, and they are projected to obtain query projection, key projection, and value projection. The key projection is reshaped and a dot product operation is performed with the query projection to obtain the attention map. Subsequently, the activation attention matrix obtained from the token selection module (TKTS) is used to weight the value projection. Finally, the final output result is obtained through matrix reshaping. Taking the i-th adaptive token interaction module as an example, its processing can be written as follows:

[0116] o i =ATI(p i ,q i )

[0117] Where ATI represents the Adaptive Token Interaction Module; o i p represents the output of the i-th adaptive token interaction module; i ,q i This represents the input of the i-th adaptive Token interaction module.

[0118] For the adaptive token interaction module in the first feature fusion submodule, its input is a set of tokens describing the reconstructed spatial attributes and a set of tokens describing the reconstructed spectral feature attributes, output by the fourth feature learning submodule; for the adaptive token interaction modules in the remaining feature fusion submodules, their input is the feature output by the previous feature fusion submodule.

[0119] In this embodiment, as Figure 5 As shown, taking the adaptive Token interaction module in the first feature fusion submodule as an example, it uses a set of tokens (i.e., p) describing the reconstructed spectral feature attributes. 1 Each channel feature map is treated as a token. After layer normalization, a second normalized feature is obtained. The second normalized feature is then passed through three new convolutional branches, and the query projection is output by the first new convolutional branch. The output key is projected from the second new convolution branch. Projected by the output value of the third new convolution branch

[0120]

[0121] in, These represent the weight matrices of the standard convolutional layer and the deep convolutional layer in the first new convolutional branch, respectively. These represent the weight matrices of the standard convolutional layer and the deep convolutional layer in the second new convolutional branch, respectively. These represent the weight matrices of the standard convolutional layer and the deep convolutional layer in the third convolutional branch, respectively.

[0122] By query projection and the key projection after shape reshaping Perform a dot product operation to calculate attention map A:

[0123]

[0124] in, These are the learnable parameters for adaptive attention maps.

[0125] Project a set of tokens and values ​​describing the properties of the reconstructed space. Multiplying yields a weighted importance matrix. This weighted importance matrix, after reshaping, is then dot-productted with the normalized attention map to obtain a weighted attention map. Reshaping this weighted attention map yields the output of the adaptive Token interaction module in the first feature fusion submodule. 1 :

[0126]

[0127] For the second, third, and fourth feature fusion submodules, the eighth, tenth, and twelfth intermediate features replace the set of tokens describing the reconstructed spectral feature attributes input in the first feature fusion submodule. After layer normalization and convolutional branching, the query projection, value projection, and key projection of the second, third, and fourth feature fusion submodules are obtained respectively. The seventh, ninth, and eleventh intermediate features replace the set of tokens describing the reconstructed spatial attributes input in the first feature fusion submodule. They are used to multiply with the value projections generated by their respective feature fusion submodules to obtain the weighted importance matrix of the corresponding feature fusion submodule. The specific process will not be elaborated further.

[0128] It should be noted that the aforementioned spatial-spectral token learning network also includes a downsampling module and an upsampling module. The specific structures of these modules are not limited in this invention, as long as they can cooperate with the spatial-spectral token learning network of this invention and implement the corresponding downsampling and upsampling processes. In this embodiment, the downsampling module is implemented through a 4×4 convolutional layer with stride, which downsamples the feature map while doubling the number of channels. The upsampling module is implemented through a 2×2 deconvolutional layer with stride, which upsamples the feature map while halving the number of channels.

[0129] It should also be noted that, during the training of the spatial-spectral token learning network of this invention, a reconstruction loss is introduced into the total loss. This enables the spatial-spectral token learning network to generate HR-MS images similar to the corresponding ground truth values. An auxiliary loss is also introduced. To help the Token Reconstruction Module (FGTR) correct for correlations within the spectrum according to true values, the total loss function used is... Specifically:

[0130]

[0131] Where τ is the balance factor; Indicates the losses incurred during reconstruction; Indicates auxiliary loss; X out It is the HR-MS image output by the spatial-spectral token learning network, X gt It is the ground truth of the HR-MS image; A out It is the amplitude spectrum of the features output by the Token reconstruction module in the last feature learning submodule; A gt The amplitude spectrum is obtained by performing a fast Fourier transform on the true value of the HR-MS image.

[0132] The present invention will now demonstrate the application effect of the panchromatic sharpening method based on the spatial-spectral token learning network described in the above embodiments on a specific dataset through a specific example, so as to facilitate understanding of the essence of the present invention.

[0133] Example

[0134] The full-color sharpening method based on the spatial-spectral token learning network used in this embodiment is implemented as described above and will not be repeated here.

[0135] The overall process in this embodiment can be divided into three stages: data preprocessing, model training, and HR-MS image generation. Figure 6 As shown.

[0136] 1. Data Preprocessing Stage

[0137] This embodiment conducted extensive experiments on three image sets collected by QuickBird, GaoFen-2, and WorldView-3 satellites. For comparison with other methods, this embodiment performed experiments under the Wald protocol and full-scale settings. According to the Wald protocol, both PAN and MS images were downsampled to a lower scale so that the original MS images could be used as ground truth images. This embodiment used a Gaussian filter to blur the original PAN / LR-MS / ground truth image pairs, and then downsampled them with a scaling factor of 4. Since the remote sensing images are very large, it is difficult to input them into a spatial-spectral token learning network; therefore, this embodiment cropped these images into small patches to form the training and test sets. On the QuickBird dataset, this embodiment cropped the test images into 356 patches to construct the test set, while the training set consisted of 12,119 patches randomly cropped from other images. Then, this embodiment performed the same processing on the WorldView-3 dataset, which contained 3,639 test patches and 11,856 training patches. On the GaoFen-2 dataset, 35,725 training blocks and 3,370 test blocks were generated. During training, this embodiment provides the model with PAN images of size 128×128×1 and LR-MS images of size 32×32×4. During testing, the sizes of the PAN and LR-MS images are 128×128×1 and 32×32×4, respectively. Furthermore, 70% of the dataset is allocated to the training set and 30% to the test set.

[0138] 2. Model Training

[0139] During the training phase, the Adam optimizer with β1 = 0.9 and β2 = 0.999 was used for optimization, with a batch size of 4 and 30,000 iterations. The initial learning rate was set to 1 × 10⁻⁶. -4 Furthermore, the performance decays by 0.85 after every 3000 iterations. All experiments were conducted on the PyTorch framework using four NVIDIA RTX A6000 GPUs.

[0140] 3. HR-MS image generation

[0141] The images from the test set are directly used as input, and through the trained Spatial-Spectral Token Learning Network (SSTL), an HR-MS image with rich spatial and spectral features is finally generated.

[0142] In the low-resolution experiment of this invention embodiment, the test results of the WorldView-3 case are as follows: Figure 7 As shown, the test results for the QuickBrid case are as follows: Figure 8 As shown, the test results of GaoFen-2 are as follows: Figure 9 As shown.

[0143] Analysis of the WorldView-3 case reveals that deep learning-based methods achieve higher fusion accuracy than traditional methods, as evidenced by the lower brightness of their error maps. Among deep learning-based methods, the method presented in this invention, SSTL, exhibits the smallest difference between itself and the real image, demonstrating the best fusion performance, followed by LGTEUN and MDCUN.

[0144] Analysis of the QuickBrid case reveals that, among deep learning-based methods, MSDCNN, DCFNet, and ARFNet exhibit poor reconstruction performance, with high brightness values ​​in their error maps, even comparable to traditional methods. Furthermore, the SSTL method of this invention demonstrates a significant advantage in preserving spectral information, resulting in reconstructions with minimal visual difference in color distribution compared to the real image.

[0145] Analysis of the case study GaoFen-2 reveals that the SSTL method of this invention exhibits superior fusion performance. Furthermore, it was found that, apart from ARFNet, the reconstruction performance of deep learning-based methods is better than that of traditional methods.

[0146] To evaluate the model's performance and generalization ability in real-world scenarios, this embodiment further conducted experiments on some full-resolution examples. Specifically, the pre-trained model obtained from down-resolution data was directly applied to some full-resolution samples. The test results for the WorldView-3 case in the full-resolution experiments are as follows: Figure 10 As shown, the test results for the QuickBrid case are as follows: Figure 11 As shown, the test results of GaoFen-2 are as follows: Figure 12 As shown. Figure 10 , Figure 11 and Figure 12 Visual comparisons of several methods are presented on representative full-resolution test cases. It is readily apparent from these figures that the proposed spatial-spectral token learning network exhibits superior spatial texture while maintaining pleasing spectral properties. In contrast, other competing methods either contain blurry spatial textures and distorted edges or display significant spectral distortion.

[0147] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A panchromatic sharpening method based on a spatial-spectral token learning network, characterized in that, Includes the following steps: A set of PAN and LR-MS images to be panchromatic sharpening are input into a trained spatial-spectral token learning network. The PAN and LR-MS images are each extracted using the first convolutional layer. The extracted latent features of the PAN and LR-MS images are input into the feature learning module. The set of tokens describing the reconstructed spectral features and the set of tokens describing the reconstructed spatial features output by the feature learning module are input into the feature fusion module to obtain aggregated features that fuse spatial and spectral characteristics. After the aggregated features are processed by the second convolutional layer, the target HR-MS image is obtained. The spatial-spectral token learning network has a U-shaped structure. The feature learning module includes four feature learning sub-modules. The first three feature learning sub-modules each include a token reconstruction module, a token selection module, and a downsampling module. The last feature learning sub-module only includes a token reconstruction module and a token selection module. The feature fusion module includes four feature fusion sub-modules. The first three feature fusion sub-modules each include an adaptive token interaction module and an upsampling module. The last feature fusion sub-module only includes an adaptive token interaction module. Furthermore, the first three feature fusion sub-modules perform feature fusion through skip connections before entering the next adjacent feature fusion sub-module to reduce the information loss caused by the downsampling operation. In the adaptive token interaction module, each channel feature map of the features extracted from the token reconstruction module is regarded as a token, and they are projected to obtain the query projection, key projection, and value projection; the key projection is reshaped and a dot product operation is performed with the query projection to obtain the attention map. Subsequently, the activation attention matrix obtained from the Token selection module is used for weighted projection; finally, the final output result is obtained through matrix reshaping operation. The specific processing flow in the Token reconstruction module is as follows: First, the two features input to the Token reconstruction module are normalized and subjected to Discrete Fourier Transform to obtain the Fourier domain Token for each input feature. Then, the Fourier domain Token for each feature is approximated in complex form to facilitate implicit modeling. The specific calculation formula is as follows: ； ； in, A real mapping representing the Fourier domain token of an LR-MS image; The real mapping of the Fourier domain token of the PAN image; The virtual mapping representing the Fourier domain token of an LR-MS image; The virtual mapping representing the Fourier domain token of the PAN image; Each represents the complex form of the Fourier domain token for each feature; Represents the imaginary unit; Then, the real mapping of the Fourier domain token of the LR-MS image and the real mapping of the Fourier domain token of the PAN image are connected to obtain the fused real mapping. The virtual mapping of the Fourier domain token of the LR-MS image and the virtual mapping of the Fourier domain token of the PAN image are connected to obtain the fused virtual mapping. The fused real mapping and the fused virtual mapping are input into a Fourier domain MLP network composed of L cascaded Fourier domain MLP layers. The implicit feature interaction result of the Fourier domain is output and subjected to discrete inverse Fourier transform operation to obtain the output of the token reconstruction module.

2. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 1, characterized in that, The specific processing flow in the spatial-spectral token learning network is as follows: After the PAN image passes through the first convolutional layer, the latent features of the PAN image are obtained. After the LR-MS image passes through the first convolutional layer, the latent features of the LR-MS image are obtained. The two latent features are used as inputs to the first feature learning submodule. The token selection module in the first feature learning submodule processes the latent features of the PAN image to obtain the first intermediate feature. Then, the latent features of the PAN image and the latent features of the LR-MS image are input together into the token reconstruction module in the first feature learning submodule to obtain the second intermediate feature. The first and second intermediate features are each downsampled and used as input to the second feature learning submodule. The token selection module in the second feature learning submodule processes the downsampled first intermediate feature to obtain the third intermediate feature. The downsampled first and second intermediate features are then input together into the token reconstruction module in the second feature learning submodule to obtain the fourth intermediate feature. The third and fourth intermediate features are each downsampled and used as input to the third feature learning submodule. The token selection module in the third feature learning submodule processes the downsampled third intermediate feature to obtain the fifth intermediate feature. The downsampled third and fourth intermediate features are then input together into the token reconstruction module in the third feature learning submodule to obtain the sixth intermediate feature. The fifth and sixth intermediate features are downsampled and used as input to the fourth feature learning submodule. The Token selection module in the fourth feature learning submodule processes the downsampled fifth intermediate feature to obtain a set of tokens describing the reconstructed spatial attributes. Then, the downsampled fifth and sixth intermediate features are input together into the Token reconstruction module in the fourth feature learning submodule to obtain a set of tokens describing the reconstructed spectral feature attributes. A set of tokens describing the reconstructed spectral features and a set of tokens describing the reconstructed spatial features are input together into the adaptive token interaction module in the first feature fusion submodule to obtain the first interactive feature. The upsampled first interactive feature is concatenated with the fifth intermediate feature to obtain the first concatenated feature. After the first concatenated feature is processed by the third convolutional layer, the seventh intermediate feature is obtained. The upsampled first interactive feature is concatenated with the sixth intermediate feature to obtain the second concatenated feature. After the second concatenated feature is processed by the third convolutional layer, the eighth intermediate feature is obtained. The seventh and eighth intermediate features are input together into the adaptive Token interaction module in the second feature fusion submodule to obtain the second interaction feature. The upsampled second interaction feature is concatenated with the third intermediate feature to obtain the third concatenated feature. The third concatenated feature is processed by the fourth convolutional layer to obtain the ninth intermediate feature. The upsampled second interaction feature is concatenated with the fourth intermediate feature to obtain the fourth concatenated feature. The fourth concatenated feature is processed by the fourth convolutional layer to obtain the tenth intermediate feature. The ninth and tenth intermediate features are input together into the adaptive Token interaction module in the third feature fusion submodule to obtain the third interaction feature. The upsampled third interaction feature is concatenated with the first intermediate feature to obtain the fifth concatenated feature. The fifth concatenated feature is processed by the fifth convolutional layer to obtain the eleventh intermediate feature. The upsampled third interaction feature is concatenated with the second intermediate feature to obtain the sixth concatenated feature. The sixth concatenated feature is processed by the fifth convolutional layer to obtain the twelfth intermediate feature. The eleventh and twelfth intermediate features are input together into the adaptive Token interaction module in the fourth feature fusion submodule to obtain the fourth interaction feature, which is used as the aggregated feature to fuse spatial and spectral characteristics. Finally, the aggregated feature is processed by the second convolutional layer to obtain the target HR-MS image.

3. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 1, characterized in that, No. The specific processing procedure for each Fourier domain MLP layer is as follows: ； In the formula, Indicates the first The implicit feature interaction results output by each Fourier domain MLP layer; They represent the first The real and imaginary parts of the input to a Fourier domain MLP layer; Indicates the activation function; They represent the first The real and imaginary parts of the complex weight matrix of a Fourier domain MLP layer; They represent the first The real and imaginary parts of the complex bias of a Fourier domain MLP layer.

4. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 2, characterized in that, The specific processing flow in the Token selection module is as follows: First, the features input to the Token selection module are normalized to obtain the first normalized features. The first normalized features are then passed through three convolutional branches. The query matrix is ​​output from the first convolutional branch, and the key matrix is ​​output from the second convolutional branch. The output value matrix is ​​generated by the third convolution branch. From the query matrix Calculate the attention matrix using the reshaped bond matrix. : ； in, Indicates the dimension of the key matrix; Indicates matrix transpose; Next, the elements in the attention matrix are masked to obtain a mask matrix. The mask matrix is ​​then normalized to obtain an activation attention matrix with a density ratio. Finally, the activation attention matrix is ​​multiplied by the value matrix to obtain the output of the token selection module.

5. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 4, characterized in that, In the Token selection module, the specific process of the masking operation is as follows: The attention matrix is... The k-th largest element value in the row As a relevance threshold, if the attention matrix of the first... Line number Column element values ​​are greater than or equal to Then the attention matrix will be located at the th Line number The element values ​​of the column are retained; otherwise, the attention matrix is ​​placed in the [position missing]. Line number The values ​​of the column elements are reset to negative infinity until each row of the attention matrix has been evaluated, resulting in a complete mask matrix.

6. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 2, characterized in that, In the first feature fusion submodule, the specific processing flow of the adaptive token interaction module is as follows: A set of tokens describing the reconstructed spectral feature attributes are normalized through a layer to obtain the second normalized feature. The second normalized feature is then passed through three new convolutional branches, and the query projection is output by the first new convolutional branch. The query projection is obtained by projecting the output value of the third new convolutional branch onto the key projection after reshaping the output of the second new convolutional branch. The attention map is then calculated by performing a dot product operation between the query projection and the reshaped key projection. : ； in, The result of shape reshaping after key projection; These are learnable parameters for adaptive attention maps; This represents the dot product operation; A set of tokens describing the reconstructed spatial attributes are multiplied by the value projection to obtain a weighted importance matrix. After reshaping the weighted importance matrix, a dot product operation is performed with the normalized attention map to obtain a weighted attention map. After reshaping the weighted attention map, the output of the adaptive token interaction module in the first feature fusion submodule is obtained.

7. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 6, characterized in that, In the second, third, and fourth feature fusion sub-modules, the eighth, tenth, and twelfth intermediate features are used for layer normalization and convolutional branching, while the seventh, ninth, and eleventh intermediate features are multiplied by the projection of the values ​​generated by their respective feature fusion sub-modules to obtain the weighted importance matrix of the corresponding feature fusion sub-module.

8. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 4 or 6, characterized in that, Each convolutional branch consists of a standard convolutional layer and a depthwise convolutional layer cascaded together. The kernel size of the standard convolutional layer is 1×1, and the kernel size of the depthwise convolutional layer is 3×3.

9. The panchromatic sharpening method based on a spatial-spectral token learning network as described in claim 1, characterized in that, The loss function used during training of the spatial-spectral token learning network is as follows: ； ； ； in, It is a balancing factor; Indicates the losses incurred during reconstruction; Indicates auxiliary loss; It is an HR-MS image output by a spatial-spectral token learning network. It is the ground truth value of the HR-MS image; It is the amplitude spectrum of the features output by the Token reconstruction module in the last feature learning submodule; The amplitude spectrum is obtained by performing a fast Fourier transform on the true value of the HR-MS image.

Images