Pan-sharpening method and system based on high-frequency differential spatial attention mechanism

Abstract

The present invention relates to a panchromatic sharpening method and system based on a high-frequency differential spatial attention mechanism, including: modeling a low-resolution multispectral (LRMS) image as a spatially degraded version of a high-resolution multispectral (HRMS) image to obtain a spectral fidelity term; then, through an attention injection module, generating an attention injection weight map, inputting the panchromatic (PAN) image and its low-pass filtered image into a differential attention module respectively to calculate a high-frequency differential attention map; then, constructing a high-frequency differential spatial fidelity term according to the attention injection weight map and the high-frequency differential spatial attention map, introducing the logarithmic nuclear norm of the high-resolution multispectral (HRMS) image, combining the spectral fidelity term with the high-frequency differential spatial attention spatial fidelity term and a tensor low-rank constraint term, building a mathematical model of the panchromatic sharpening algorithm; and finally, using the alternating direction method of multipliers to optimize and solve the mathematical model, and finally outputting the result of the high-resolution multispectral (HRMS) image.

Description

Technical Field

[0001] The present invention relates to a panchromatic sharpening method and system based on a high-frequency differential spatial attention mechanism, and belongs to the technical field of panchromatic sharpening. Background Art

[0002] Panchromatic sharpening technology is an image processing method that fuses a pair of high-spatial-resolution panchromatic PAN images and low-spatial-resolution but multi-band low-resolution multispectral LRMS images to generate high-resolution multispectral HRMS images with high spatial resolution and maintaining multispectral spectral information. This technology stems from the continuous progress of remote sensing satellites and earth observation technologies, and a large number of high-spatial-resolution panchromatic PAN images and low-spatial-resolution but multi-band low-resolution multispectral LRMS images have emerged. However, relying solely on multispectral images cannot obtain clear detail and texture features, while panchromatic images, although having a high spatial resolution, lack spectral dimension information. Therefore, panchromatic sharpening technology has emerged, which is of great significance in fields such as agricultural monitoring, ecological assessment, urban planning, disaster prevention and mitigation, etc., and can discover the distribution and changes of ground objects at a finer scale and improve the comprehensive utilization value of earth observation data.

[0003] At present, the research methods of pan-sharpening are mainly divided into the following categories: methods based on component substitution (ComponentSubstitution), methods based on multi-resolution analysis (Multi-Resolution Analysis), methods based on variational optimization (Variational / Optimization), and methods based on deep learning (Deep Learning). Typical component substitution methods include IHS, PCA, Brovey, Gram–Schmidt, etc. These methods mainly project the LRMS image into a component space such as a certain luminance or principal component, then use the PAN image to replace or inject this main component, and then inverse-transform back to the multi-spectral domain to generate a high-resolution multi-spectral (HRMS) image. Such methods are simple to implement and have a relatively fast operation speed. To a certain extent, they can improve the clarity of the image. However, due to the insufficient fidelity of spectral information during the substitution process, they are prone to serious spectral distortion and cannot capture the complex relationships between multi-spectral channels at a deeper level. Methods based on multi-resolution analysis include wavelet transform, Contourlet, Laplacian Pyramid, etc. These methods mainly perform multi-scale or multi-resolution decomposition on the PAN image and the LRMS image respectively, then fuse the high-frequency components at the corresponding levels, and finally reconstruct to obtain a high-resolution multi-spectral (HRMS) image. Such methods can better inject high-frequency details to solve the spectral distortion problem, but are prone to fusion artifacts in scenes with strong noise or large-scale differences and have high requirements for filter design. Methods based on variational optimization mainly focus on mathematically modeling a more appropriate and effective relationship between the HRMS image and the LRMS image, between the HRMS image and the PAN image, and between the spatial and spectral prior constraints of the HRMS image. They incorporate spectral fidelity, smooth prior, spatial details, etc. into a unified optimization framework, and at the same time use regularization to constrain the fusion process to obtain a high-resolution multi-spectral (HRMS) image. With the success of deep neural networks in tasks such as image super-resolution and style transfer, pan-sharpening methods have obtained new development. In the existing technology, deep neural networks such as CNN, GAN, and Transformer are applied to the pan-sharpening task to learn the mapping relationship between multi-band and high-frequency features in an end-to-end manner. Methods based on deep learning can produce very promising fusion results, but often require a large amount of labeled training data, have high requirements for network structure and training strategies, and have low generalization performance. In summary, there is still a large room for development in the existing pan-sharpening technologies. Summary of the Invention

[0004] The object of the present invention is to overcome the deficiencies in the prior art and provide a panchromatic sharpening method based on a high-frequency differential spatial attention mechanism. By obtaining image data from a remote sensing satellite dataset, the image data includes a panchromatic (PAN) image and a low-resolution multispectral (LRMS) image, and preprocessing operations such as pixel-level registration and geometric correction are performed on the images to obtain a perfectly matched PAN image and LRMS image; the low-resolution multispectral (LRMS) image is modeled as a spatially degraded version of the high-resolution multispectral (HRMS) image, and a spectral fidelity constraint is established; the LRMS image is upsampled to obtain an upsampled version of the LRMS image, and the upsampled version of the LRMS image, the PAN image, and the low-pass filtered image of the PAN image are concatenated along the channel dimension, and then through an attention injection module, a pixel-level attention injection weight map is obtained; then the PAN image and the low-pass filtered image of the PAN image are used to calculate the high-frequency differential spatial attention , and a high-frequency differential spatial attention map is obtained , and a high-frequency differential spatial fidelity constraint is established; according to the low tube rank characteristic of the HRMS image, the logarithmic tensor nuclear norm is introduced to obtain a tensor low tube rank constraint term; further, the high-frequency differential spatial attention spatial fidelity term, the spectral fidelity term, and the tensor low tube rank constraint term are combined to construct a unified mathematical optimization model, and the model is iteratively solved by the alternating direction method of multipliers (ADMM), continuously correcting and enhancing edges and texture details, and finally outputting a high-resolution multispectral (HRMS) image. It realizes not only pixel-level spatial fidelity, but also aligns the spatial structures of the PAN image and the LRMS image in the feature domain constructed by the high-frequency differential spatial attention, enabling the model to more accurately extract real high-frequency spatial details, reducing the influence of noise, solving spectral distortion while improving the data generalization performance, and obtaining a high-resolution multispectral (HRMS) image with higher spatial resolution and maintaining multispectral spectral information.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] In a first aspect, a panchromatic sharpening method based on a high-frequency differential spatial attention mechanism provided by the present invention includes the following steps:

[0007] Step 1: According to the high-resolution multispectral (HRMS) image, model the low-resolution multispectral (LRMS) image as a spatially degraded version of the high-resolution multispectral (HRMS) image to obtain a spectral fidelity term;

[0008] Step 2: Upsample the low-resolution multispectral (LRMS) image to obtain an upsampled version of the LRMS image, concatenate the upsampled version of the LRMS image, the panchromatic (PAN) image, and the low-pass filtered image of the panchromatic (PAN) image along the channel dimension, and then through an attention injection module, obtain a pixel-level attention injection weight map;

[0009] Step 3: Input the panchromatic PAN image and the low-pass filtered image of the panchromatic PAN image into the differential attention module respectively to calculate the high-frequency differential spatial attention matrix, and reshape the high-frequency differential spatial attention matrix to obtain the high-frequency differential spatial attention map;

[0010] Step 4: Obtain the high-frequency differential spatial attention space fidelity term according to the attention injection weight map and the high-frequency differential spatial attention map;

[0011] Step 5: According to the low tube-rank characteristic of the high-resolution multispectral HRMS image, introduce the logarithmic tensor nuclear norm to obtain the tensor low tube-rank constraint term;

[0012] Step 6: Combine the spectral fidelity term, the high-frequency differential spatial attention space fidelity term and the tensor low tube-rank constraint term to build a mathematical model of the high-frequency differential spatial attention mechanism panchromatic sharpening algorithm;

[0013] Step 7: Use the alternating direction method of multipliers to optimize and solve the mathematical model, and the final result obtained is the high-resolution multispectral HRMS image.

[0014] Furthermore, the HRMS image is expressed as , the LRMS image is expressed as , the PAN image is expressed as , where represents the set of real numbers, and are the height and width respectively, is the number of bands of the multispectral image, is the spatial resolution ratio;

[0015] In Step 1, the LRMS image is modeled as the spatial degradation version of the HRMS image ,

[0016] that is , where represents the spatial downsampling matrix, represents the Gaussian blur matrix,

[0017] represents zero-mean Gaussian noise, and the obtained spectral fidelity term is:

[0018] ,

[0019] where represents the square of the norm of the matrix.

[0020] Furthermore, the specific steps of Step 2 are as follows:

[0021] Step 2-1: LRMS image Perform upsampling operation to obtain the upsampled version LRMS image ,

[0022] Step 2-2: PAN image The low-pass filtered image of the PAN image is obtained by low-pass filtering. The low-pass filter is connected in series with two The convolutional layer is learned by introducing the ReLU activation function in the middle, as follows:

[0023] ,

[0024] in, express Convolutional layer, ReLU represents the activation function;

[0025] Step 2-3: Upsample the LRMS image , PAN image And the low-pass filtered image of the PAN image Cascade along the channel dimension to get a tensor ;

[0026] Step 2-4: Tensor Through a The convolutional layer performs preliminary feature extraction and obtains the corresponding feature map. The convolutional feature map is sent to the ReLU activation function and then passes through another The convolutional layer integrates the features activated by the ReLU activation function to obtain the attention injection weight map ,as follows:

[0027] .

[0028] Furthermore, the specific steps of step 3 are as follows:

[0029] Step 3-1: PAN image And low-pass filtered image of PAN image Input into the differential attention module to calculate its attention matrix and get the PAN image The attention matrix , low-pass filtered image of PAN image The attention matrix ,in, for:

[0030] ,

[0031] in, , Represents a query of the network model, , represents the key of the network model, , represents the transposed matrix of the key of the network model, represents the value of the network model, represents a learnable scalar, represents the dimension of each branch,

[0032] Step 3-2: According to the obtained attention matrix and , calculate the high-frequency differential spatial attention , ,

[0033] Step 3-3: Take the average of each row of the high-frequency differential spatial attention to calculate the average attention difference and obtain a -dimensional vector as follows:

[0034] , ,

[0035] where, represents the dimension of , represents the th row and the th column attention difference value, represents the average attention difference of the th row,

[0036] Step 3-4: Reshape the vector back to the patch grid, represents the reshaped matrix, and each element corresponds to the attention difference of a certain patch in the image;

[0037] Step 3-5: Upsample the reshaped matrix back to the original image size to obtain the final high-frequency differential spatial attention map .

[0038] Furthermore, in step 4, according to the attention injection weight map and the high-frequency differential spatial attention map , obtain the high-frequency differential spatial attention spatial fidelity term, expressed as:

[0039] ,

[0040] where, Indicates element multiplication.

[0041] Furthermore, in step 5, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term:

[0042] ,

[0043] where, represents the logarithmic tensor nuclear norm.

[0044] Furthermore, in step 6, the spectral fidelity term is combined with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism pan-sharpening algorithm, and the mathematical model is formulated as:

[0045]

[0046] ,

[0047] ,

[0048] where, and are two balancing parameters; the tensor auxiliary variable is introduced, and the mathematical optimization model is re-expressed as:

[0049]

[0050] ,

[0051] ,

[0052] where, is the tensor auxiliary variable, which is used to assist in iteratively reconstructing the HRMS image .

[0053] Furthermore, in step 7, the Alternating Direction Method of Multipliers (ADMM) is used to optimize and solve the mathematical optimization model. The specific steps are as follows:

[0054] The constrained optimization problem is transformed into an unconstrained optimization problem using the augmented Lagrangian function, and the derivation of the augmented Lagrangian function is as follows:

[0055]

[0056] ,

[0057] where, is the Lagrange multiplier, represents the tensor inner product, is the penalty parameter; through initialization, , , Iteratively optimize the model alternately until the set termination condition is met. The finally obtained result is the HRMS image of this panchromatic sharpening task; update the augmented Lagrangian function alternately and iteratively , that is:

[0058] .

[0059] Furthermore, iteratively update :

[0060] ,

[0061] wherein, represents the -th iteration, represents solving the sub-problem in the -th iteration. The sub-problem is deduced as:

[0062]

[0063] ,

[0064] Iteratively update :

[0065] ,

[0066] represents solving the sub-problem in the -th iteration. The sub-problem is deduced as:

[0067] ,

[0068] Iteratively update the Lagrange multiplier . According to the standard practice of the augmented Lagrangian method, the multiplier is updated by gradient ascent based on the current degree of constraint violation represents solving the sub-problem in the -th iteration. The sub-problem is deduced as:

[0069] .

[0070] In a second aspect, the present invention provides a panchromatic sharpening system based on a high-frequency difference spatial attention mechanism, including:

[0071] Image acquisition module: It is used to obtain image data from remote sensing satellite datasets. The image data includes panchromatic (PAN) images and low-resolution multispectral (LRMS) images, and perform preprocessing operations such as pixel-level registration and geometric correction on the images to ensure their consistency in the sampling area and acquisition time, and obtain perfectly matched PAN images and LRMS images , and use the processed PAN images and LRMS images as input data for subsequent models and high-frequency differential spatial attention calculation;

[0072] Spectral modeling module: It is used to model spectral fidelity between multispectral observations according to the spectral degradation relationship of HRMS images to obtain a spectral fidelity term;

[0073] Attention injection module: It is used to cascade the upsampled version of the LRMS image , PAN images and the low-pass filtered image of the PAN image along the channel dimension, and then obtain a pixel-level attention injection weight map through attention injection ;

[0074] High-frequency differential spatial attention module: It is used to input the PAN image and its low-pass filtered image into the differential attention module, calculate their attention matrices respectively, calculate the high-frequency differential spatial attention according to the attention matrices , reshape the high-frequency differential spatial attention to obtain a high-frequency differential spatial attention map , combine the attention injection weight map and the high-frequency differential spatial attention map to obtain a high-frequency differential spatial attention spatial fidelity term;

[0075] Mathematical modeling module: It is used to combine the spectral fidelity term, the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build an overall mathematical optimization model and obtain the final mathematical optimization model;

[0076] ADMM optimization output module: It is used to solve the mathematical optimization model by the alternating direction method of multipliers (ADMM), and update respectively in each iteration , , , until the set termination conditions are met; the finally output result is the high-resolution multispectral (HRMS) image obtained by the panchromatic sharpening of this system.

[0077] Compared with the prior art, the present invention adopts the above technical solutions and has the following beneficial effects:

[0078] 1. The present invention proposes a pan-sharpening method based on a high-frequency differential spatial attention mechanism, which can better retain high-resolution details during the fusion process. By injecting attention, the high-frequency differential spatial attention of the panchromatic image extracted is combined with the low-resolution multispectral image, which can capture the dependencies between pixels at a long distance in the image, and can further highlight key texture edges and spatial details on the basis of the global context, enabling the sharpened image to better take into account local detail features. At the same time, the differential operation of the two-way spatial attention can eliminate common noise to a certain extent, making the focus concentrate on the real high-frequency change region, retaining more accurate spectral information and spatial information, and then outputting a high-resolution multispectral image with higher quality;

[0079] 2. The present invention combines the traditional variational optimization method with the proposed high-frequency differential spatial attention mechanism to construct a general pan-sharpening optimization model. On the one hand, it utilizes the good interpretability and strong data generalization ability of the variational optimization method, and on the other hand, it gives full play to the advantages of the differential spatial attention mechanism in global modeling and attention injection, and organically combines the advantages of both, so as to achieve more excellent results in the pan-sharpening task. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a schematic diagram of the overall process of the pan-sharpening method of the present invention,

[0081] Figure 2 It is a schematic diagram of the structure of an attention injection module provided by the present invention,

[0082] Figure 3 It is a schematic diagram of the structure of a differential attention module provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0083] In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments.

[0084] Example 1: Refer to Figure 1, A pan-sharpening method based on a high-frequency differential spatial attention mechanism, which obtains image data from remote sensing satellite datasets Worldview-3 and GaoFen2. The image data includes a panchromatic (PAN) image and a low-resolution multispectral (LRMS) image. By combining with the high-resolution multispectral (HRMS) image to model the spectral degradation of the LRMS image, a spectral fidelity constraint is established to ensure the accuracy of the fusion result in the spectral domain. The low-resolution multispectral (LRMS) image is upsampled to obtain an upsampled version of the LRMS image. Along the channel dimension, the upsampled version of the LRMS image, the panchromatic (PAN) image, and the low-pass filtered image of the panchromatic (PAN) image are concatenated, and then through the attention injection module, a pixel-level attention injection weight map is obtained. Then, the panchromatic (PAN) image and the low-pass filtered image of the panchromatic (PAN) image pass through the differential attention module to calculate the high-frequency differential spatial attention. , The high-frequency differential spatial attention is reshaped to obtain a high-frequency differential spatial attention map. , A high-frequency differential spatial attention fidelity constraint is established. On this basis, according to the low tube-rank characteristic of the HRMS image, the logarithmic nuclear norm is introduced to establish a tensor low tube-rank constraint. Further, the high-frequency differential spatial attention spatial fidelity term is combined with the spectral fidelity term and the tensor low tube-rank constraint term to construct a unified mathematical optimization model, and the alternating direction method of multipliers (ADMM) is used for iterative solution to continuously correct and enhance edges and texture details, and finally a high-resolution multispectral (HRMS) image is output.

[0085] Figure 1 is a flow schematic diagram of a pan-sharpening method based on a high-frequency differential spatial attention mechanism of the present invention. First, the low-resolution multispectral (LRMS) image is modeled as a spatially degraded version of the high-resolution multispectral (HRMS) image to obtain a spectral fidelity term. Subsequently, along the channel dimension, the upsampled version of the LRMS image, the panchromatic (PAN) image, and the low-pass filtered image of the panchromatic (PAN) image are concatenated, and through the attention injection module, a pixel-level attention injection weight map is generated. At the same time, the panchromatic (PAN) image and the low-pass filtered image of the panchromatic (PAN) image are respectively input into the differential attention module to calculate the high-frequency differential attention map. Then, according to the attention injection weight map and the high-frequency differential spatial attention map, a high-frequency differential spatial attention spatial fidelity term is constructed. According to the low tube-rank characteristic of the HRMS image, the logarithmic nuclear norm is introduced to strengthen the tensor low tube-rank constraint. Then, the spectral fidelity term is combined with the high-frequency differential spatial attention spatial fidelity term and the tensor low tube-rank constraint term to build a mathematical model of the pan-sharpening algorithm based on the high-frequency differential spatial attention mechanism. Finally, the alternating direction method of multipliers is used to optimize and solve the mathematical model, and finally the result of a high-resolution multispectral (HRMS) image is output.

[0086] A pan-sharpening method based on a high-frequency differential spatial attention mechanism according to the present invention includes the following content, specifically as follows:

[0087] Step 1: According to the high-resolution multispectral (HRMS) image, model the low-resolution multispectral (LRMS) image as a spatially degraded version of the high-resolution multispectral (HRMS) image to obtain a spectral fidelity term;

[0088] Step 2: Upsample the low-resolution multispectral (LRMS) image to obtain an upsampled version of the LRMS image. Concatenate the upsampled version of the LRMS image, the panchromatic (PAN) image, and the low-pass filtered image of the panchromatic (PAN) image along the channel dimension, and then obtain a pixel-level attention injection weight map through the attention injection module;

[0089] Step 3: Input the panchromatic (PAN) image and the low-pass filtered image of the panchromatic (PAN) image into the differential attention module respectively to calculate the high-frequency differential spatial attention matrix, and reshape the high-frequency differential spatial attention matrix to obtain a high-frequency differential spatial attention map;

[0090] Step 4: Obtain a high-frequency differential spatial attention spatial fidelity term according to the attention injection weight map and the high-frequency differential spatial attention map;

[0091] Step 5: According to the low tube-rank characteristic of the high-resolution multispectral (HRMS) image, introduce the logarithmic tensor nuclear norm to obtain a tensor low tube-rank constraint term;

[0092] Step 6: Combine the spectral fidelity term, the high-frequency differential spatial attention spatial fidelity term, and the tensor low tube-rank constraint term to build a mathematical model of the high-frequency differential spatial attention mechanism pan-sharpening algorithm;

[0093] Step 7: Use the alternating direction method of multipliers to optimize and solve the mathematical model, and the final result obtained is the high-resolution multispectral (HRMS) image.

[0094] In Step 1, two remote sensing satellite datasets, Worldview-3 and GaoFen2, were used in the experiment. The spatial sampling intervals of the PAN image and the LRMS image in the Worldview-3 dataset are 0.3m and 1.2m respectively, and the number of bands of the multispectral image is 8, and the spatial resolution ratio between the two images is 4. The resolutions of the PAN image, the LRMS image, and the GT image are 512×512, 128×128×8, and 512×512×8 respectively. GT represents the ground truth image, which is obtained through the Walt protocol and is used for reference. The spatial sampling intervals of the PAN image and the LRMS image in the GaoFen2 dataset are 1m and 4m respectively, and the number of bands of the multispectral image is 4, and the spatial resolution ratio between two images is 4. The resolutions of the PAN image, LRMS image, and GT image are 512×512, 128×128×4, and 512×512×4 respectively. For the two deep learning-based algorithms in the comparison algorithms, they are trained on the Worldview-3 and GaoFen2 datasets respectively, and the training set and the test set do not overlap spatially. The training set, validation set, and test set are divided in the ratio of 70%, 20%, and 10%. The experiments are all implemented by training on a computer with an 11th Gen Intel(R) Core(TM) i5-11260H CPU, 2.60 GHz, and 16.0G of memory using an NVIDIA GeForce RTX 3050 Laptop GPU.

[0095] In step 1, in order to keep the spectrum unchanged between the HRMS image and the LRMS image , the LRMS image is modeled as a spatially degraded version of the HRMS image , that is , where represents the spatial downsampling matrix, represents the Gaussian blur matrix,

[0096] represents zero-mean Gaussian noise, and the obtained spectral fidelity term is:

[0097] ,

[0098] where represents the square of the norm of the matrix.

[0099] The specific steps of step 2 are as follows:

[0100] Step 2-1: Upsample the LRMS image to obtain the upsampled version LRMS image ,

[0101] Step 2-2: Obtain the low-pass filtered image of the PAN image by passing the PAN image through a low-pass filter. The low-pass filter is learned through two cascaded

[0102] convolutional layers with a ReLU activation function introduced in the middle, as follows:

[0103] where represents Convolutional layer, where ReLU represents the activation function;

[0104] Step 2-3: Concatenate the upsampled version of the LRMS image , the PAN image and the low-pass filtered image of the PAN image along the channel dimension to obtain a tensor ;

[0105] Step 2-4: Pass the tensor through a convolutional layer for preliminary feature extraction and obtain the corresponding feature map. Feed the convolved feature map into the ReLU activation function, and then pass it through another convolutional layer to integrate the features activated by the ReLU activation function to obtain the attention injection weight map , as follows:

[0106] .

[0107] The specific steps of Step 3 are as follows:

[0108] Step 3-1: Input the PAN image and the low-pass filtered image of the PAN image into the differential attention module to calculate their attention matrices respectively, and obtain the attention matrix of the PAN image , the attention matrix of the low-pass filtered image of the PAN image , where, is:

[0109] ,

[0110] where, , represents the query of the network model, , represents the key of the network model, , represents the transposed matrix of the key of the network model, represents the value of the network model, is a learnable scalar, represents the dimension of each branch,

[0111] Step 3-2: According to the obtained attention matrices and , calculate the high-frequency differential spatial attention , ,

[0112] Step 3-3: For the high-frequency differential spatial attention Average each row, calculate the average attention difference, and obtain a dimensional vector , as follows:

[0113] , ,

[0114] Among them, represents the dimension of represents the th row and the th column attention difference value, represents the th row average attention difference,

[0115] Step 3-4: Reshape the vector back to the patch grid, represents the reshaped matrix, and each element corresponds to the attention difference of a certain patch in the image;

[0116] Step 3-5: Upsample the reshaped matrix back to the original image size , and obtain the final high-frequency differential spatial attention map .

[0117] Furthermore, in step 4, according to the attention injection weight map and the high-frequency differential spatial attention map , obtain the high-frequency differential spatial attention spatial fidelity term, which is expressed as:

[0118] ,

[0119] Among them, represents element-wise multiplication.

[0120] Furthermore, in step 5, introduce the logarithmic tensor nuclear norm to obtain the tensor low-rank constraint term:

[0121] ,

[0122] Among them, represents the logarithmic tensor nuclear norm.

[0123] Furthermore, in step 6, combine the spectral fidelity term with the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency differential spatial attention mechanism panchromatic sharpening algorithm, and obtain the mathematical model formula as:

[0124]

[0125] ,

[0126] ,

[0127] Among them, and are two balance parameters; introducing the tensor auxiliary variable , the mathematical optimization model is re-expressed as:

[0128]

[0129] ,

[0130] ,

[0131] Among them, is the tensor auxiliary variable, used to assist in iterating the HRMS image .

[0132] Furthermore, in step 7, the alternating direction method of multipliers (ADMM) is used to optimize and solve the mathematical optimization model. The specific steps are as follows:

[0133] Using the augmented Lagrangian function to convert the constrained optimization problem into an unconstrained optimization problem, and its augmented Lagrangian function is derived as:

[0134]

[0135] ,

[0136] Among them, is the Lagrange multiplier, represents the tensor inner product, is the penalty parameter;

[0137] Through initialization, , , alternately and iteratively optimize the model until the set termination condition is met. The finally obtained result is the HRMS image of this panchromatic sharpening task;

[0138] Alternately and iteratively update the augmented Lagrangian function , that is:

[0139] .

[0140] Furthermore, iteratively update :

[0141] ,

[0142] Among them, Denote the th iteration, denote the sub-problem solved in the th iteration, and the sub-problem is derived as:

[0143]

[0144] ,

[0145] Iteratively update :

[0146] ,

[0147] denote the sub-problem solved in the th iteration, and the sub-problem is derived as:

[0148] ,

[0149] Iteratively update the Lagrange multiplier . According to the standard practice of the augmented Lagrangian method, the multiplier is updated by gradient ascent with the current degree of constraint violation. Denote the sub-problem solved in the th iteration, and the sub-problem is derived as:

[0150] .

[0151] Figure 2 This is the structural schematic diagram of an attention injection weight module provided by the present invention. First, perform an upsampling operation on the low-resolution multispectral image to obtain an upsampled version of the low-resolution multispectral image; then perform two consecutive convolution operations on the panchromatic image and add a ReLU function activation in the middle to learn the low-pass filtered image of the panchromatic image; then, concatenate the upsampled version of the low-resolution multispectral image, the panchromatic image, and the low-pass filtered image of the panchromatic image along the channel dimension to form a tensor containing multi-source information; then, perform preliminary feature extraction on this tensor through a convolutional layer and send the convolution result into the ReLU activation function, and then integrate the activated features through another convolutional layer to finally obtain the attention injection weight map.

[0152] Figure 3 ​​​Schematic diagram of the structure of a differential attention module of the present invention. The differential attention module first obtains , , , , through two sets of linear mappings, and then calculates the differential attention of and , multiplies it by , normalizes the output of the differential attention, and then fuses it with the input features in the ratio of . Finally, it is sent to a linear layer to obtain the final result, thereby highlighting the differential information while retaining the advantages of the original attention mechanism.

[0153] Table 1 shows the quantitative evaluation indicators of all comparison methods on the downsampled WorldView-3 dataset.

[0154] Table 2 shows the quantitative evaluation indicators of all comparison methods on the downsampled GaoFen2 dataset.

[0155] Tables 1 and 2 show the comparison of the quantitative evaluation indicators of six classic pansharpening methods and the method of the present invention in the comparative experiment on the WordView-3 dataset and the GaoFen2 dataset respectively; among them, BDSD, BDSD-PC and GS are based on the component substitution method, SR-D is based on the variational optimization method, CANNet and Pan-Mamba are based on deep learning methods, and Ours represents the result of the pansharpening method of the present invention based on the high-frequency differential spatial attention mechanism. It can be seen from the comparison that the method of the present invention has achieved good results in various indicators of the high-resolution multispectral image, and has good accuracy, generalization and efficiency.

[0156] Table 1: Quantitative evaluation indicators of all comparison methods on the downsampled WorldView-3 dataset

[0157]

[0158] Table 2: Quantitative evaluation indicators of all comparison methods on the downsampled GaoFen2 dataset

[0159] 。

[0160] Example 2: The present invention provides a pansharpening system based on a differential spatial attention mechanism, including:

[0161] Image acquisition module: It is used to obtain image data from remote sensing satellite datasets. The image data includes panchromatic (PAN) images and low-resolution multispectral (LRMS) images, and performs preprocessing operations such as pixel-level registration and geometric correction on the images to ensure their consistency in the sampling area and acquisition time, obtaining perfectly matched PAN images and LRMS images , and uses the processed PAN images and LRMS images as input data for subsequent models and high-frequency differential spatial attention calculations;

[0162] Spectral modeling module: It is used to model spectral fidelity between multispectral observations based on the spectral degradation relationship of HRMS images to obtain a spectral fidelity term;

[0163] Attention injection module: It is used to cascade the upsampled version of the LRMS image , the PAN image and the low-pass filtered image of the PAN image along the channel dimension, and then obtains a pixel-level attention injection weight map through attention injection ;

[0164] High-frequency differential spatial attention module: It is used to input the PAN image and its low-pass filtered image into the differential attention module, calculate their attention matrices respectively, calculate the high-frequency differential spatial attention , reshape the high-frequency differential spatial attention to obtain a high-frequency differential spatial attention map , combine the attention injection weight map and the high-frequency differential spatial attention map to obtain a high-frequency differential spatial attention spatial fidelity term;

[0165] Mathematical modeling module: It is used to combine the spectral fidelity term, the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build an overall mathematical optimization model to obtain the final mathematical optimization model;

[0166] ADMM optimization output module: It is used to solve the mathematical optimization model by the alternating direction method of multipliers (ADMM), and update respectively in each iteration , , , until the set termination conditions are met; the finally output result is the high-resolution multispectral (HRMS) image obtained by the system for panchromatic sharpening.

[0167] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A panchromatic sharpening method based on a high-frequency differential spatial attention mechanism, characterized in that The method includes the following steps: Step 1: Based on the high-resolution multi-spectral (HRMS) image, model the low-resolution multi-spectral (LRMS) image as a spatially degraded version of the high-resolution multi-spectral (HRMS) image to obtain a spectral fidelity term; Step 2: Upsample the low-resolution multi-spectral (LRMS) image to obtain an upsampled version of the LRMS image. Concatenate the upsampled version of the LRMS image, the panchromatic (PAN) image, and the low-pass filtered image of the panchromatic (PAN) image along the channel dimension, and then obtain a pixel-level attention injection weight map through an attention injection module; Step 3: Input the panchromatic (PAN) image and the low-pass filtered image of the panchromatic (PAN) image into a differential attention module respectively to calculate a high-frequency differential spatial attention matrix, and reshape the high-frequency differential spatial attention matrix to obtain a high-frequency differential spatial attention map; Step 4: Based on the attention injection weight map and the high-frequency differential spatial attention map, obtain a high-frequency differential spatial attention spatial fidelity term; Step 5: According to the low-rank characteristic of the high-resolution multi-spectral (HRMS) image, introduce a logarithmic tensor nuclear norm to obtain a tensor low-rank constraint term; Step 6: Combine the spectral fidelity term, the high-frequency differential spatial attention spatial fidelity term, and the tensor low-rank constraint term to build a mathematical model of a high-frequency differential spatial attention mechanism panchromatic sharpening algorithm; Step 7: Use the alternating direction method of multipliers to optimize and solve the mathematical model, and the final result obtained is the high-resolution multi-spectral (HRMS) image; Among them, in step 4, according to the attention injection weight map and the high-frequency differential spatial attention map obtain the high-frequency differential spatial attention spatial fidelity term, which is expressed as: Among them, ⊙ represents element-wise multiplication, represents the upsampled version of the LRMS image, represents the HRMS image.

2. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 1, wherein The HRMS image is represented as The LRMS image is represented as The PAN image is represented as where represents the set of real numbers, H and W are the height and width respectively, C is the number of bands of the multispectral image, and r is the spatial resolution ratio; In step 1, the LRMS image is modeled as a spatially degraded version of the HRMS image and Namely wherein represents a spatial downsampling matrix, represents a Gaussian blur matrix, Denote zero-mean Gaussian noise, and the obtained spectral fidelity term is as follows: Among them, represents the square of the Frobenius norm of the matrix.

3. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 2, wherein The specific steps of Step 2 are as follows: Step 2-1: Upsample the LRMS image to obtain an upsampled version of the LRMS image Step 2-2: Take the PAN image and obtain the low-pass filtered image of the PAN image through a low-pass filter The low-pass filter is learned through two cascaded 5×5 convolutional layers with a ReLU activation function introduced in the middle, as follows: Among them, Conv 5×5 represents a 5×5 convolutional layer, and ReLU represents an activation function; Step 2-3: Concatenate the upsampled LRMS image PAN image and the low-pass filtered image of the PAN image along the channel dimension to obtain a tensor Step 2-4: Take the tensor and perform preliminary feature extraction through a 3×3 convolutional layer to obtain the corresponding feature map. Feed the convolved feature map into the ReLU activation function, and then integrate the features activated by the ReLU activation function through another 3×3 convolutional layer to obtain the attention injection weight map as follows:

4. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 3, characterized in that, The specific steps of Step 3 are as follows: Step 3-1: Input the PAN image and the low-pass filtered image of the PAN image into the differential attention module to calculate their attention matrices respectively, obtaining the attention matrix of the PAN image and the attention matrix of the low-pass filtered image of the PAN image wherein is as follows: where Q1 and Q2 represent queries of the network model, K1 and K2 represent keys of the network model, represents the transposed matrix of the keys of the network model, V represents the values of the network model, λ is a learnable scalar; d represents the dimension of each branch; Step 3-2: According to the obtained attention matrix and calculate the high-frequency differential spatial attention Step 3-3, calculate the average of each row of the high-frequency differential spatial attention to calculate the average attention difference and obtain an N-dimensional vector e = (e1, e2, …, e N ) is as follows: where N represents the dimension of which represents the difference value of the attention between the i-th row and the j-th column, and e i represents the average attention difference of the i-th row. Step 3-4: Reshape the vector e back into a patch grid. Let E denote the reshaped matrix, and each element E ij corresponds to the attention difference of the patch in the image; Step 3-5: Upsample the reshaped matrix E back to the original image size H×W to obtain the final high-frequency differential spatial attention map 5. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 4, characterized in that In Step 5, introduce a logarithmic tensor nuclear norm to obtain a tensor low-rank constraint term: Among them, represents the logarithmic tensor nuclear norm.

6. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 5, characterized in that In Step 6, combine the spectral fidelity term, the high-frequency differential spatial attention spatial fidelity term, and the tensor low-rank constraint term to build a mathematical model of a high-frequency differential spatial attention mechanism panchromatic sharpening algorithm, and the mathematical model is formulated as: where α and β are two balance parameters; Introduce tensor auxiliary variables The mathematical optimization model is re-expressed as: Among them, is a tensor auxiliary variable used to assist in iterating the HRMS image 7. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 6, wherein In Step 7, use the alternating direction method of multipliers (ADMM) to optimize and solve the mathematical optimization model, and the specific steps are as follows: Use the augmented Lagrangian function to convert the constrained optimization problem into an unconstrained optimization problem, and its augmented Lagrangian function derivation is: wherein, is a Lagrange multiplier, represents the inner product of tensors, η > 0 is the penalty parameter; Through initialization, alternately iterate and optimize the model in sequence until the set termination condition is met. The finally obtained result is the high-resolution multispectral (HRMS) image. Alternating iterative update of the augmented Lagrangian function That is:

8. The panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to claim 7, wherein, Iterative update where k represents the k-th iteration, represents the sub-problem solved in the k-th iteration , and the sub-problem is deduced as: Iterative update Denote the sub-problem solved in the k-th iteration , which is derived as follows: Iterative update of Lagrange multipliers According to the standard practice of the augmented Lagrangian method, the multipliers are updated by gradient ascent with the current degree of constraint violation, denoting the solution of the subproblem at the k-th iteration, and the subproblem is derived as:

9. A panchromatic sharpening system based on a high-frequency differential spatial attention mechanism for implementing the panchromatic sharpening method based on the high-frequency differential spatial attention mechanism according to any one of claims 1-8, characterized in that, The system includes Image acquisition module: It is used to obtain image data from remote sensing satellite datasets. The image data includes panchromatic (PAN) images and low-resolution multispectral (LRMS) images, and performs preprocessing operations of pixel-level registration and geometric correction on the images to ensure their consistency in the sampling area and acquisition time, and obtain a perfectly matched PAN image and LRMS image The processed PAN image and LRMS image are used as input data for subsequent model and high-frequency differential spatial attention calculation; Spectral modeling module: used to model the spectral fidelity between multi-spectral observations according to the spectral degradation relationship of the HRMS image to obtain the spectral fidelity term Attention injection module: used to concatenate the upsampled version of the LRMS image along the channel dimension PAN image and the low-pass filtered image of the PAN image and then obtain a pixel-level attention injection weight map through attention injection High-frequency differential spatial attention module: used to input the PAN image and its low-pass filtered image into the differential attention module, calculate their attention matrices respectively, and calculate the high-frequency differential spatial attention according to the attention matrices Reshape the high-frequency differential spatial attention to obtain the high-frequency differential spatial attention map Inject the attention into the weight map and the high-frequency differential spatial attention map to combine and obtain the high-frequency differential spatial attention spatial fidelity term; A mathematical modeling module: used to combine the spectral fidelity term, the high-frequency differential spatial attention spatial fidelity term, and the tensor low-rank constraint term to build an overall mathematical optimization model and obtain the final mathematical optimization model; ADMM Optimization Output Module: Used to solve the mathematical optimization model by the Alternating Direction Method of Multipliers (ADMM), and update separately in each iteration until the set termination condition is met; finally, output the high-resolution multispectral (HRMS) image.

Images