Panchromatic sharpening method and system based on high-frequency differential space attention mechanism

Abstract

The invention relates to a panchromatic sharpening method and system based on a high-frequency difference space attention mechanism, and the method comprises the steps: carrying out the modeling of a low-resolution multispectral LRMS image into a space degradation version of a high-resolution multispectral HRMS image, obtaining a spectrum fidelity term, generating an attention injection weight map through an attention injection module, and carrying out the processing of a spectrum fidelity term. Respectively inputting the panchromatic PAN image and the low-pass filtering image into a difference attention book module to calculate to obtain a high-frequency difference attention graph, then constructing a high-frequency difference space fidelity term according to an attention injection weight graph and the high-frequency difference space attention graph, and introducing a logarithm nuclear norm of a high-resolution multispectral HRMS image to obtain a high-resolution multispectral HRMS image; according to the method, a spectral fidelity term is combined with a high-frequency differential space attention space fidelity term and a tensor low-tube-rank constraint term, a panchromatic sharpening algorithm mathematical model is built, finally, a substitution direction multiplier method is adopted to carry out optimization solution on the mathematical model, and finally, a high-resolution multispectral HRMS image result is output.

Description

Technical Field

[0001] The present invention relates to a full color sharpening method and system based on a high frequency differential spatial attention mechanism, belonging to the technical field of full color sharpening. Background Art

[0002] Panchromatic sharpening technology is an image processing method that fuses a pair of high spatial resolution panchromatic PAN images with low spatial resolution but multi-band low-resolution multispectral LRMS images to generate a high-resolution multispectral HRMS image with high spatial resolution and maintaining multispectral spectral information. This technology originates from the continuous advancement of remote sensing satellites and earth observation technology. Various types of high spatial resolution panchromatic PAN images and low spatial resolution but multi-band low-resolution multispectral LRMS images have emerged in large numbers. However, relying solely on multispectral images cannot obtain clear details and texture features, and although panchromatic images have high spatial resolution, they lack information in the spectral dimension. Therefore, panchromatic sharpening technology came into being. This technology is of great significance in the fields of agricultural monitoring, ecological assessment, urban planning, disaster prevention and mitigation, etc. It can discover the distribution and changes of objects at a finer scale and improve the comprehensive utilization value of earth observation data.

[0003] At present, the research methods of panchromatic sharpening are mainly divided into the following categories: methods based on component substitution, methods based on multi-resolution analysis, methods based on variational optimization, and methods based on deep learning. Typical component substitution methods include IHS, PCA, Brovey, Gram–Schmidt, etc., which mainly project the LRMS image to a component space such as brightness or principal component, and then use the PAN image to replace or inject this principal component, and then inversely transform it back to the multispectral domain to generate a high-resolution multispectral HRMS image. This type of method is simple to implement and has a fast calculation speed. It can improve the clarity of the image to a certain extent, but due to the insufficient fidelity of the spectral information during the replacement process, it is easy to produce serious spectral distortion and cannot capture the complex relationship between multispectral channels at a deeper level. Methods based on multi-resolution analysis include wavelet transform, contourlet, Laplacian Pyramid, etc., which mainly decompose the PAN image and the LRMS image into multi-scale or multi-resolution respectively, and then fuse the high-frequency components at the corresponding level, and finally reconstruct the high-resolution multispectral HRMS image. This type of method can better inject high-frequency details to solve the problem of spectral distortion, but it is easy to cause fusion artifacts in strong noise or large-scale difference scenes and has high requirements for filter design. The variational optimization-based method mainly mathematically models the more appropriate and effective relationship between HRMS images and LRMS images, between HRMS images and PAN images, and between the spatial and spectral prior constraints of HRMS images, incorporates spectral fidelity, smoothing priors, spatial details, etc. into a unified optimization framework, and uses regularization to constrain the fusion process to obtain high-resolution multispectral HRMS images. With the success of deep neural networks in tasks such as image super-resolution and style transfer, panchromatic sharpening methods have made new developments. In the existing technology, deep neural networks such as CNN, GAN, and Transformer are applied to panchromatic sharpening tasks to learn the mapping relationship between multi-band and high-frequency features in an end-to-end manner. The deep learning-based method can produce very promising fusion results, but it often requires a large amount of labeled training data, and has high requirements on network structure and training strategy, and has low generalization performance. In summary, the existing panchromatic sharpening technology still has a lot of room for development. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a panchromatic sharpening method based on a high-frequency differential spatial attention mechanism. Image data is obtained from a remote sensing satellite data set, and the image data includes a panchromatic PAN image and a low-resolution multispectral LRMS image. The image is preprocessed by pixel-level registration and geometric correction to obtain a completely matched PAN image and LRMS image. The low-resolution multispectral LRMS image is modeled as a spatially degraded version of a high-resolution multispectral HRMS image, and a spectral fidelity constraint is established. The LRMS image is upsampled to obtain an upsampled version of the LRMS image, and the upsampled version of the LRMS image, the PAN image and the low-pass filtered image of the PAN image are cascaded along the channel dimension, and then an attention injection module is used to obtain a pixel-level attention injection weight map. The PAN image and the low-pass filtered image of the PAN image are then subjected to a differential attention module to calculate a high-frequency differential spatial attention. , get the high-frequency differential spatial attention map , establish high-frequency differential spatial fidelity constraints; according to the low-rank characteristics of HRMS images, introduce the logarithmic tensor nuclear norm to obtain the tensor low-rank constraint term; further combine the high-frequency differential spatial attention spatial fidelity term, spectral fidelity term and tensor low-rank constraint term to build a unified mathematical optimization model, and solve iteratively through the alternative direction multiplier method ADMM, continuously correct and enhance edge and texture details, and finally output a high-resolution multispectral HRMS image. It achieves more than just pixel-level spatial fidelity, but aligns the spatial structure of PAN images and LRMS images in the feature domain constructed by high-frequency differential spatial attention, so that the model can more accurately extract real high-frequency spatial details, reduce the impact of noise, solve spectral distortion, and improve data generalization performance, and obtain a high-resolution multispectral HRMS image with higher spatial resolution and multispectral spectral information.

[0005] The present invention adopts the following technical solutions to solve the above technical problems: In a first aspect, the present invention provides a full color sharpening method based on a high-frequency differential spatial attention mechanism, the method comprising the following steps: Step 1: Based on the high-resolution multispectral HRMS image, the low-resolution multispectral LRMS image is modeled as a spatially degraded version of the high-resolution multispectral HRMS image to obtain a spectral fidelity term; Step 2: Upsample the low-resolution multispectral LRMS image to obtain an upsampled version of the LRMS image, concatenate the upsampled version of the LRMS image, the full-color PAN image, and the low-pass filtered image of the full-color PAN image along the channel dimension, and then obtain a pixel-level attention injection weight map through the attention injection module; Step 3: Input the full-color PAN image and the low-pass filtered image of the full-color PAN image into the differential attention module to calculate the high-frequency differential spatial attention matrix, reshape the high-frequency differential spatial attention matrix, and obtain the high-frequency differential spatial attention map; Step 4: According to the attention injection weight map and the high-frequency difference spatial attention map, a high-frequency difference spatial attention spatial fidelity item is obtained; Step 5, according to the low-rank characteristics of the high-resolution multispectral HRMS image, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term; Step 6: Combine the spectral fidelity term with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism pan-sharpening algorithm; Step 7: Use the alternative direction multiplier method to optimize and solve the mathematical model, and the final result obtained is a high-resolution multispectral HRMS image.

[0006] Furthermore, the HRMS image is represented as , the LRMS image is represented as , the PAN image is represented as ,in represents the set of real numbers, and are the height and width respectively, is the number of bands of the multispectral image, is the spatial resolution ratio; In step 1, the LRMS image Modeling as HRMS images The spatially degenerate version of Right now ,in represents the spatial downsampling matrix, represents the Gaussian blur matrix, represents zero-mean Gaussian noise, and the obtained spectral fidelity term is: , in, Represents the matrix The square of the norm.

[0007] Furthermore, the specific steps of step 2 are as follows: Step 2-1: LRMS image Perform upsampling operation to obtain the upsampled version LRMS image , Step 2-2: PAN image The low-pass filtered image of the PAN image is obtained by low-pass filtering. The low-pass filter is connected in series with two The convolutional layer is learned by introducing the ReLU activation function in the middle, as follows: , in, express Convolutional layer, ReLU represents the activation function; Step 2-3: Upsample the LRMS image , PAN image And the low-pass filtered image of the PAN image Cascade along the channel dimension to get a tensor ; Step 2-4: Tensor Through a The convolutional layer performs preliminary feature extraction and obtains the corresponding feature map. The convolutional feature map is sent to the ReLU activation function and then passes through another The convolutional layer integrates the features activated by the ReLU activation function to obtain the attention injection weight map ,as follows: .

[0008] Furthermore, the specific steps of step 3 are as follows: Step 3-1: PAN image And low-pass filtered image of PAN image Input into the differential attention module to calculate its attention matrix and get the PAN image The attention matrix , low-pass filtered image of PAN image The attention matrix ,in, for: , in, , represents a query of the network model, , A key representing a network model, , The transposed matrix of the keys representing the network model, represents the value of the network model, represents a learnable scalar, represents the dimension of each branch, Step 3-2: Based on the obtained attention matrix and , calculate high-frequency differential spatial attention , , Step 3-3: Attention to high-frequency differential space Take the average of each row of , calculate the average attention difference, and get a Dimensional vector ,as follows: , , in, express The dimension of Representative Row and The difference value of column attention, Indicates The average attention difference of the rows, Step 3-4: Vector Reshape back to the patch mesh, Represents the reshaped matrix, each element The attention difference corresponding to a patch in the image; Step 3-5: Reshape the matrix Upsample back to original image size , and obtain the final high-frequency differential spatial attention map .

[0009] Furthermore, in step 4, the weight map is injected according to the attention and high-frequency differential spatial attention map , we get the high-frequency difference spatial attention spatial fidelity term, expressed as: , in, Represents element-wise multiplication.

[0010] Furthermore, in step 5, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term: , in, Represents the logarithmic tensor nuclear norm.

[0011] Furthermore, in step 6, the spectral fidelity term is combined with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism panchromatic sharpening algorithm, and the mathematical model is formulated as: , , in, and are two balance parameters; tensor auxiliary variables are introduced , the mathematical optimization model is reformulated as: , , in, A tensor auxiliary variable used to assist in iterating HRMS images .

[0012] Furthermore, in step 7, the alternative direction multiplier method ADMM is used to optimize and solve the mathematical optimization model. The specific steps are as follows: The augmented Lagrangian function is used to transform the constrained optimization problem into an unconstrained optimization problem. The augmented Lagrangian function is derived as follows: , in, is the Lagrange multiplier, represents the inner product of the tensor, is the penalty parameter; by initialization, , , The optimization model is iterated alternately in sequence until the set termination condition is met. The final result is the HRMS image of this full-color sharpening task; the augmented Lagrangian function is updated alternately and iteratively ,Right now: .

[0013] Further, iterative updates : , in, Indicates Iterations, Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: , Iterative Updates : , Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: , Iteratively update the Lagrange multiplier , according to the standard practice of augmented Lagrangian method, the multiplier is updated by gradient ascent according to the current degree of constraint violation, Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: .

[0014] In a second aspect, the present invention provides a full color sharpening system based on a high-frequency differential spatial attention mechanism, comprising: Image acquisition module: used to obtain image data from remote sensing satellite data sets, including full-color PAN images and low-resolution multispectral LRMS images, and perform pixel-level registration and geometric correction preprocessing operations on the images to ensure the consistency of the two in sampling area and acquisition time, and obtain a completely matched PAN image and LRMS images , the processed PAN image and LRMS image are used as input data for subsequent models and high-frequency differential spatial attention calculations; Spectral modeling module: used to The spectral degradation relationship of is used to model the spectral fidelity between multi-spectral observations and obtain the spectral fidelity term; Attention injection module: used to inject the upsampled version of LRMS image along the channel dimension , PAN image And the low-pass filtered image of the PAN image Cascade, and then inject attention to get the pixel-level attention injection weight map ; High-frequency difference spatial attention module: used to transform PAN images and its low-pass filtered image Input into the differential attention module, calculate its attention matrix respectively, and calculate the high-frequency differential spatial attention according to the attention matrix , the high-frequency differential spatial attention Reshape to get the high-frequency differential spatial attention map , injecting attention into the weight map and high-frequency differential spatial attention map Combined, we get the high-frequency difference spatial attention spatial fidelity term; Mathematical modeling module: used to combine the spectral fidelity term with the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build an overall mathematical optimization model and obtain the final mathematical optimization model; ADMM optimization output module: used to solve the mathematical optimization model through the alternative direction multiplier method ADMM, and update the , , , until the set termination condition is met; the final output result is the high-resolution multi-spectral HRMS image obtained by the system after completing full-color sharpening.

[0015] Compared with the prior art, the present invention adopts the above technical solution and has the following beneficial effects: 1. The present invention proposes a panchromatic sharpening method based on a high-frequency differential spatial attention mechanism, which can better retain high-resolution details in the fusion process. The extracted high-frequency differential spatial attention of the panchromatic image is combined with the low-resolution multispectral image through attention injection, which can capture the dependency between long-distance pixels in the image, and can further highlight the key texture edges and spatial details on the basis of the global context, so that the sharpened image can better take into account the local detail features. At the same time, the differential operation of the two-way spatial attention can eliminate common noise to a certain extent, so that the focus is focused on the real high-frequency change area, retaining more accurate spectral information and spatial information, and then outputting a higher-quality high-resolution multispectral image; 2. The present invention constructs a universal full-color sharpening optimization model by integrating the traditional variational optimization method with the proposed high-frequency differential spatial attention mechanism. On the one hand, it utilizes the good interpretability and strong data generalization ability of the variational optimization method, and on the other hand, it gives full play to the advantages of the differential spatial attention mechanism in global modeling and attention injection, organically combining the advantages of the two to achieve better results in the full-color sharpening task. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall process of the full color sharpening method of the present invention, Figure 2 A structural diagram of an attention injection module provided by the present invention, Figure 3 A structural schematic diagram of a differential attention module provided by the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clear, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments.

[0018] Example 1: See Figure 1A panchromatic sharpening method based on high-frequency differential spatial attention mechanism is proposed. Image data are obtained from remote sensing satellite datasets Worldview-3 and GaoFen2. The image data include panchromatic PAN images and low-resolution multispectral LRMS images. The spectral degradation of LRMS images is modeled in combination with HRMS images, and spectral fidelity constraints are established to ensure that the fusion results remain accurate in the spectral domain. The low-resolution multispectral LRMS images are upsampled to obtain upsampled LRMS images. The upsampled LRMS images, panchromatic PAN images and low-pass filtered images of panchromatic PAN images are spliced ​​along the channel dimension, and then the attention injection module is used to obtain the pixel-level attention injection weight map. The panchromatic PAN images and the low-pass filtered images of panchromatic PAN images are then passed through the differential attention module to calculate the high-frequency differential spatial attention weight map. , the high-frequency differential spatial attention Reshape to get the high-frequency differential spatial attention map , a high-frequency differential spatial attention fidelity constraint is established; on this basis, according to the low-rank characteristics of the HRMS image, the logarithmic nuclear norm is introduced to establish the tensor low-rank constraint; the high-frequency differential spatial attention spatial fidelity term is further combined with the spectral fidelity term and the tensor low-rank constraint term to construct a unified mathematical optimization model, and the alternative direction multiplier method ADMM is used to iteratively solve the problem, continuously correct and enhance the edge and texture details, and finally output a high-resolution multispectral HRMS image.

[0019] Figure 1 The invention is a flow chart of a panchromatic sharpening method based on a high-frequency differential spatial attention mechanism of the present invention. First, a low-resolution multispectral LRMS image is modeled as a spatially degraded version of a high-resolution multispectral HRMS image to obtain a spectral fidelity term; then, the upsampled version LRMS image, the panchromatic PAN image and the low-pass filtered image of the panchromatic PAN image are spliced ​​along the channel dimension, and a pixel-level attention injection weight map is generated through an attention injection module; at the same time, the panchromatic PAN image and the low-pass filtered image of the panchromatic PAN image are respectively input into the differential attention module to calculate a high-frequency differential attention map; then, a high-frequency differential spatial attention spatial fidelity term is constructed according to the attention injection weight map and the high-frequency differential spatial attention map; according to the low-rank characteristic of the HRMS image, a logarithmic nuclear norm is introduced to strengthen the tensor low-rank constraint; then, the spectral fidelity term is combined with the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the panchromatic sharpening algorithm of the high-frequency differential spatial attention mechanism; finally, the alternative direction multiplier method is used to optimize and solve the mathematical model, and finally a high-resolution multispectral HRMS image result is output.

[0020] The full color sharpening method based on high-frequency differential spatial attention mechanism described in the present invention includes the following contents, specifically as follows: Step 1: Based on the high-resolution multispectral HRMS image, the low-resolution multispectral LRMS image is modeled as a spatially degraded version of the high-resolution multispectral HRMS image to obtain a spectral fidelity term; Step 2: Upsample the low-resolution multispectral LRMS image to obtain an upsampled version of the LRMS image, concatenate the upsampled version of the LRMS image, the full-color PAN image, and the low-pass filtered image of the full-color PAN image along the channel dimension, and then obtain a pixel-level attention injection weight map through the attention injection module; Step 3: Input the full-color PAN image and the low-pass filtered image of the full-color PAN image into the differential attention module to calculate the high-frequency differential spatial attention matrix, reshape the high-frequency differential spatial attention matrix, and obtain the high-frequency differential spatial attention map; Step 4: According to the attention injection weight map and the high-frequency difference spatial attention map, a high-frequency difference spatial attention spatial fidelity item is obtained; Step 5, according to the low-rank characteristics of the high-resolution multispectral HRMS image, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term; Step 6: Combine the spectral fidelity term with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism pan-sharpening algorithm; Step 7: Use the alternative direction multiplier method to optimize and solve the mathematical model, and the final result obtained is a high-resolution multispectral HRMS image.

[0021] In step 1, the experiment used two remote sensing satellite datasets, Worldview-3 and GaoFen2. The spatial sampling intervals of the PAN image and LRMS image of the Worldview-3 dataset were 0.3m and 1.2m respectively, and the band number of the multispectral image was is 8, the spatial resolution ratio between the two images is The resolutions of PAN, LRMS and GT images are 512×512, 128×128×8 and 512×512×8 respectively. GT represents the ground truth image, which is obtained through the Walt protocol and is used for reference. The spatial sampling intervals of PAN and LRMS images in the GaoFen2 dataset are 1m and 4m respectively. The number of bands of multispectral images is The spatial resolution ratio between the two images is 4, and the resolutions of PAN, LRMS, and GT images are 512×512, 128×128×4, and 512×512×4, respectively. For the two deep learning algorithms in the comparison algorithm, they were trained on the Worldview-3 and GaoFen2 datasets respectively, and the training set and the test set did not overlap in space. The training set, validation set, and test set were divided into 70%, 20%, and 10% respectively. The experiments were all implemented by NVIDIA GeForce RTX 3050 Laptop GPU training on a computer with 11th Gen Intel(R) Core(TM) i5-11260HCPU, 2.60GHz, and 16.0G memory.

[0022] In step 1, in order to and LRMS images Keep the spectrum unchanged and convert the LRMS image Modeling as HRMS images The spatially degenerate version of ,in represents the spatial downsampling matrix, represents the Gaussian blur matrix, represents zero-mean Gaussian noise, and the obtained spectral fidelity term is: , in, Represents the matrix The square of the norm.

[0023] Step 2 The specific steps are as follows: Step 2-1: LRMS image Perform upsampling operation to obtain the upsampled version LRMS image , Step 2-2: PAN image The low-pass filtered image of the PAN image is obtained by low-pass filtering. The low-pass filter is connected in series with two The convolutional layer is learned by introducing the ReLU activation function in the middle, as follows: , in, express Convolutional layer, ReLU represents the activation function; Step 2-3: Upsample the LRMS image , PAN image And the low-pass filtered image of the PAN image Cascade along the channel dimension to get a tensor ; Step 2-4: Tensor Through a The convolutional layer performs preliminary feature extraction and obtains the corresponding feature map. The convolutional feature map is sent to the ReLU activation function and then passes through another The convolutional layer integrates the features activated by the ReLU activation function to obtain the attention injection weight map ,as follows: .

[0024] Step 3 The specific steps are as follows: Step 3-1: PAN image And low-pass filtered image of PAN image Input into the differential attention module to calculate its attention matrix and get the PAN image The attention matrix , low-pass filtered image of PAN image The attention matrix ,in, for: , in, , represents a query of the network model, , A key representing a network model, , The transposed matrix of the keys representing the network model, represents the value of the network model, is a learnable scalar, represents the dimension of each branch, Step 3-2: Based on the obtained attention matrix and , calculate high-frequency differential spatial attention , , Step 3-3: Attention to high-frequency differential space Take the average of each row of , calculate the average attention difference, and get a Dimensional vector ,as follows: , , in, express The dimension of Representative Row and The difference value of column attention, Indicates The average attention difference of the rows, Step 3-4: Vector Reshape back to the patch mesh, Represents the reshaped matrix, each element The attention difference corresponding to a patch in the image; Step 3-5: Reshape the matrix Upsample back to original image size , and obtain the final high-frequency differential spatial attention map .

[0025] Furthermore, in step 4, the weight map is injected according to the attention and high-frequency differential spatial attention map , we get the high-frequency difference spatial attention spatial fidelity term, expressed as: , in, Represents element-wise multiplication.

[0026] Furthermore, in step 5, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term: , in, Represents the logarithmic tensor nuclear norm.

[0027] Furthermore, in step 6, the spectral fidelity term is combined with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism panchromatic sharpening algorithm, and the mathematical model is formulated as: , , in, and are two balance parameters; tensor auxiliary variables are introduced , the mathematical optimization model is reformulated as: , , in, A tensor auxiliary variable used to assist in iterating HRMS images .

[0028] Furthermore, in step 7, the alternative direction multiplier method ADMM is used to optimize and solve the mathematical optimization model. The specific steps are as follows: The augmented Lagrangian function is used to transform the constrained optimization problem into an unconstrained optimization problem. The augmented Lagrangian function is derived as follows: , in, is the Lagrange multiplier, represents the inner product of the tensor, is the penalty parameter; By initializing, , , The optimization model is iterated alternately in sequence until the set termination condition is met, and the final result is the HRMS image of this full color sharpening task; Alternating iterative update of augmented Lagrangian function ,Right now: .

[0029] Further, iterative updates : , in, Indicates Iterations, Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: , Iterative Updates : , Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: , Iteratively update the Lagrange multiplier , according to the standard practice of augmented Lagrangian method, the multiplier is updated by gradient ascent according to the current degree of constraint violation, Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: .

[0030] Figure 2 This is a structural diagram of an attention injection weight module provided by the present invention. First, the low-resolution multispectral image is upsampled to obtain an upsampled low-resolution multispectral image; then the full-color image is processed by two series connections. Convolution operation is performed, and ReLU function activation is added in the middle to learn the low-pass filtered image of the full-color image; then, the upsampled low-resolution multispectral image, the full-color image, and the low-pass filtered image of the full-color image are cascaded along the channel dimension to form a tensor containing multi-source information; then, through a The convolution layer performs preliminary feature extraction on the tensor and sends the convolution result to the ReLU activation function, and then passes through another The convolutional layer integrates the activated features and finally obtains the attention injection weight map.

[0031] Figure 3 This is a schematic diagram of the structure of a differential attention module of the present invention. The differential attention module first obtains , , , , Then calculate through differential attention and The differential attention is then multiplied by , and then normalize the differenced attention output and combine it with the input features to The ratio of the two layers is fused and finally sent to the linear layer to obtain the final result, thereby highlighting the difference information while retaining the advantages of the original attention mechanism.

[0032] Table 1 shows the quantitative evaluation indicators of all compared methods on the down-resolution WorldView-3 dataset. Table 2 shows the quantitative evaluation indicators of all compared methods on the down-resolution GaoFen2 dataset.

[0033] Tables 1 and 2 show the quantitative evaluation index comparisons of six classic panchromatic sharpening methods and the method of the present invention on the WordView-3 dataset and the GaoFen2 dataset in the comparative experiment; BDSD, BDSD-PC and GS are based on component replacement method, SR-D is based on variational optimization method, CANNet and Pan-Mamba are based on deep learning, and Ours represents the panchromatic sharpening method result based on high-frequency differential spatial attention mechanism of the present invention. By comparison, it can be seen that the high-resolution multispectral image obtained by the method of the present invention has achieved good results in various indicators, and has good accuracy, generalization and efficiency.

[0034] Table 1: Quantitative evaluation metrics of all compared methods on the down-scaled WorldView-3 dataset Table 2: Quantitative evaluation indicators of all compared methods on the down-resolution GaoFen2 dataset .

[0035] Embodiment 2: The present invention provides a full color sharpening system based on a differential spatial attention mechanism, comprising: Image acquisition module: used to obtain image data from remote sensing satellite data sets, including full-color PAN images and low-resolution multispectral LRMS images, and perform pixel-level registration and geometric correction preprocessing operations on the images to ensure the consistency of the two in sampling area and acquisition time, and obtain a completely matched PAN image and LRMS images , the processed PAN image and LRMS image are used as input data for subsequent models and high-frequency differential spatial attention calculations; Spectral modeling module: used to The spectral degradation relationship of is used to model the spectral fidelity between multi-spectral observations and obtain the spectral fidelity term; Attention injection module: used to inject the upsampled version of LRMS image along the channel dimension , PAN image And the low-pass filtered image of the PAN image Cascade, and then inject attention to get the pixel-level attention injection weight map ; High-frequency difference spatial attention module: used to transform PAN images and its low-pass filtered image Input into the differential attention module, calculate its attention matrix respectively, and calculate the high-frequency differential spatial attention according to the attention matrix , the high-frequency differential spatial attention Reshape to get the high-frequency differential spatial attention map , injecting attention into the weight map and high-frequency differential spatial attention map Combined, we get the high-frequency difference spatial attention spatial fidelity term; Mathematical modeling module: used to combine the spectral fidelity term with the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build an overall mathematical optimization model and obtain the final mathematical optimization model; ADMM optimization output module: used to solve the mathematical optimization model through the alternative direction multiplier method ADMM, and update the , , , until the set termination condition is met; the final output result is the high-resolution multi-spectral HRMS image obtained by the system after completing full-color sharpening.

[0036] The above description is only a specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims

1. A full color sharpening method based on high-frequency differential spatial attention mechanism, characterized in that: The method comprises the following steps: Step 1: Based on the high-resolution multispectral HRMS image, the low-resolution multispectral LRMS image is modeled as a spatially degraded version of the high-resolution multispectral HRMS image to obtain a spectral fidelity term; Step 2: Upsample the low-resolution multispectral LRMS image to obtain an upsampled version of the LRMS image, concatenate the upsampled version of the LRMS image, the full-color PAN image, and the low-pass filtered image of the full-color PAN image along the channel dimension, and then obtain a pixel-level attention injection weight map through the attention injection module; Step 3: Input the full-color PAN image and the low-pass filtered image of the full-color PAN image into the differential attention module to calculate the high-frequency differential spatial attention matrix, reshape the high-frequency differential spatial attention matrix, and obtain the high-frequency differential spatial attention map; Step 4: According to the attention injection weight map and the high-frequency difference spatial attention map, a high-frequency difference spatial attention spatial fidelity item is obtained; Step 5, according to the low-rank characteristics of the high-resolution multispectral HRMS image, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term; Step 6: Combine the spectral fidelity term with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism pan-sharpening algorithm; Step 7: Use the alternative direction multiplier method to optimize and solve the mathematical model, and the final result obtained is a high-resolution multispectral HRMS image.

2. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 1, characterized in that: HRMS images are represented as , the LRMS image is represented as , the PAN image is represented as ,in represents the set of real numbers, and are the height and width respectively, is the number of bands of the multispectral image, is the spatial resolution ratio; In step 1, the LRMS image Modeling as HRMS images The spatially degenerate version of Right now ,in represents the spatial downsampling matrix, represents the Gaussian blur matrix, represents zero-mean Gaussian noise, and the obtained spectral fidelity term is: ， in, Represents the matrix The square of the norm.

3. The panchromatic sharpening method based on high-frequency differential spatial attention mechanism according to claim 2, characterized in that: Step 2 The specific steps are as follows: Step 2-1: LRMS image Perform upsampling operation to obtain the upsampled version LRMS image , Step 2-2: PAN image The low-pass filtered image of the PAN image is obtained by low-pass filtering. The low-pass filter is connected in series with two The convolutional layer is learned by introducing the ReLU activation function in the middle, as follows: ， in, express Convolutional layer, ReLU represents the activation function; Step 2-3: Upsample the LRMS image , PAN image And the low-pass filtered image of the PAN image Cascade along the channel dimension to get a tensor ; Step 2-4: Tensor Through a The convolutional layer performs preliminary feature extraction and obtains the corresponding feature map. The convolutional feature map is sent to the ReLU activation function and then passes through another The convolutional layer integrates the features activated by the ReLU activation function to obtain the attention injection weight map ,as follows: 。 4. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 3, characterized in that: Step 3 The specific steps are as follows: Step 3-1: PAN image And low-pass filtered image of PAN image Input into the differential attention module to calculate its attention matrix and get the PAN image The attention matrix , low-pass filtered image of PAN image The attention matrix ,in, for: ， in, , represents a query of the network model, , A key representing a network model, , The transposed matrix of the keys representing the network model, represents the value of the network model, is a learnable scalar; represents the dimension of each branch, Step 3-2: Based on the obtained attention matrix and , calculate high-frequency differential spatial attention , , Step 3-3: Attention to high-frequency differential space Take the average of each row of , calculate the average attention difference, and get a Dimensional vector ,as follows: ， ， in, express The dimension of Representative Row and The difference value of column attention, Indicates The average attention difference of the rows, Step 3-4: Vector Reshape back to the patch mesh, Represents the reshaped matrix, each element Corresponding to the attention difference of the patch in the image, Step 3-5: Reshape the matrix Upsample back to original image size , and obtain the final high-frequency differential spatial attention map .

5. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 4, characterized in that: In step 4, the weight map is injected according to the attention and high-frequency differential spatial attention map , we get the high-frequency difference spatial attention spatial fidelity term, expressed as: ， in, Represents element-wise multiplication.

6. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 5, characterized in that: In step 5, the logarithmic tensor nuclear norm is introduced to obtain the tensor low-rank constraint term: ， in, Represents the logarithmic tensor nuclear norm.

7. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 6, characterized in that: In step 6, the spectral fidelity term is combined with the high-frequency difference spatial attention spatial fidelity term and the tensor low-rank constraint term to build a mathematical model of the high-frequency difference spatial attention mechanism panchromatic sharpening algorithm, and the mathematical model is formulated as: ， ， in, and are two equilibrium parameters; Introducing tensor auxiliary variables , the mathematical optimization model is reformulated as: ， ， in, A tensor auxiliary variable used to assist in iterating HRMS images .

8. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 7, characterized in that: In step 7, the alternative direction multiplier method ADMM is used to optimize and solve the mathematical optimization model. The specific steps are as follows: The augmented Lagrangian function is used to transform the constrained optimization problem into an unconstrained optimization problem. The augmented Lagrangian function is derived as follows: ， in, is the Lagrange multiplier, represents the inner product of the tensor, is the penalty parameter; By initializing, , , The optimization model is iterated alternately in sequence until the set termination condition is met, and the final result is a high-resolution multispectral HRMS image; Alternating iterative update of augmented Lagrangian function ,Right now: 。 9. The method for panchromatic sharpening based on high-frequency differential spatial attention mechanism according to claim 8, characterized in that: Iterative Updates : ， in, Indicates Iterations, Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: ， Iterative Updates : ， Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: ， Iteratively update the Lagrange multiplier , according to the standard practice of augmented Lagrangian method, the multiplier is updated by gradient ascent according to the current degree of constraint violation, Indicated in Solved in iterations Sub-problems, sub-problems It is derived as follows: 。 10. A full-color sharpening system based on a high-frequency differential spatial attention mechanism, used to implement the full-color sharpening method based on a high-frequency differential spatial attention mechanism according to any one of claims 1 to 9, characterized in that: The system comprises Image acquisition module: used to obtain image data from remote sensing satellite data sets, including full-color PAN images and low-resolution multispectral LRMS images, and perform pixel-level registration and geometric correction preprocessing operations on the images to ensure the consistency of the two in sampling area and acquisition time, and obtain a completely matched PAN image and LRMS images , the processed PAN image and LRMS image are used as input data for subsequent models and high-frequency differential spatial attention calculations; Spectral modeling module: used to The spectral degradation relationship of is used to model the spectral fidelity between multi-spectral observations and obtain the spectral fidelity term; Attention injection module: used to inject the upsampled version of LRMS image along the channel dimension , PAN image And the low-pass filtered image of the PAN image Cascade, and then inject attention to get the pixel-level attention injection weight map ; High-frequency difference spatial attention module: used to transform PAN images and its low-pass filtered image Input into the differential attention module, calculate its attention matrix respectively, and calculate the high-frequency differential spatial attention according to the attention matrix , the high-frequency differential spatial attention Reshape to get the high-frequency differential spatial attention map , injecting attention into the weight map and high-frequency differential spatial attention map Combined, we get the high-frequency difference spatial attention spatial fidelity term; Mathematical modeling module: used to combine the spectral fidelity term with the high-frequency differential spatial attention spatial fidelity term and the tensor low-rank constraint term to build an overall mathematical optimization model and obtain the final mathematical optimization model; ADMM optimization output module: used to solve the mathematical optimization model through the alternative direction multiplier method ADMM, and update the , , , until the set termination condition is met; finally, a high-resolution multispectral HRMS image is output.

Images