A remote sensing image pan-sharpening method and system based on two-stage optimization driving high-frequency enhancement

Abstract

The application discloses a remote sensing image pan-sharpening method and system based on double-stage optimization driving high-frequency enhancement, which comprises the following steps: processing original remote sensing images, dividing the original remote sensing images into a training set, a verification set and a test set, and forming a remote sensing image data set; applying mutual information maximization and contrast learning double constraints between different modalities to the images in the remote sensing image data set for double-stage optimization, and constructing a network model based on double-stage optimization driving high-frequency enhancement; training the network model based on double-stage optimization driving high-frequency enhancement by using the training set in the remote sensing image data set, obtaining a trained network model, and the network parameters corresponding to the model are globally optimal network parameters; and obtaining a pan-sharpened remote sensing image by using the trained network model, and completing the pan-sharpening of the remote sensing image. The application realizes accurate detail reconstruction, improves the precision and efficiency of fusion, and has excellent numerical indicators and visual effects on different data sets.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing and multispectral image fusion technology, specifically to a method and system for general sharpening of remote sensing images based on bipolar optimization-driven high-frequency enhancement. Background Technology

[0002] With the improvement of remote sensing technology and satellite image acquisition capabilities, remote sensing image fusion technology has become one of the important research directions in the field of remote sensing image processing. Pan-Sharpening is a widely used technique that combines high-resolution panchromatic remote sensing images with low-resolution multispectral remote sensing images to generate multispectral images with high spatial resolution and high spectral fidelity.

[0003] Traditional multispectral image fusion methods, such as those based on component substitution and multiresolution analysis, encounter a trade-off between spatial and spectral distortion in practical applications. In recent years, advancements in deep learning technology have brought new opportunities for image fusion, especially CNN (Convolutional Neural Networks) and Transformer models, which have demonstrated superior performance in image feature extraction and fusion. However, these methods are often better at capturing low-frequency information, while lacking in the ability to learn high-frequency information. High-frequency reconstruction is a key aspect of generalized sharpening, and various high-frequency enhancement methods have been introduced into this field in recent years. Although existing high-frequency enhancement methods have shown strong performance, they neglect the negative impact of blurred high-frequency details caused by the low resolution of low-resolution multispectral remote sensing images on generalized sharpening performance, thus affecting the overall sharpening effect. Therefore, how to enhance the learning of high-frequency information and achieve accurate multimodal fusion has become a key focus of current research in the field of generalized sharpening. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and system for generalized sharpening of remote sensing images based on dual-level optimization-driven high-frequency enhancement. By applying dual constraints of mutual information maximization and contrastive learning between different modalities to perform bipolar optimization, inaccurate high-frequency information in the generalized sharpening process is calibrated, and accurate high-frequency detail reconstruction is achieved.

[0005] To solve the above technical problems, the present invention adopts the following technical solution:

[0006] A method for general sharpening remote sensing images based on bipolar optimization-driven high-frequency enhancement includes the following steps:

[0007] S1. Process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images. Divide the original remote sensing images and the obtained remote sensing images into training set, validation set and test set to form a remote sensing image dataset.

[0008] S2. By applying dual constraints of mutual information maximization and contrastive learning between different modalities, the images in the remote sensing image dataset are optimized, and a network model based on bipolar optimization-driven high-frequency enhancement is constructed.

[0009] S3. Train a network model based on bipolar optimization-driven high-frequency enhancement using the training set in the remote sensing image dataset to obtain the trained network model. The network parameters corresponding to this model are the globally optimal network parameters. Use the trained network model to obtain the generalized sharpened remote sensing image, thus completing the generalized sharpening of the remote sensing image.

[0010] Furthermore, in step S1, the remote sensing image dataset includes the following:

[0011] S101. Using the corresponding modulation transfer function of the satellite, the original remote sensing image GT∈R is processed. H×W×B Blur processing was performed, and the lowest-nearest-neighbor interpolation method was used to obtain the low-resolution multispectral remote sensing image MS∈R. H / r×W / r×B and high-resolution panchromatic remote sensing image PAN∈R H×W×1 .

[0012] Where H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to that of the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image.

[0013] S102. The low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image are cropped to 64×64×1 and 16×16×8 pixels respectively, and the original remote sensing image is cropped to 64×64×8 pixels.

[0014] S103. Divide the processed remote sensing images into training set, validation set and test set according to a ratio of 7:2:1.

[0015] Furthermore, in step S2, constructing a network model based on bipolar optimization-driven high-frequency enhancement includes the following:

[0016] S201. The network model based on bipolar optimization-driven high-frequency enhancement includes a high-frequency alignment module, a high-frequency calibration module, and a high-frequency injection module. The high-frequency injection module includes a spatial attention module and a channel attention module.

[0017] S202. Using inter23tap interpolation, the low-resolution multispectral remote sensing image is upsampled to the same size as the high-resolution panchromatic remote sensing image. The resulting remote sensing image is denoted as MS. U Using convolution calculation to convert MS U High-resolution panchromatic remote sensing images and high-resolution multispectral remote sensing images are projected into a high-dimensional space to obtain the corresponding high-dimensional features. The specific formula is as follows:

[0018] F G =Conv GT (GT)

[0019] F P =Conv PAN (PAN)

[0020]

[0021] Where Conv(·) represents a convolutional layer, F G F represents the characteristics of high-resolution multispectral remote sensing images. P F represents the characteristics of high-resolution panchromatic remote sensing imagery. M This indicates the characteristics of low-resolution multispectral remote sensing images.

[0022] S203, For F obtained in step S202 P and F M By fusing the data, we obtain the basic feature F. b The specific formula is as follows:

[0023]

[0024] Where Cat(·) indicates concatenation along the channel dimension. This represents the mapping of the residual block.

[0025] S204. Perform wavelet transform on the features obtained in step S202 to extract high-frequency components. The specific formula is as follows:

[0026] DWT(Z) = Z LL Z HL Z LH Z HH

[0027] Z H =Cat(Z) HL Z LH Z HH )

[0028] Where Z represents the feature, Z = F G ,F P ,F M DWT(·) represents Discrete Wavelet Transform, ZLL Z represents the low-frequency component of Z. HL Z LH Z HH These represent the high-frequency components of Z in the horizontal, vertical, and diagonal directions, respectively. H This represents the high-frequency component obtained by splicing three high-frequency components along the channel dimension.

[0029] Then we get F G F P and F M Corresponding high-frequency components and

[0030] S205, to and Perform high-frequency alignment, then... and High-frequency calibration is performed, and the specific details are as follows:

[0031] S2051, will and Simultaneously, the data is input into the high-frequency alignment module, where it is processed through convolutional layers and fully connected layers. and They are respectively transformed into vectors V g and V p ,right and The mutual information value is estimated using the following formula:

[0032]

[0033] MI=D KL (P(V g V p )||P(V g )P(V p ))

[0034] MI′=D JS (P(V g V p )||P(V g )P(V p ))

[0035] Where FC(·) represents a fully connected layer, MI represents the mutual information value, MI′ represents the estimated value of the mutual information, P(·) and P(·,·) represent the marginal probability distribution and the joint probability distribution, respectively, and D KL (·) represents the KL divergence, D JS (·) represents the JS divergence.

[0036] By utilizing the fitting ability of neural networks, the JS divergence is estimated, yielding the estimated JS divergence value. The specific formula is as follows:

[0037]

[0038] in, This represents the estimated value of the JS divergence, where M and N both represent arbitrary random distributions, and m and n represent elements in M ​​and N respectively, i.e., m∈M, n∈N, E p(·) Let θ represent the expected value, and let θ represent the neural network T. θ Optimized network parameters.

[0039] Based on the JS divergence estimate, the mutual information loss function L is used. MI Optimize V g With V p The mutual information value between them, and the expression for the mutual information loss function are:

[0040]

[0041] S2052, will and Simultaneously, the input is fused into the high-frequency calibration module to obtain the first fused feature F. merge The specific formula is as follows:

[0042]

[0043] structure and The formula for positive and negative sample pairs is as follows:

[0044]

[0045] Among them, F + F represents a positive sample. - This represents a negative sample, and Norm(·) represents normalization.

[0046] Utilizing the idea of ​​contrastive learning, combined with the contrastive learning loss function L CL , for F merge The high-frequency information is calibrated to obtain the multimodal high-frequency feature F. h The expression for the contrastive learning loss function is:

[0047]

[0048] Where ||·||1 represents the L1 norm.

[0049] S206, F b With F h The difference, i.e., F b -F hThe input is fed into the high-frequency injection module, passes through the spatial attention module, and obtains the corresponding features through convolution. These features are then subjected to global average pooling and global max pooling, concatenated along the channel dimension, and finally processed by convolution and the sigmoid activation function to generate a spatial attention map. The specific formula is as follows:

[0050] SA(F b -F h )=Sigmoid(Conv(Cat(GMP(Conv(F b -F h )),GAP(Conv(F b -F h )))))

[0051] Where SA(·) represents the spatial attention map, GMP(·) represents global max pooling, GAP(·) represents global average pooling, and Sigmoid(·) represents the Sigmoid activation function.

[0052] Multiply the obtained spatial attention map by F h The features of the texture details are obtained, and the texture details are connected with F through residual connection. b Combined The specific formula is as follows:

[0053]

[0054] Will and F h The input features are concatenated along the channel dimension, then processed sequentially through a channel attention module, including a first convolution, global average pooling, a second convolutional block, and a sigmoid activation function, to generate a channel attention map. This channel attention map is then used to weight the input features, followed by convolution to obtain the output of the channel attention module. The specific formula is as follows:

[0055]

[0056] Where CA(·) represents the output of the channel attention module.

[0057] Compare the output of the channel attention module with F b The addition yields the second fusion feature F. i The specific formula is as follows:

[0058]

[0059] S207. Repeat steps S205-S206 until the set number of repetitions T is reached, and stop. Then, record the L calculated in each step of S205. MI and L CL Output.

[0060] Furthermore, in step S3, the obtained generalized sharpened remote sensing image includes the following:

[0061] S301. The low-resolution multispectral remote sensing image with a size of 16×16×8 pixels in the training set is interpolated using inter23tap to obtain an upsampled low-resolution multispectral remote sensing image LRMS∈R with a size of 64×64×8. H×W×B The LRMS and a high-resolution panchromatic remote sensing image of size 64×64×1 pixels from the training set are simultaneously input into a network model based on bipolar optimization-driven high-frequency enhancement to obtain a high-resolution multispectral remote sensing image X∈R of size 64×64×8 pixels. H ×W×B .

[0062] S302. Calculate the difference between the high-resolution multispectral remote sensing image and the original remote sensing image using a loss function. The expression for the loss function is:

[0063]

[0064] Among them, L total L represents the loss function; rec L represents the reconstruction loss function. rec =||X-GT||1;L str L represents the structural loss function. str =||1-SSIM(X,GT)||1; SSIM(·,·) represents the structural similarity index; i represents the i-th repetition.

[0065] S303. Calculate the gradient of network parameters in the bipolar optimization-driven high-frequency augmentation network model using backpropagation of the loss function, and update the parameters. The specific formula is as follows:

[0066]

[0067] Where θ′ represents the updated θ, and lr represents the hyperparameter learning rate. This represents the gradient of the network parameters.

[0068] S304. Repeat steps S301-S303 until the loss function converges. At this point, training is complete, and a trained network model based on bipolar optimization-driven high-frequency enhancement is obtained.

[0069] S305. Input the test set from the remote sensing image dataset into the trained network model based on bipolar optimization-driven high-frequency enhancement to obtain the generalized sharpened remote sensing image Y∈R. H×W×B .

[0070] Furthermore, step S4 is included: evaluating the quality of the oversharpened remote sensing image using evaluation metrics.

[0071] Furthermore, in step S4, evaluating the quality of the remote sensing image includes the following:

[0072] (1) The generalized sharpened remote sensing image is evaluated using the spectral angle mapping index, the relative global error index, the spatial correlation coefficient (SCC), and the Q index. The specific calculations are as follows:

[0073] 1) Spectral angle mapping index:

[0074] The angle between corresponding pixels in the generalized sharpened remote sensing image and the original remote sensing image is calculated. This metric treats each spectral channel as a coordinate axis to calculate the phase angle, setting Y... {k} =[Y {k},1 ,…,Y {k},B Let ] be the k-th pixel vector of a generalized sharpened remote sensing image with B spectral channels, and GT {k} =[GT {k},1 ,…,GT {k},B Let Y be the k-th pixel vector of the original remote sensing image with B spectral channels. {k} and GT {k} The SAM index between them is:

[0075]

[0076] Where arccos(·) represents the inverse cosine function, <·,·> represents the inner product, ||·|| represents the l2 norm, and Y represents the generalized sharpened remote sensing image.

[0077] The global SAM value of the entire remote sensing image is obtained by averaging the SAM indices of all pixel vectors.

[0078] 2) ERGAS indicator:

[0079]

[0080] Mean() represents the inner product.

[0081]

[0082] Among them, GT b Y represents the original remote sensing image of the b-th spectral channel. b This represents the generalized sharpened remote sensing image of the b-th spectral channel.

[0083] 3) The spatial correlation coefficient (SCC) is used to describe the degree of geometric deformation. Its specific expression is:

[0084]

[0085] Where SCC represents the spatial correlation coefficient; CCS represents the cross-correlation between individual remote sensing images.

[0086] 4) Q indicator:

[0087] Using the Q4 indicator to measure Y {k} The model is based on quaternions, and the specific expression is:

[0088] Y {k} =Y+k1Y {k},2 +k2Y {k},3 +k3Y {k},4

[0089] Where k1, k2, and k3 represent the linear combination coefficients of the corresponding channels.

[0090] The formula for calculating the Q4 indicator is:

[0091]

[0092] in, and They represent GT respectively {k} and Y {k} The mean, GT {k} and Y {k} covariance, and They represent GT respectively {k} and Y {k} The standard deviation.

[0093] The Q4 indicator is further extended to cases where the number of channels is greater than 4, resulting in the Q indicator.

[0094] (2) Use the no-reference quality assessment index QNR and the spectral distortion index D λ and spatial distortion index D s The evaluation of the over-sharpened remote sensing image is performed using the following formula:

[0095] QNR=(1-D λ ) α (1-D S ) β

[0096]

[0097]

[0098] Where α and β both represent coefficients; d b,c(MS,Y) represents the similarity between the low-resolution multispectral remote sensing image and the oversharpened remote sensing image, d b,c (MS,Y)=Q(MS b ,MS)-Q(Y b ,Y c MS b Y represents the low-resolution multispectral remote sensing image of the b-th spectral channel. c This represents the generalized sharpened remote sensing image of the c-th spectral channel, where Q() represents the Q index; p represents D. λ The order of the square root of D; q represents the order of the square root of D. S The order of the square root of P; LP This represents the low-pass portion of a high-resolution panchromatic remote sensing image.

[0099] Furthermore, this invention also proposes a remote sensing image general sharpening system based on two-level optimization-driven high-frequency enhancement, comprising:

[0100] The dataset construction module is used to process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images. The original remote sensing images and the obtained remote sensing images are divided into training set, validation set and test set to form a remote sensing image dataset.

[0101] The network model building module is used to optimize images in the remote sensing image dataset by applying dual constraints of mutual information maximization and contrastive learning between different modalities, and to build a network model based on bipolar optimization-driven high-frequency enhancement.

[0102] The generalized sharpening module for remote sensing images is used to train a network model based on bipolar optimization-driven high-frequency enhancement using the training set in the remote sensing image dataset. The trained network model has globally optimal network parameters. The trained network model is then used to obtain the generalized sharpened remote sensing image, thus completing the generalized sharpening of the remote sensing image.

[0103] The quality assessment module is used to evaluate the quality of the over-sharpened remote sensing images using evaluation metrics.

[0104] Furthermore, the present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the remote sensing image generalization sharpening method based on dual-level optimization-driven high-frequency enhancement.

[0105] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the aforementioned method for generalized sharpening of remote sensing images based on two-level optimization-driven high-frequency enhancement.

[0106] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0107] (1) The method proposed in this invention makes full use of the guidance role of the original remote sensing image GT in the dataset, and enhances the consistency between high-resolution panchromatic features and GT features by introducing a high-frequency alignment module and maximizing mutual information. Since high-resolution panchromatic features are usually used to guide the super-resolution process of low-resolution multispectral images, high-frequency alignment can ensure that the fused image obtains more accurate high-frequency information.

[0108] (2) This invention fully considers the inaccurate high-frequency details in low-resolution multispectral images. By designing a high-frequency calibration module to apply contrast learning constraints, and using the high-resolution panchromatic features optimized by the high-frequency alignment module as guidance, the high-frequency information in the multimodal fusion process is calibrated, resulting in a high-resolution multispectral image with accurate spatial details and rich spectral information. Attached Figure Description

[0109] Figure 1 This is a flowchart illustrating the overall implementation of the present invention.

[0110] Figure 2 This is a schematic diagram of a low-resolution multispectral, high-resolution panchromatic input image and a high-resolution multispectral output image according to an embodiment of the present invention.

[0111] Figure 3 This is a schematic diagram of the model architecture for bipolar optimization-driven high-frequency enhancement according to an embodiment of the present invention.

[0112] Figure 4 This is a schematic diagram of the spatial attention module and the channel attention module according to an embodiment of the present invention.

[0113] Figure 5 This is a schematic diagram of the generalized sharpened remote sensing image obtained according to an embodiment of the present invention. Detailed Implementation

[0114] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0115] To achieve the above objectives, this invention proposes a method for generalized sharpening of remote sensing images based on bipolar optimization-driven high-frequency enhancement, such as... Figure 1 As shown, the specific steps are as follows:

[0116] S1. Process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images. Divide the original and obtained remote sensing images into training, validation, and test sets to form a remote sensing image dataset. Specific content includes:

[0117] S101. Using the corresponding modulation transfer function of the satellite, the original remote sensing image GT∈R is processed. H×W×B Blur processing was performed, and the lowest-nearest-neighbor interpolation method was used to obtain the low-resolution multispectral remote sensing image MS∈R. H / r×W / r×B and high-resolution panchromatic remote sensing image PAN∈R H×W×1 Specific remote sensing images such as Figure 2 As shown.

[0118] Where H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to that of the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image.

[0119] S102. The low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image are cropped to 64×64×1 and 16×16×8 pixels respectively, and used as network input; the original remote sensing image is cropped to 64×64×8 pixels and used as the reference image for the model to evaluate the quality of the generated image.

[0120] S103. Divide the processed remote sensing images into training set, validation set and test set according to a ratio of 7:2:1.

[0121] In this embodiment, the value of r is 4.

[0122] S2. By applying dual constraints of maximizing mutual information and contrastive learning across different modalities, the images in the remote sensing image dataset are optimized, and a network model based on bipolar optimization-driven high-frequency enhancement is constructed; the specific content is as follows:

[0123] S201, such as Figure 3 As shown, the network model based on bipolar optimization-driven high-frequency enhancement includes a high-frequency alignment module, a high-frequency calibration module, and a high-frequency injection module. The high-frequency injection module includes a spatial attention module and a channel attention module. This model is used to solve the generalized sharpening task and outputs high-resolution multispectral images.

[0124] S202. Using inter23tap interpolation, the low-resolution multispectral remote sensing image is upsampled to the same size as the high-resolution panchromatic remote sensing image. The resulting remote sensing image is denoted as MS. U Using convolution calculation to convert MS U High-resolution panchromatic remote sensing images and high-resolution multispectral remote sensing images are projected into a high-dimensional space to obtain the corresponding high-dimensional features. The specific formula is as follows:

[0125] F G =Conv GT (GT)

[0126] FP =Conv PAN (PAN)

[0127]

[0128] Where Conv(·) represents a convolutional layer, F G F represents the characteristics of high-resolution multispectral remote sensing images. P F represents the characteristics of high-resolution panchromatic remote sensing imagery. M This indicates the characteristics of low-resolution multispectral remote sensing images.

[0129] S203, For F obtained in step S202 P and F M By fusing the data, we obtain the basic feature F. b The specific formula is as follows:

[0130]

[0131] Where Cat(·) indicates concatenation along the channel dimension. This represents the mapping of the residual block.

[0132] S204. Perform wavelet transform on the features obtained in step S202 to extract high-frequency components. The specific formula is as follows:

[0133] DWT(Z) = Z LL Z HL Z LH Z HH

[0134] Z H =Cat(Z) HL Z LH Z HH )

[0135] Where Z represents the feature, Z = F G ,F P ,F M DWT(·) represents Discrete Wavelet Transform, Z LL Z represents the low-frequency component of Z. HL Z LH Z HH These represent the high-frequency components of Z in the horizontal, vertical, and diagonal directions, respectively. H This represents the high-frequency component obtained by splicing three high-frequency components along the channel dimension.

[0136] Then we get F G F P and F M Corresponding high-frequency components and

[0137] S205, to and Perform high-frequency alignment, then... and High-frequency calibration is performed, and the specific details are as follows:

[0138] (1) and Simultaneously input into the high-frequency alignment module, so that and Maximizing mutual information by utilizing mutual information loss The distribution is optimized to match To make them closer together, so that they tend to align.

[0139] Through convolutional layers and fully connected layers and They are respectively transformed into vectors V g and V p ,right and The mutual information value is estimated using the following formula:

[0140]

[0141] MI=D KL (P(V g V p )||P(V g )P(V p ))

[0142] MI′=D JS (P(V g V p )||P(V g )P(V p ))

[0143] Where FC(·) represents a fully connected layer, MI represents the mutual information value, MI′ represents the estimated value of the mutual information, P(·) and P(·,·) represent the marginal probability distribution and the joint probability distribution, respectively, and D KL (·) represents the KL divergence, D JS (·) represents the JS divergence.

[0144] Due to its advantages of symmetry and ease of optimization, JS divergence is used to replace KL divergence to represent the estimated value of mutual information.

[0145] By utilizing the fitting ability of neural networks, the JS divergence is estimated, yielding the estimated JS divergence value. The specific formula is as follows:

[0146]

[0147] in, This represents the estimated value of the JS divergence, where M and N both represent arbitrary random distributions, and m and n represent elements in M ​​and N respectively, i.e., m∈M, n∈N, E p(·) Let θ represent the expected value, and let θ represent the neural network T. θ Optimized network parameters.

[0148] Based on the estimation of JS divergence, the JS divergence estimate is used to replace the mutual information estimate MI′ in the expression. Since the goal is to maximize mutual information, a negative sign is added to obtain the mutual information loss function L. MI Used to optimize V g With V p The mutual information value between them, and the expression for the mutual information loss function are:

[0149]

[0150] (2) After high-frequency alignment Distribution and They will gradually converge. and Simultaneously, the input is fused into the high-frequency calibration module to obtain the first fused feature F. merge The specific formula is as follows:

[0151]

[0152] structure and The formula for positive and negative sample pairs is as follows:

[0153]

[0154] Among them, F + F represents a positive sample. - This represents a negative sample, and Norm(·) represents normalization.

[0155] Utilizing the idea of ​​contrastive learning, combined with the contrastive learning loss function L CL , for F merge The high-frequency information is calibrated to obtain the multimodal high-frequency feature F. h Specifically, it refers to proximity within the feature space. give up Zhongyu Inconsistent information. The expression for the contrastive learning loss function is:

[0156]

[0157] Where ||·||1 represents the L1 norm.

[0158] S206, F h and F b Simultaneously, the input is fed into the high-frequency injection module, where spatial attention and channel attention modules are used to achieve F. h High-efficiency injection of F b .

[0159] (1) Spatial attention module: F b With F h The difference, i.e., F b -F h Injecting spatial attention modules, such as Figure 4 As shown, the corresponding features are first obtained through convolution. Global average pooling and global max pooling are then applied to these features to aggregate spatial information. The data is then concatenated along the channel dimension and finally processed by convolution and the sigmoid activation function to generate a spatial attention map. The specific formula is as follows:

[0160] SA(F b -F h )=Sigmoid(Conv(Cat(GMP(Conv(F b -F h )),GAP(Conv(F b -F h )))))

[0161] Where SA(·) represents the spatial attention map, GMP(·) represents global max pooling, GAP(·) represents global average pooling, and Sigmoid(·) represents the Sigmoid activation function.

[0162] The obtained spatial attention map SA(F) b -F h Multiply by F h To perform high-frequency texture information compensation, features with richer texture details are obtained. The result of high-frequency texture information compensation is then connected with F through residual connection. b Combined The specific formula is as follows:

[0163]

[0164] (2) Channel attention module: and F h The data is concatenated along the channel dimension and then input into the channel attention module, such as... Figure 4As shown, the input features are processed sequentially through the first convolution, global average pooling, the second convolutional block, and the sigmoid activation function to generate a channel attention map. This channel attention map is then used to weight the input features, producing a more informative feature representation. Finally, convolution is used to further refine the output of the channel attention module. The specific formula is as follows:

[0165]

[0166] Where CA(·) represents the output of the channel attention module.

[0167] Compare the output of the channel attention module with F b The addition yields the second fusion feature F. i The specific formula is as follows:

[0168]

[0169] S207. Repeat steps S205-S206 until the set number of repetitions T is reached, and stop. Then, record the L calculated in each step of S205. MI and L CL Output.

[0170] S3. Train a network model based on bipolar optimization-driven high-frequency enhancement using the training set in the remote sensing image dataset to obtain the trained network model. The network parameters of this model are globally optimal. Use the trained network model to obtain the generalized sharpened remote sensing image, thus completing the generalized sharpening of the remote sensing image. The specific content is as follows:

[0171] S301. The low-resolution multispectral remote sensing image with a size of 16×16×8 pixels in the training set is interpolated using inter23tap to obtain an upsampled low-resolution multispectral remote sensing image LRMS∈R with a size of 64×64×8. H×W×B The LRMS and a high-resolution panchromatic remote sensing image of size 64×64×1 pixels from the training set are simultaneously input into a network model based on bipolar optimization-driven high-frequency enhancement to obtain a high-resolution multispectral remote sensing image X∈R of size 64×64×8 pixels. H ×W×B .

[0172] S302. Calculate the difference between the high-resolution multispectral remote sensing image and the original remote sensing image using a loss function. The expression for the loss function is:

[0173]

[0174] Among them, L total L represents the loss function; rec L represents the reconstruction loss function.rec =||X-GT||1;L str L represents the structural loss function. str =||1-SSIM(X,GT)||1; SSIM(·,·) represents the structural similarity index; i represents the i-th repetition.

[0175] S303. Calculate the gradient of network parameters in the bipolar optimization-driven high-frequency augmentation network model using backpropagation of the loss function, and update the parameters. The specific formula is as follows:

[0176]

[0177] Where θ′ represents the updated θ, and lr represents the hyperparameter learning rate. This represents the gradient of the network parameters.

[0178] S304. Repeat steps S301-S303 until the loss function converges, i.e., the number of iterations reaches the hyperparameter I. MAX At this point, training is complete, and a trained network model based on bipolar optimization-driven high-frequency enhancement is obtained.

[0179] S305. Input the test set from the remote sensing image dataset into the trained network model based on bipolar optimization-driven high-frequency enhancement to obtain the generalized sharpened remote sensing image Y∈R. H×W×B .

[0180] S4. Evaluate the quality of the over-sharpened remote sensing image using evaluation metrics. The specific content is as follows:

[0181] (1) The SAM (Spectral Angle Mapper) index, ERGAS (ErreurRelative Globale Adimensionnellede Synthèse) index, spatial correlation coefficient SCC, and Q index were used to evaluate the generalized sharpened remote sensing image. The specific calculations are as follows:

[0182] 1) Spectral angle mapping index:

[0183] The angle between corresponding pixels in the generalized sharpened remote sensing image and the original remote sensing image is calculated. This metric treats each spectral channel as a coordinate axis to calculate the phase angle, setting Y... {k} =[Y {k},1 ,…,Y {k},B Let ] be the k-th pixel vector of a generalized sharpened remote sensing image with B spectral channels, and GT {k} =[GT {k},1 ,…,GT {k},B Let Y be the k-th pixel vector of the original remote sensing image with B spectral channels.{k} and GT {k} The SAM index between them is:

[0184]

[0185] Where arccos(·) represents the inverse cosine function, <·,·> represents the inner product, ||·|| represents the l2 norm, and Y represents the generalized sharpened remote sensing image.

[0186] The global SAM value of the entire remote sensing image is obtained by averaging the SAM index of all pixel vectors; the optimal value of the SAM index is 0.

[0187] 2) ERGAS indicator:

[0188]

[0189] Mean() represents the inner product.

[0190] RMSE (Root Mean Square Error) is a common metric for evaluating the error between two graphs. It reaches its optimal value of 0 if and only if the two graphs are completely equal.

[0191]

[0192] Among them, GT b Y represents the original remote sensing image of the b-th spectral channel. b denoted as the generalized sharpened remote sensing image of the b-th spectral channel, and r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to that of the low-resolution multispectral remote sensing image.

[0193] ERGAS is a multimodal index that takes into account the relative errors of all channels, and its optimal value is 0.

[0194] 3) The spatial correlation coefficient (SCC) is used to describe the degree of geometric deformation. Its specific expression is:

[0195]

[0196] Where SCC represents the spatial correlation coefficient; CCS represents the cross-correlation between individual remote sensing images.

[0197] 4) Q indicator:

[0198] Q2 m It is a multi-band extension of the general remote sensing image quality index, using the Q4 index to adjust the Y band. {k} The model is based on quaternions, and the specific expression is:

[0199] Y{k} =Y+k1Y {k},2 +k2Y {k},3 +k3Y {k},4

[0200] Where k1, k2, and k3 represent the linear combination coefficients of the corresponding channels.

[0201] The formula for calculating the Q4 indicator is:

[0202]

[0203] in, and They represent GT respectively {k} and Y {k} The mean, GT {k} and Y {k} covariance, and They represent GT respectively {k} and Y {k} The standard deviation.

[0204] The Q4 indicator is further extended to cases where the number of channels is greater than 4, resulting in the Q indicator.

[0205] (2) Use the no-reference quality assessment index QNR and the spectral distortion index D λ and spatial distortion index D s The evaluation of the over-sharpened remote sensing image is performed using the following formula:

[0206] QNR=(1-D λ ) α (1-D S ) β

[0207]

[0208] Where α and β both represent coefficients; d b,c (MS,Y) represents the similarity between the low-resolution multispectral remote sensing image and the oversharpened remote sensing image, d b,c (MS,Y)=Q(MS b ,MS)-Q(Y b ,Y c MS b Y represents the low-resolution multispectral remote sensing image of the b-th spectral channel. c This represents the generalized sharpened remote sensing image of the c-th spectral channel, where Q() represents the Q index; p represents D. λ The order of the square root is set to 1; q represents D. S The order of the square root is set to 1; P LPThis represents the low-pass portion of a high-resolution panchromatic remote sensing image.

[0209] A higher QNR index indicates better fusion quality; when D λ and D s When all values ​​are equal to 0, the QNR index reaches its maximum theoretical value of 1.

[0210] The average value of each evaluation indicator is: QNR / D λ / D S =0.9487 / 0.0175 / 0.0344.

[0211] In addition, a comparison of visual effects was made, such as Figure 5 As shown in the figure, the experimental results of the proposed method and other general sharpening methods on the WorldView-3 real dataset are presented. It can be seen from the figure that the network model based on bipolar optimization-driven high-frequency enhancement designed by the present invention exhibits better visual effects than other general sharpening networks.

[0212] Through the above embodiments and simulation results, this invention achieves efficient and accurate multispectral image fusion by using a bipolar optimization-driven high-frequency enhancement model, outperforming existing methods in multiple evaluation metrics. Experimental results demonstrate that this invention exhibits excellent performance in spectral fidelity, spatial resolution, and computational efficiency.

[0213] This invention also proposes a remote sensing image general sharpening system based on bipolar optimization-driven high-frequency enhancement, including a dataset construction module, a network model construction module, a remote sensing image general sharpening module, a quality assessment module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.

[0214] This invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, it corresponds to the specific steps of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0215] This invention also proposes a computer-readable storage medium storing a computer program. It should be noted that when the computer program is executed by a processor, it corresponds to the specific steps of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0216] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for general sharpening remote sensing images based on two-level optimization-driven high-frequency enhancement, characterized in that, include: S1. Process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images. Divide the original remote sensing images and the obtained remote sensing images into training set, validation set and test set to form a remote sensing image dataset. S2. By applying dual constraints of mutual information maximization and contrastive learning across different modalities, the images in the remote sensing image dataset are optimized, and a network model based on bipolar optimization-driven high-frequency enhancement is constructed; specifically: S201. The network model based on bipolar optimization-driven high-frequency enhancement includes a high-frequency alignment module, a high-frequency calibration module, and a high-frequency injection module. The high-frequency injection module includes a spatial attention module and a channel attention module. S202. Using inter23tap interpolation, the low-resolution multispectral remote sensing image is upsampled to the same size as the high-resolution panchromatic remote sensing image. The resulting remote sensing image is denoted as MS. U Using convolution calculation to convert MS U High-resolution panchromatic remote sensing images and high-resolution multispectral remote sensing images are projected into a high-dimensional space to obtain the corresponding high-dimensional features. The specific formula is as follows: F G =Conv GT (GT) F P =Conv PAN (PAN) F M =Conv MSU (MS U ) Where Conv(·) represents a convolutional layer, F G F represents the characteristics of high-resolution multispectral remote sensing images. P F represents the characteristics of high-resolution panchromatic remote sensing imagery. M GT represents the characteristics of low-resolution multispectral remote sensing images; GT represents the original remote sensing image; PAN represents the high-resolution panchromatic remote sensing image. S203, For F obtained in step S202 P and F M By fusing the data, we obtain the basic feature F. b The specific formula is as follows: Where Cat(·) indicates concatenation along the channel dimension. Represents the mapping of residual blocks; S204. Perform wavelet transform on the features obtained in step S202 to extract high-frequency components. The specific formula is as follows: DWT(Z)=Z LL ,WITH HL ,WITH LH ,WITH HH Z H =Cat(Z HL ,Z LH ,Z HH ) Where Z represents the feature, Z = F G ,F P ,F M DWT(·) represents Discrete Wavelet Transform, Z LL Z represents the low-frequency component of Z. HL Z LH Z HH These represent the high-frequency components of Z in the horizontal, vertical, and diagonal directions, respectively. H This represents the high-frequency component obtained by splicing three high-frequency components along the channel dimension. Then we get F G F P and F M Corresponding high-frequency components and S205, to and Perform high-frequency alignment, then... and High-frequency calibration is performed, and the specific details are as follows: S2051, will and Simultaneously, the data is input into the high-frequency alignment module, where it is processed through convolutional layers and fully connected layers. and They are respectively transformed into vectors V g and V p ,right and The mutual information value is estimated using the following formula: MI=D KL (P(V g ,V p )||P(V g )P(V p )) MI′=D JS (P(V g ,V p )||P(V g )P(V p )) Where FC(·) represents a fully connected layer, MI represents the mutual information value, MI′ represents the estimated value of the mutual information, P(·) and P(·,·) represent the marginal probability distribution and the joint probability distribution, respectively, and D KL (·) represents the KL divergence, D JS (·) represents the JS divergence; By utilizing the fitting ability of neural networks, the JS divergence is estimated, yielding the estimated JS divergence value. The specific formula is as follows: in, This represents the estimated value of the JS divergence, where M and N both represent arbitrary random distributions, and m and n represent elements in M ​​and N respectively, i.e., m∈M, n∈N, E p(·) Let θ represent the expected value, and let θ represent the neural network T. θ Optimized network parameters; Based on the JS divergence estimate, the mutual information loss function L is used. MI Optimize V g With V p The mutual information value between them, and the expression for the mutual information loss function are: S2052, will and Simultaneously, the input is fused into the high-frequency calibration module to obtain the first fused feature F. merge The specific formula is as follows: structure and The formula for positive and negative sample pairs is as follows: Among them, F + F represents a positive sample. - This represents a negative sample, and Norm(·) represents normalization. Utilizing the idea of ​​contrastive learning, combined with the contrastive learning loss function L CL , for F merge The high-frequency information is calibrated to obtain the multimodal high-frequency feature F. h The expression for the contrastive learning loss function is: Where ||·||1 represents the L1 norm; S206, F b With F h The difference, i.e., F b -F h The input is fed into the high-frequency injection module, passes through the spatial attention module, and obtains the corresponding features through convolution. These features are then subjected to global average pooling and global max pooling, concatenated along the channel dimension, and finally processed by convolution and the sigmoid activation function to generate a spatial attention map. The specific formula is as follows: SA(F b -F h )=Sigmoid(Conv(Cat(GMP(Conv(F b -F h )),GAP(Conv(F b -F h ))))) Where SA(·) represents the spatial attention map, GMP(·) represents global max pooling, GAP(·) represents global average pooling, and Sigmoid(·) represents the Sigmoid activation function; Multiply the obtained spatial attention map by F h The features of the texture details are obtained, and the texture details are connected with F through residual connection. b Combined The specific formula is as follows: Will and F h The input features are concatenated along the channel dimension, then processed sequentially through a channel attention module, including a first convolution, global average pooling, a second convolutional block, and a sigmoid activation function, to generate a channel attention map. This channel attention map is then used to weight the input features, followed by convolution to obtain the output of the channel attention module. The specific formula is as follows: Where CA(·) represents the output of the channel attention module; Compare the output of the channel attention module with F b The addition yields the second fusion feature F. i The specific formula is as follows: S207. Repeat steps S205-S206 until the set number of repetitions T is reached, and stop. Then, record the L calculated in each step of S205. MI and L CL Output; S3. Train a network model based on bipolar optimization-driven high-frequency enhancement using the training set in the remote sensing image dataset to obtain the trained network model. The network parameters corresponding to this model are the globally optimal network parameters. Use the trained network model to obtain the generalized sharpened remote sensing image, thus completing the generalized sharpening of the remote sensing image.

2. The remote sensing image generalization sharpening method based on dual-level optimization-driven high-frequency enhancement according to claim 1, characterized in that, In step S1, the remote sensing image dataset includes the following: S101. Using the corresponding modulation transfer function of the satellite, the original remote sensing image GT∈R is processed. H×W×B Blur processing was performed, and the lowest-nearest-neighbor interpolation method was used to obtain the low-resolution multispectral remote sensing image MS∈R. H / r×W / r×B and high-resolution panchromatic remote sensing image PAN∈R H ×W×1 ; Where H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to that of the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image. S102. The low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image are cropped to 64×64×1 and 16×16×8 respectively, and the original remote sensing image is cropped to 64×64×8 pixels. S103. Divide the processed remote sensing images into training set, validation set and test set according to a ratio of 7:2:

1.

3. The remote sensing image generalization sharpening method based on dual-level optimization-driven high-frequency enhancement according to claim 1, characterized in that, In step S3, the obtained generalized sharpened remote sensing image includes the following: S301. The low-resolution multispectral remote sensing image with a size of 16×16×8 pixels in the training set is interpolated using inter23tap to obtain an upsampled low-resolution multispectral remote sensing image LRMS∈R with a size of 64×64×8. H×W×B The LRMS and a high-resolution panchromatic remote sensing image of size 64×64×1 pixels from the training set are simultaneously input into a network model based on bipolar optimization-driven high-frequency enhancement to obtain a high-resolution multispectral remote sensing image X∈R of size 64×64×8 pixels. H×W×B ; Where H represents the height of the remote sensing image, W represents the width of the remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image. S302. Calculate the difference between the high-resolution multispectral remote sensing image and the original remote sensing image using a loss function. The expression for the loss function is: Among them, L total L represents the loss function; rec L represents the reconstruction loss function. rec =||X-GT||1;L str L represents the structural loss function. str = ||1-SSIM(X,GT)||1; SSIM(·,·) represents the structural similarity index; i represents the i-th repetition; S303. Calculate the gradient of network parameters in the bipolar optimization-driven high-frequency augmentation network model using backpropagation of the loss function, and update the parameters. The specific formula is as follows: Where θ′ represents the updated θ, and θ represents the neural network T θ In the optimized network parameters, lr represents the hyperparameter learning rate. The gradient represents the network parameters; S304. Repeat steps S301-S303 until the loss function converges. At this point, training is complete, and a trained network model based on bipolar optimization-driven high-frequency enhancement is obtained. S305. Input the test set from the remote sensing image dataset into the trained network model based on bipolar optimization-driven high-frequency enhancement to obtain the generalized sharpened remote sensing image Y∈R. H×W×B .

4. The remote sensing image generalization sharpening method based on dual-level optimization-driven high-frequency enhancement according to claim 1, characterized in that, It also includes step S4: using evaluation metrics to assess the quality of the oversharpened remote sensing image.

5. The remote sensing image generalization sharpening method based on dual-level optimization-driven high-frequency enhancement according to claim 4, characterized in that, In step S4, the quality assessment of the remote sensing image includes the following: (1) The generalized sharpened remote sensing image is evaluated using the spectral angle mapping index, the relative global error index, the spatial correlation coefficient (SCC), and the Q index. The specific calculations are as follows: 1) Spectral angle mapping index: The angle between corresponding pixels in the generalized sharpened remote sensing image and the original remote sensing image is calculated. This metric treats each spectral channel as a coordinate axis to calculate the phase angle, setting Y... {k} =[Y {k} , },1 ,…,Y {k},B Let ] be the k-th pixel vector of a generalized sharpened remote sensing image with B spectral channels, and GT {k} =[GT {k},1 ,…,GT {k},B Let Y be the k-th pixel vector of the original remote sensing image with B spectral channels. {k} and GT {k} The SAM index between them is: Where arccos(·) represents the inverse cosine function, <·,·> represents the inner product, ||·|| represents the l2 norm, and Y represents the generalized sharpened remote sensing image; The global SAM value of the entire remote sensing image is obtained by averaging the SAM indexes of all pixel vectors. 2) ERGAS indicator: Where Mean() represents the inner product; Among them, GT b Y represents the original remote sensing image of the b-th spectral channel. b denoted as the generalized sharpened remote sensing image of the b-th spectral channel, and r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to that of the low-resolution multispectral remote sensing image. 3) The spatial correlation coefficient (SCC) is used to describe the degree of geometric deformation. Its specific expression is: Where SCC represents the spatial correlation coefficient; CCS represents the cross-correlation between individual remote sensing images. 4) Q indicator: Using the Q4 indicator to measure Y {k} The model is based on quaternions, and the specific expression is: AND {k} =Y+k1Y {k},2 +k2Y {k},3 +k3Y {k},4 Where k1, k2, and k3 represent the linear combination coefficients of the corresponding channels; The formula for calculating the Q4 indicator is: in, and They represent GT respectively {k} and Y {k} The mean, GT {k} and Y {k} covariance, and They represent GT respectively {k} and Y {k} Standard deviation; The Q4 indicator is further extended to cases where the number of channels is greater than 4, resulting in the Q indicator; (2) Use the no-reference quality assessment index QNR and the spectral distortion index D λ and spatial distortion index D s The evaluation of the over-sharpened remote sensing image is performed using the following formula: QNR=(1-D λ ) α (1-D S ) β Where α and β both represent coefficients; d b,c (MS,Y) represents the similarity between the low-resolution multispectral remote sensing image and the oversharpened remote sensing image, d b,c (MS,Y)=Q(MS b ,MS)-Q(Y b ,Y c MS b Y represents the low-resolution multispectral remote sensing image of the b-th spectral channel. c This represents the generalized sharpened remote sensing image of the c-th spectral channel, where Q() represents the Q index; p represents D. λ The order of the square root of D; q represents the order of the square root of D. S The order of the square root of P; LP This represents the low-pass portion of a high-resolution panchromatic remote sensing image.

6. A system applied to the remote sensing image generalization sharpening method based on two-level optimization-driven high-frequency enhancement as described in claim 1, characterized in that, include: The dataset construction module is used to process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images. The original remote sensing images and the obtained remote sensing images are divided into training set, validation set and test set to form a remote sensing image dataset. The network model building module is used to optimize images in the remote sensing image dataset by applying dual constraints of mutual information maximization and contrastive learning between different modalities, and to build a network model based on bipolar optimization-driven high-frequency enhancement. The generalized sharpening module for remote sensing images is used to train a network model based on bipolar optimization-driven high-frequency enhancement using the training set in the remote sensing image dataset. The trained network model has globally optimal network parameters. The trained network model is then used to obtain the generalized sharpened remote sensing image, thus completing the generalized sharpening of the remote sensing image. The quality assessment module is used to evaluate the quality of the over-sharpened remote sensing images using evaluation metrics.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the remote sensing image generalization sharpening method based on dual-level optimization-driven high-frequency enhancement as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by the processor, performs the remote sensing image generalization sharpening method based on two-level optimization-driven high-frequency enhancement as described in any one of claims 1 to 5.

Images