Remote sensing image sharpening method and system based on bilateral adaptive evolution Transform

Abstract

The invention discloses a remote sensing image pan-sharpening method and system based on bilateral adaptive evolution Transform, and the method comprises the steps: processing an original remote sensing image, dividing the original remote sensing image into a training set, a verification set and a test set, and forming a remote sensing image data set; respectively carrying out self-attention calculation in horizontal and vertical dimensions on the images in the remote sensing image data set, updating the weight according to an adaptive evolution mechanism, and constructing a network model based on bilateral adaptive evolution Transformer; a training set in the remote sensing image data set is used for training a network model based on bilateral adaptive evolution Transform, a trained network model is obtained, and network parameters corresponding to the model are globally optimal network parameters; and using the trained network model to obtain the remote sensing image after the sharpening, and completing the sharpening of the remote sensing image. According to the method, the calculation amount is remarkably reduced, the fusion precision and efficiency are improved, and the method has excellent numerical indexes and visual effects on different data sets.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing image processing and multispectral image fusion, and in particular to a remote sensing image pan-sharpening method and system based on bilateral adaptive evolving Transformer. Background Art

[0002] With the improvement of remote sensing technology and satellite image acquisition capabilities, remote sensing image fusion technology has become one of the important research directions in the field of remote sensing image processing. Pan-Sharpening is a widely used technology that combines high-resolution panchromatic remote sensing images with low-resolution multispectral remote sensing images to generate multispectral images with high spatial resolution and high spectral fidelity.

[0003] Traditional multispectral image fusion methods, such as those based on component replacement and multi-resolution analysis, encounter the problem of balancing spatial and spectral distortions in practical applications. In recent years, the progress of deep learning technology has brought new opportunities for image fusion, especially CNN (Convolutional Neural Networks), which has shown superior performance in image feature extraction and fusion. However, CNN models are usually unable to effectively capture the global information in the image, especially when processing large-scale remote sensing images, which tend to ignore long-distance dependencies.

[0004] To overcome this limitation, the Transformer model was introduced into the field of image processing due to its global modeling capability of self-attention mechanism. Although the Transformer has shown strong performance in image fusion tasks, its computational complexity is still a major challenge, especially when processing high-resolution images, where computational and memory overheads increase significantly. Therefore, how to reduce the computational complexity of the Transformer while maintaining its global modeling advantage has become the focus of current image fusion research. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a remote sensing image pan-sharpening method and system based on bilateral adaptive evolutionary Transformer, which overcomes the limitations of the existing Transformer framework in terms of computational complexity and memory overhead by introducing bilateral attention calculation and adaptive attention evolution, reduces repeated calculations and optimizes computational efficiency.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer includes the following steps:

[0008] S1. Process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images, and divide the original remote sensing images and the obtained remote sensing images into a training set, a validation set, and a test set to form a remote sensing image dataset.

[0009] S2. Perform self-attention calculations in the horizontal and vertical dimensions for the images in the remote sensing image dataset, update the weights according to the adaptive evolution mechanism, and construct a network model based on the bilateral adaptive evolution Transformer.

[0010] S3. Use the training set in the remote sensing image data set to train the network model based on the bilateral adaptive evolved Transformer to obtain a trained network model, and the network parameters corresponding to the model are the globally optimal network parameters; use the trained network model to obtain the remote sensing image after pan-sharpening, and complete the pan-sharpening of the remote sensing image.

[0011] Furthermore, in step S1, the remote sensing image data set includes the following contents:

[0012] S101, using the satellite's corresponding modulation transfer function to transform the original remote sensing image GT∈RH× W×B Fuzzy processing is performed and the nearest neighbor interpolation method is used to obtain the low-resolution multispectral remote sensing image MS∈R H / r×W / r×B and high-resolution panchromatic remote sensing images PAN∈R H×W×1 .

[0013] Among them, H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image.

[0014] S102, cropping the low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image to 64×64×1 and 16×16×8 respectively, and cropping the original remote sensing image to 64×64×8 pixels.

[0015] S103, dividing the processed remote sensing images into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0016] Furthermore, in step S2, constructing a network model based on the bilateral adaptive evolutionary Transformer includes the following contents:

[0017] S201, first use inter23tap interpolation to upsample the low-resolution multispectral remote sensing image to the same size as the high-resolution panchromatic remote sensing image, and the obtained remote sensing image is recorded as MS U , using convolution calculation to convert MS UAnd high-resolution panchromatic remote sensing images are projected into high-dimensional space to obtain the corresponding high-dimensional features. The specific formula is:

[0018]

[0019] F P =Conv PAN (PAN)

[0020] Among them, Conv() represents the convolution layer, F M Represents the characteristics of low-resolution multispectral remote sensing images, F P Represents the characteristics of high-resolution panchromatic remote sensing imagery.

[0021] S202: Use the multi-head self-attention mechanism in the horizontal dimension to process the features obtained in step S201 to obtain the horizontal attention weight map A H , the specific formula is:

[0022]

[0023]

[0024]

[0025] in, Represents the first-level query matrix of high-resolution panchromatic remote sensing images, It represents the first-level answer matrix of low-resolution multispectral remote sensing image, and Softmax represents the softmax function.

[0026] S203: Use the multi-head self-attention mechanism in the vertical dimension to process the features obtained in step S201 to obtain the vertical attention weight map A V , the specific formula is:

[0027]

[0028]

[0029]

[0030] in, The second level query matrix of high resolution panchromatic remote sensing image is shown. Represents the second-level answer matrix of low-resolution multispectral remote sensing images.

[0031] S204, according to A H and A V , and obtain the horizontal cross-modal aggregation feature Z H and the vertical cross-modal aggregation feature Z V, the specific formula is:

[0032]

[0033]

[0034] in, Represents the first-level value matrix of low-resolution multispectral remote sensing images, Represents the second-level value matrix of low-resolution multispectral remote sensing images,

[0035] S205, based on Z H and Z V , the total feature Z is obtained using the convolutional neural network. The specific formula is:

[0036] Z=g(Z H , Z V )

[0037] Here, g(·,·) represents a convolutional neural network.

[0038] S206, respectively update A through the adaptive evolution mechanism H and A V Specifically, the update kernel is generated by the corresponding position of the attention weight map, and then convolved with the original weight to complete the update. The formula is as follows:

[0039] A′ ij =f(U(A ij ))*U(A ij )

[0040] Among them, A′ ij represents the updated attention weight map, A ij represents the attention weight map, i∈H, j∈W; f(·) represents the neural network; U(A ij ) means A ij 's neighborhood.

[0041] S207: Based on the updated attention weight map and the total feature Z, a new horizontal cross-modal aggregation feature Z′ is obtained. H And the new vertical cross-modal aggregation feature Z′ V , the specific formula is:

[0042] Z′ H =A′ H ×Z

[0043] Z′ V =A′ V ×Z.

[0044] S208, based on Z′H and Z′ V , the convolutional neural network is used to calculate the final total feature Z′. The specific formula is:

[0045] Z′=g(Z′ H , Z′ V ).

[0046] S209, repeating steps S205-S208 until the set number of times is reached, completing the modeling of high-dimensional features and obtaining a network model based on the bilateral adaptive evolutionary Transformer.

[0047] Furthermore, in step S3, the trained bilateral adaptive evolving Transformer network model includes the following contents:

[0048] S301, the low-resolution multispectral remote sensing image with a size of 16×16×8 pixels in the training set is interpolated by inter23tap to obtain an upsampled low-resolution multispectral remote sensing image LRMS∈R with a size of 64×64×8 H×W×B , and then the LRMS and the high-resolution panchromatic remote sensing image with a size of 64×64×1 pixels in the training set are simultaneously input into the network model based on bilateral adaptive evolutionary Transformer to obtain a high-resolution multispectral remote sensing image X∈R with a size of 64×64×8 pixels H×W×B .

[0049] S302, using a loss function to calculate the difference between the high-resolution multispectral remote sensing image and the original remote sensing image, the loss function expression is:

[0050] L total =L rec +L str

[0051] Among them, L total represents the loss function; L rec represents the reconstruction loss function, L rec =||X-GT||1;L str represents the structural loss function, L str =||1-SSIM(X, GT)||1; ||·||1 represents the L1 norm; SSIM(·,·) represents the structural similarity index.

[0052] S303, using the loss function back propagation to calculate the gradient of the network parameters in the bilateral adaptive evolution Transformer network model, and update the parameters. The specific formula is:

[0053]

[0054] Among them, θ′ represents the updated network parameters, θ represents the network parameters, lr represents the hyperparameter learning rate, represents the gradient of the network parameters.

[0055] S304. Repeat steps S301-S303 until the loss function converges. At this time, the training is completed, and a trained network model based on the bilateral adaptive evolving Transformer is obtained.

[0056] S305, input the test set in the remote sensing image data set into the trained network model based on bilateral adaptive evolutionary Transformer to obtain the pan-sharpened remote sensing image Y∈R H×W×B .

[0057] Furthermore, the method further comprises step S4: using an evaluation index to evaluate the quality of the remote sensing image after the pan-sharpening.

[0058] Furthermore, in step S4, evaluating the quality of the remote sensing image includes the following:

[0059] (1) The SAM (Spectral Angle Mapper), ERGAS (Erreur Relative Globale Adimensionnellede Synthèse), spatial correlation coefficient SCC and Q index are used to evaluate the remote sensing image after pan-sharpening. The specific calculation is as follows:

[0060] 1) SAM indicator:

[0061] Calculate the angle between the corresponding pixels of the pan-sharpened remote sensing image and the original remote sensing image. This indicator regards each spectral channel as a coordinate axis to calculate the phase angle. Set Y {i} =[Y {i}，1 , ..., Y {i}，B ] is the i-th pixel vector of the remote sensing image after pan-sharpening with B spectral channels, GT {i} =[GT {i}，1 , ..., GT {i}，B ] is the i-th pixel vector of the original remote sensing image with B spectral channels, then Y {i} and GT {i} The SAM indicators between are:

[0062]

[0063] Here, arccos(·) represents the inverse cosine function, <·,·> represents the inner product, and ||·|| represents the l2 norm.

[0064] By averaging the SAM indicators of all pixel vectors, the global SAM value of the entire remote sensing image is obtained.

[0065] 2) ERGAS indicator:

[0066]

[0067] Among them, Mean() represents the inner product.

[0068]

[0069] Among them, GT b represents the original remote sensing image of the bth spectral channel, Y b Represents the remote sensing image after pan-sharpening of the b-th spectral channel.

[0070] 3) The spatial correlation coefficient SCC is used to describe the degree of geometric deformation. The specific expression is:

[0071]

[0072] Among them, SCC represents the spatial correlation coefficient; CCS represents the cross-correlation between single remote sensing images.

[0073] 4) Q indicator:

[0074] Using Q4 indicators, Y {i} Modeled as quaternion, the specific expression is:

[0075] Y {i} =Y+k1Y {i}，2 +k2Y {i}，3 +k3Y {i}，4

[0076] Among them, k1, k2 and k3 represent the linear combination coefficients of the corresponding channels.

[0077] The calculation formula of Q4 indicator is:

[0078]

[0079] in, and Respectively represent GT {i} and Y {i} The mean of Indicates GT {i} With Y {i} The covariance of and Respectively represent GT {i} and Y {i} The standard deviation of .

[0080] The Q4 index is further extended to the case where the number of channels is greater than 4, and the Q index is obtained.

[0081] (2) Using the reference-free quality evaluation index QNR and the spectral distortion index D λ and spatial distortion index D s The specific formula for evaluating the remote sensing image after pan-sharpening is:

[0082] QNR=(1-D λ ) α (1-D S ) β

[0083]

[0084]

[0085] Among them, α and β are coefficients; d b,c (MS,Y) represents the similarity between the low-resolution multispectral remote sensing image and the remote sensing image after pan-sharpening, d b,c (MS,Y)=Q(MS b , MS)-Q(Y b , Y c ), MS b represents the low-resolution multispectral remote sensing image of the bth spectral channel, Y c represents the pan-sharpened remote sensing image of the cth spectral channel, Q() represents the Q index; p represents D λ The order of the square root of S The order of the square root of LP Represents the low-pass portion of a high-resolution panchromatic remote sensing image.

[0086] Furthermore, the present invention also proposes a remote sensing image pan-sharpening system based on bilateral adaptive evolutionary Transformer, comprising:

[0087] The data set construction module is used to process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images, and divide the original remote sensing images and the obtained remote sensing images into training sets, validation sets and test sets to form a remote sensing image data set.

[0088] The network model construction module is used to perform self-attention calculations on the images in the remote sensing image dataset in the horizontal and vertical dimensions, update the weights according to the adaptive evolution mechanism, and construct a network model based on the bilateral adaptive evolution Transformer.

[0089] The network model training module is used to train the network model based on the bilateral adaptive evolution Transformer using the training set in the remote sensing image data set to obtain the network model with the smallest loss function value, which is the trained network model. At this time, the training is completed and the network parameters corresponding to the model are the globally optimal network parameters.

[0090] The remote sensing image pan-sharpening module is used to calculate the pan-sharpened remote sensing image using the network model trained in the network model training module, thereby completing the pan-sharpening of the remote sensing image.

[0091] The quality assessment module is used to evaluate the quality of the remote sensing image after pan-sharpening using evaluation indicators.

[0092] Furthermore, the present invention also proposes an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program and performs the steps of the remote sensing image sharpening method based on the bilateral adaptive evolutionary Transformer.

[0093] Furthermore, the present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, it executes the remote sensing image sharpening method based on bilateral adaptive evolutionary Transformer.

[0094] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0095] (1) The method proposed in this paper utilizes an adaptive evolution mechanism, which significantly reduces the redundant operations in multi-head self-attention calculations, thereby greatly reducing the computational complexity while maintaining the global modeling capability of the Transformer model.

[0096] (2) The method proposed in the present invention adopts a bilateral self-attention mechanism. By fusing features in the horizontal and vertical dimensions, the spatial details of the image are improved while maintaining the spectral information. The generated high-resolution multispectral images can also achieve excellent performance in multiple remote sensing datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is an overall implementation flow chart of the present invention.

[0098] Figure 2 It is a schematic diagram of a low-resolution multispectral remote sensing image, a high-resolution panchromatic remote sensing image, and a high-resolution multispectral remote sensing image according to an embodiment of the present invention.

[0099] Figure 3It is a schematic diagram of the model architecture of the bilateral adaptive evolutionary Transformer according to an embodiment of the present invention.

[0100] Figure 4 Schematic diagram of a remote sensing image after pan-sharpening according to an embodiment of the present invention. DETAILED DESCRIPTION

[0101] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0102] To achieve the above objectives, the present invention proposes a remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer, such as Figure 1 As shown, the specific steps are as follows:

[0103] S1. Process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images, and divide the original remote sensing images and the obtained remote sensing images into training sets, validation sets and test sets to form a remote sensing image dataset; the specific contents are:

[0104] S101. Use the satellite's corresponding modulation transfer function to modify the original remote sensing image GT∈R H×W×B Fuzzy processing is performed and the nearest neighbor interpolation method is used to obtain the low-resolution multispectral remote sensing image MS∈R H / r×W / r×B and high-resolution panchromatic remote sensing images PAN∈R H×W×1 Specific remote sensing images such as Figure 2 shown.

[0105] Among them, H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image.

[0106] S102. The low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image are cropped to 64×64×1 and 16×16×8 respectively and used as network input; the original remote sensing image is cropped to 64×64×8 pixels and used as a reference image of the model to evaluate the quality of the generated image.

[0107] S103, dividing the processed data into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0108] In this embodiment, the value of r is 4.

[0109] S2. Perform self-attention calculations on the images in the remote sensing image dataset in the horizontal and vertical dimensions, update the weights according to the adaptive evolution mechanism, and build a network model based on the bilateral adaptive evolution Transformer. The specific contents are as follows:

[0110] S201, first use inter23tap interpolation to upsample the low-resolution multispectral remote sensing image to the same size as the high-resolution panchromatic remote sensing image, and the obtained remote sensing image is recorded as MS U , using convolution calculation to convert MS U And high-resolution panchromatic remote sensing images are projected into high-dimensional space to obtain the corresponding high-dimensional features. The specific formula is:

[0111]

[0112] F P =Conv PAN (PAN)

[0113] Among them, Conv() represents the convolution layer, F M Represents the characteristics of low-resolution multispectral remote sensing images, F P Represents the characteristics of high-resolution panchromatic remote sensing imagery.

[0114] S202: Use the multi-head self-attention mechanism in the horizontal dimension to process the features obtained in step S201 to obtain the horizontal attention weight map A H , the specific formula is:

[0115]

[0116]

[0117]

[0118] in, Represents the first-level query matrix of high-resolution panchromatic remote sensing images, It represents the first-level answer matrix of low-resolution multispectral remote sensing image, and Softmax represents the softmax function.

[0119] S203: Use the multi-head self-attention mechanism in the vertical dimension to process the features obtained in step S201 to obtain the vertical attention weight map A V , the specific formula is:

[0120]

[0121]

[0122]

[0123] in, The second level query matrix of high resolution panchromatic remote sensing image is shown. Represents the second-level answer matrix of low-resolution multispectral remote sensing images.

[0124] S204, according to A H and A V , and obtain the horizontal cross-modal aggregation feature Z H and the vertical cross-modal aggregation feature Z V , the specific formula is:

[0125]

[0126]

[0127] Represents the first-level value matrix of low-resolution multispectral remote sensing images, Represents the second-level value matrix of low-resolution multispectral remote sensing images,

[0128] S205, based on Z H and Z V , the total feature Z is obtained using the convolutional neural network. The specific formula is:

[0129] Z=g(Z H , Z V )

[0130] Here, g(·,·) represents a convolutional neural network.

[0131] S206, respectively update A through the adaptive evolution mechanism H and A V Specifically, the update kernel is generated by the corresponding position of the attention weight map, and then convolved with the original weight to complete the update. The formula is as follows:

[0132] A′ ij =f(U(A ij ))*U(A ij )

[0133] Among them, A′ ij represents the updated attention weight map, A ij represents the attention weight map, ij = H, V; f(·) represents a neural network, which includes a convolutional layer, a nonlinear activation layer, a pooling layer and a linear layer; U(A ij ) means A ij The neighborhood size is controlled by a hyperparameter, and the neighborhood direction is consistent with the attention weight map direction.

[0134] S207: Based on the updated attention weight map and the total feature Z, a new horizontal cross-modal aggregation feature Z′ is obtained. H And the new vertical cross-modal aggregation feature Z′ V , the specific formula is:

[0135] Z′ H =A′ H ×Z

[0136] Z′ V =A′ V ×Z.

[0137] S208, based on Z′ H and Z′ V , the convolutional neural network is used to calculate the final total feature Z′. The specific formula is:

[0138] Z′=g(Z′ H , Z′ V ).

[0139] S209, repeating steps S205-S208 until the set number of times is reached, completing the modeling of high-dimensional features, and obtaining a network model based on bilateral adaptive evolutionary Transformer using Python language in the pytorch environment; Figure 3 As shown in the figure, the model includes horizontal cross attention module HCA, horizontal adaptive evolution module HAE, vertical cross attention module VCA, vertical adaptive evolution module VAE and bilateral feature aggregation module BFIA. The modules all include basic components of neural network such as convolutional layer, linear layer, activation function, etc. BAEFormer Block consists of HAE, VAE, BFIA and layer normalization, which is the repeated structure of the network model.

[0140] S3. Use the training set in the remote sensing image data set to train the network model based on the bilateral adaptive evolution Transformer, and obtain the network model with the minimum loss function value, which is the trained network model. At this time, the training is completed, and the network parameters corresponding to the model are the global optimal network parameters; the specific content is:

[0141] S301, the low-resolution multispectral remote sensing image with a size of 16×16×8 pixels in the training set is interpolated by inter23tap to obtain an upsampled low-resolution multispectral remote sensing image LRMS∈R with a size of 64×64×8 H×W×B, and then the LRMS and the high-resolution panchromatic remote sensing image with a size of 64×64×1 pixels in the training set are simultaneously input into the network model based on bilateral adaptive evolutionary Transformer to obtain a high-resolution multispectral remote sensing image X∈R with a size of 64×64×8 pixels H×W×B .

[0142] S302, randomly assigning initial values ​​to the parameters in the network model based on bilateral adaptive evolutionary Transformer through Gaussian distribution. The difference between the high-resolution multispectral remote sensing image and the original remote sensing image is calculated using the loss function, and the specific formula is:

[0143] The loss function expression is:

[0144] L total =L rec +L str

[0145] Among them, L total represents the loss function; L rec represents the reconstruction loss function, L rec =||X-GT||1;L str represents the structural loss function, L str =||1-SSIM(X, GT)||1; ||·||1 represents the L1 norm; SSIM(·,·) represents the structural similarity index.

[0146] S303, using the loss function back propagation to calculate the gradient of the network parameters in the bilateral adaptive evolution Transformer network model, and update the parameters. The specific formula is:

[0147]

[0148] Among them, θ′ represents the updated network parameters, θ represents the network parameters, lr represents the hyperparameter learning rate, represents the gradient of the network parameters.

[0149] S304, repeat steps S301-S303 until the loss function converges, that is, the number of iterations reaches the hyperparameter I MAX At this time, the training is completed, and a trained network model based on bilateral adaptive evolutionary Transformer is obtained.

[0150] S4. Input the test set in the remote sensing image dataset into the trained network model based on bilateral adaptive evolutionary Transformer to obtain the pan-sharpened remote sensing image Y∈R H×W×B , complete the pan-sharpening of remote sensing images.

[0151] S5. Use evaluation indicators to evaluate the quality of the remote sensing image after pan-sharpening; the specific content is:

[0152] (1) In the simulation experiment, the SAM (Spectral Angle Mapper) index, ERGAS (Erreur Relative Globale Adimensionnellede Synthèse) index, spatial correlation coefficient SCC and Q index are used to evaluate the remote sensing image after pan-sharpening. The specific calculation methods of each index are as follows:

[0153] 1) SAM indicator:

[0154] Calculate the angle between the corresponding pixels of the pan-sharpened remote sensing image and the original remote sensing image. This indicator regards each spectral channel as a coordinate axis to calculate the phase angle. Set Y {i} =[Y {i},1 ,…,Y {i},B ] is the i-th pixel vector of the remote sensing image after pan-sharpening with B spectral channels, GT {i} =[GT {i},1 ,…,GT {i},B ] is the i-th pixel vector of the original remote sensing image with B spectral channels, then Y {i} and GT {i} The SAM indicators between are:

[0155]

[0156] Where, arccos() represents the arccosine function, <·,·> represents the inner product, and ||·|| represents the l2 norm;

[0157] By averaging the SAM index of all pixel vectors, the global SAM value of the entire remote sensing image is obtained; the optimal value of the SAM index is 0.

[0158] 2) ERGAS indicator:

[0159]

[0160] Among them, Mean() represents the inner product;

[0161] RMSE (Root Mean Square Error) is a common indicator for evaluating the error between two images. It reaches the optimal value of 0 when and only when the two images are completely equal.

[0162]

[0163] Among them, GT brepresents the original remote sensing image of the bth spectral channel, Y b Represents the remote sensing image after pan-sharpening of the b-th spectral channel.

[0164] ERGAS comprehensively considers the relative errors of all channels and is a multimodal indicator whose optimal value is 0.

[0165] 3) The spatial correlation coefficient SCC can be used to describe the degree of geometric deformation. The specific expression is:

[0166]

[0167] Among them, SCC represents the spatial correlation coefficient; CCS represents the cross-correlation between single remote sensing images.

[0168]

[0169] 4) Q indicator:

[0170] Q2 m It is a multi-band extension of the universal remote sensing image quality index. {i} Modeled as quaternion, the specific expression is:

[0171] Y {i} =Y+k1Y {i}，2 +k2Y {i}，3 +k3Y {i}，4

[0172] Among them, k1, k2 and k3 represent the linear combination coefficients of the corresponding channels.

[0173] The calculation formula of Q4 indicator is:

[0174]

[0175] in, and Respectively represent GT {i} and Y {i} The mean of Indicates GT {i} With Y {i} The covariance of and Respectively represent GT {i} and Y {i} The standard deviation of .

[0176] The Q4 index is further extended to the case where the number of channels is greater than 4, and the Q index is obtained.

[0177] (2) In real data experiments, the reference-free quality evaluation index QNR and the spectral distortion index D λ and spatial distortion index Ds The specific formula for evaluating the remote sensing image after pan-sharpening is:

[0178] QNR=(1-D λ ) α (1-D S ) β

[0179]

[0180]

[0181] Among them, α and β are coefficients; d b,c (MS,Y) represents the similarity between the low-resolution multispectral remote sensing image and the remote sensing image after pan-sharpening, d b,c (MS,Y)=Q(MS b , MS)-Q(Y b , Y c ), MS b represents the low-resolution multispectral remote sensing image of the bth spectral channel, Y c represents the pan-sharpened remote sensing image of the cth spectral channel, Q() represents the Q index, which is used to calculate the difference between the two channels; p represents D λ The order of the square root of is set to 1; q represents D S The order of the square root of is set to 1; LP Represents the low-pass portion of a high-resolution panchromatic remote sensing image.

[0182] The higher the QNR index, the better the fusion quality. λ and D s When both are equal to 0, the QNR index reaches the maximum theoretical value of 1.

[0183] The average value of each evaluation index is: QNR / D λ / D S =0.9515 / 0.0158 / 0.0332.

[0184] In addition, visual comparisons are also conducted, such as Figure 4 As shown in the figure, the full-color pan-sharpening experimental results of the method proposed in the present invention and other pan-sharpening methods on the WorldView-3 real dataset are demonstrated. It can be seen from the figure that the network model based on bilateral adaptive evolving Transformer designed by the present invention shows better visual effects than other pan-sharpening networks.

[0185] Through the above embodiments and simulation results, the present invention realizes efficient and accurate multispectral image fusion through bilateral adaptive evolutionary Transformer model, which is superior to existing methods in multiple evaluation indicators. Experimental results show that the present invention performs well in spectral fidelity, spatial resolution and computational efficiency.

[0186] The embodiment of the present invention also proposes a remote sensing image sharpening system based on a bilateral adaptive evolutionary Transformer, including a data set construction module, a network model construction module, a network model training module, a remote sensing image sharpening module, a quality assessment module, and a computer program that can be run on a processor. It should be noted that each module in the above system corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0187] The embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, the specific steps of the method provided in the embodiment of the present invention are corresponding to the specific steps of the method provided in the embodiment of the present invention, and the processor has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0188] The embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. It should be noted that when the computer program is executed by the processor, it corresponds to the specific steps of the method provided in the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of the present invention.

[0189] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A remote sensing image pan-sharpening method based on bilateral adaptive evolutionary Transformer, characterized in that: include: S1. Process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images, and divide the original remote sensing images and the obtained remote sensing images into a training set, a validation set, and a test set to form a remote sensing image data set; S2, perform self-attention calculations in the horizontal and vertical dimensions for the images in the remote sensing image dataset, update the weights according to the adaptive evolution mechanism, and build a network model based on the bilateral adaptive evolution Transformer; S3. Use the training set in the remote sensing image data set to train the network model based on the bilateral adaptive evolved Transformer to obtain a trained network model, and the network parameters corresponding to the model are the globally optimal network parameters; use the trained network model to obtain the remote sensing image after pan-sharpening, and complete the pan-sharpening of the remote sensing image.

2. The remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer according to claim 1, characterized in that: In step S1, the remote sensing image data set includes the following contents: S101. Use the satellite's corresponding modulation transfer function to modify the original remote sensing image GT∈R H×W×B Fuzzy processing is performed and the nearest neighbor interpolation method is used to obtain the low-resolution multispectral remote sensing image MS∈R H / r×W / r×B and high-resolution panchromatic remote sensing images PAN∈R H ×W×1 ; Among them, H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image; S102, cropping the low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image to 64×64×1 and 16×16×8 pixels respectively, and cropping the original remote sensing image to 64×64×8 pixels; S103, dividing the processed remote sensing images into a training set, a validation set, and a test set according to a 7:2:1 ratio.

3. The remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer according to claim 1, characterized in that: In step S2, building a network model based on bilateral adaptive evolutionary Transformer includes the following: S201, first use inter23tap interpolation to upsample the low-resolution multispectral remote sensing image to the same size as the high-resolution panchromatic remote sensing image, and the obtained remote sensing image is recorded as MS U , using convolution calculation to convert MS U And high-resolution panchromatic remote sensing images are projected into high-dimensional space to obtain the corresponding high-dimensional features. The specific formula is: F P =Conv PAN (PAN) Among them, Conv() represents the convolution layer, F M Represents the characteristics of low-resolution multispectral remote sensing images, F P Represents the characteristics of high-resolution panchromatic remote sensing images; S202: Use the multi-head self-attention mechanism in the horizontal dimension to process the features obtained in step S201 to obtain the horizontal attention weight map A H , the specific formula is: in, Represents the first-level query matrix of high-resolution panchromatic remote sensing images, represents the first-level answer matrix of low-resolution multispectral remote sensing images, and Softmax represents the softmax function; S203: Use the multi-head self-attention mechanism in the vertical dimension to process the features obtained in step S201 to obtain the vertical attention weight map A V , the specific formula is: in, The second level query matrix of high resolution panchromatic remote sensing image is shown. The second level answer matrix representing low resolution multispectral remote sensing images; S204, according to A H and A V , and obtain the horizontal cross-modal aggregation feature Z H and the vertical cross-modal aggregation feature Z V , the specific formula is: in, Represents the first-level value matrix of low-resolution multispectral remote sensing images, Represents the second-level value matrix of low-resolution multispectral remote sensing images, S205, based on Z H and Z V , the total feature Z is obtained using the convolutional neural network. The specific formula is: Z=g(Z H ,WITH V ) Where g(·,·) represents a convolutional neural network; S206, respectively update A through the adaptive evolution mechanism H and A V Specifically, the update kernel is generated by the corresponding position of the attention weight map, and then convolved with the original weight to complete the update. The formula is as follows: A ′ ij =f(U(A ij ))*U(A ij ) Among them, A ′ ij represents the updated attention weight map, A ij represents the attention weight map, i∈H, j∈W; f(·) represents the neural network; U(A ij ) means A ij Neighborhood of; S207: Based on the updated attention weight map and the total feature Z, a new horizontal cross-modal aggregation feature Z is obtained. ′ H and the new vertical cross-modal aggregation feature Z ′ V , the specific formula is: From ′ H =A ′ H ×Z From ′ V =A ′ V ×Z; S208, based on Z ′ H and Z ′ V , use convolutional neural network to calculate the final total feature Z ′ , the specific formula is: WITH ′ =g(Z ′ H ,WITH ′ V ); S209, repeat steps S205-S208 until the set number of times is reached, and the modeling of high-dimensional features is completed to obtain a network model based on the bilateral adaptive evolutionary Transformer, which includes a horizontal cross-attention module, a horizontal adaptive evolution module, a vertical cross-attention module, a vertical adaptive evolution module and a bilateral feature aggregation module.

4. The remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer according to claim 1, characterized in that: In step S3, the trained bilateral adaptive evolutionary Transformer network model includes the following contents: S301, the low-resolution multispectral remote sensing image with a size of 16×16×8 pixels in the training set is interpolated by inter23tap to obtain an upsampled low-resolution multispectral remote sensing image LRMS∈R with a size of 64×64×8 H×W×B , and then the LRMS and the high-resolution panchromatic remote sensing image with a size of 64×64×1 pixels in the training set are simultaneously input into the network model based on bilateral adaptive evolutionary Transformer to obtain a high-resolution multispectral remote sensing image X∈R with a size of 64×64×8 pixels H ×W×B ; Among them, H represents the height of the remote sensing image, W represents the width of the remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image; S302, using a loss function to calculate the difference between the high-resolution multispectral remote sensing image and the original remote sensing image, the loss function expression is: L total =L rec +L str Among them, L total represents the loss function; L rec represents the reconstruction loss function, L rec =‖X-GT‖1;L str represents the structural loss function, L str =‖1-SSIM(X,GT)‖1;‖·‖1 represents the L1 norm; SSIM(·,·) represents the structural similarity index; S303, using the loss function back propagation to calculate the gradient of the network parameters in the bilateral adaptive evolution Transformer network model, and update the parameters. The specific formula is: Among them, θ ′ represents the updated network parameters, θ represents the network parameters, lr represents the hyperparameter learning rate, represents the gradient of network parameters; S304, repeating steps S301-S303 until the loss function converges, at which point the training is completed, and a trained network model based on bilateral adaptive evolving Transformer is obtained; S305, input the test set in the remote sensing image data set into the trained network model based on bilateral adaptive evolutionary Transformer to obtain the pan-sharpened remote sensing image Y∈R H×W×B .

5. The remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer according to claim 1, characterized in that: The method further comprises step S4: using an evaluation index to evaluate the quality of the remote sensing image after the pan-sharpening.

6. The remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer according to claim 5, characterized in that: In step S4, evaluating the quality of the remote sensing image includes the following: (1) The spectral angle mapping index, relative global error index, spatial correlation coefficient SCC and Q index are used to evaluate the remote sensing image after pan-sharpening. The specific calculation is: 1) Spectral angle mapping indicators: Calculate the angle between the corresponding pixels of the pan-sharpened remote sensing image and the original remote sensing image. This indicator regards each spectral channel as a coordinate axis to calculate the phase angle. Set Y {i} =[Y {i},1 ,…,Y {i},B ] is the i-th pixel vector of the remote sensing image after pan-sharpening with B spectral channels, GT {i} =[GT {i},1 ,…,GT {i},B ] is the i-th pixel vector of the original remote sensing image with B spectral channels, then Y {i} and GT {i} The SAM indicators between are: Wherein, arccos(·) represents the arccosine function, <·,·> represents the inner product, ||·|| represents the l2 norm, and Y represents the remote sensing image after pan-sharpening; By averaging the SAM indexes of all pixel vectors, the global SAM value of the entire remote sensing image is obtained; 2) ERGAS indicator: Among them, Mean() represents the inner product; Among them, GT b represents the original remote sensing image of the bth spectral channel, Y b represents the remote sensing image after pan-sharpening of the bth spectral channel, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image; 3) The spatial correlation coefficient SCC is used to describe the degree of geometric deformation. The specific expression is: Among them, SCC represents the spatial correlation coefficient; CCS represents the cross-correlation between single remote sensing images. 4) Q indicator: Using Q4 indicators, Y {i} Modeled as quaternion, the specific expression is: AND {i} =Y+k1Y {i},2 +k2Y {i},3 +k3Y {i},4 Among them, k1, k2 and k3 represent the linear combination coefficients of the corresponding channels; The calculation formula of Q4 indicator is: in, and Respectively represent GT {i} and Y {i} The mean of Indicates GT {i} With Y {i} The covariance of and Respectively represent GT {i} and Y {i} The standard deviation of The Q4 index is further extended to the case where the number of channels is greater than 4, and the Q index is obtained; (2) Using the reference-free quality evaluation index QNR and the spectral distortion index D λ and spatial distortion index D s The specific formula for evaluating the remote sensing image after pan-sharpening is: QNR=(1-D λ ) α (1-D S ) β Among them, α and β are coefficients; d b,c (MS,Y) represents the similarity between the low-resolution multispectral remote sensing image and the remote sensing image after pan-sharpening, d b,c (MS,Y)=Q(MS b ,MS)-Q(Y b ,Y c ), MS b represents the low-resolution multispectral remote sensing image of the bth spectral channel, Y c represents the pan-sharpened remote sensing image of the cth spectral channel, Q() represents the Q index; p represents D λ The order of the square root of S The order of the square root of LP Represents the low-pass portion of a high-resolution panchromatic remote sensing image.

7. A system for the remote sensing image pan-sharpening method based on bilateral adaptive evolving Transformer according to claim 1, characterized in that: include: The data set construction module is used to process the original remote sensing images to obtain low-resolution multispectral remote sensing images and high-resolution panchromatic remote sensing images, and divide the original remote sensing images and the obtained remote sensing images into training sets, validation sets and test sets to form remote sensing image data sets; The network model construction module is used to perform self-attention calculations on the images in the remote sensing image dataset in the horizontal and vertical dimensions, update the weights according to the adaptive evolution mechanism, and construct a network model based on the bilateral adaptive evolution Transformer; The network model training module is used to train the network model based on the bilateral adaptive evolution Transformer using the training set in the remote sensing image data set to obtain the network model with the minimum loss function value, which is the trained network model. At this time, the training is completed and the network parameters corresponding to the model are the global optimal network parameters; The remote sensing image pan-sharpening module is used to calculate the pan-sharpened remote sensing image using the network model trained in the network model training module, thereby completing the pan-sharpening of the remote sensing image; The quality assessment module is used to evaluate the quality of the remote sensing image after pan-sharpening using evaluation indicators.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the remote sensing image sharpening method based on bilateral adaptive evolutionary Transformer as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the remote sensing image sharpening method based on bilateral adaptive evolutionary Transformer described in any one of claims 1 to 6 is executed.

Images