A remote sensing image pan-sharpening method and system based on bilateral adaptive evolution Transformer

Abstract

The application discloses a kind of based on bilateral adaptive evolution Transform's remote sensing image pan-sharpening method and system, the method includes to original remote sensing image is handled, and is divided into training set, verification set and test set, constitutes remote sensing image dataset;Image in remote sensing image dataset respectively carries out horizontal and vertical dimension's self-attention calculation, according to adaptive evolution mechanism updates weight, constructs the network model based on bilateral adaptive evolution Transform;Using the training set in remote sensing image dataset trains the network model based on bilateral adaptive evolution Transform, obtains the network model of training completion, the network parameter corresponding to this model is global optimal network parameter;Using the network model after training, obtain the remote sensing image after pan-sharpening, complete the pan-sharpening of remote sensing image.The application significantly reduces the amount of calculation, improves the precision and efficiency of fusion, has excellent numerical index and visual effect on different datasets.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of remote sensing image processing and multispectral image fusion, and particularly relates to a remote sensing image pan-sharpening method and system based on bilateral adaptive evolution Transformer. BACKGROUND

[0002] With the improvement of remote sensing technology and satellite image acquisition capability, the fusion technology of remote sensing images has become one of the important research directions in the field of remote sensing image processing. Pan-Sharpening is a widely used technology that combines high-resolution panchromatic remote sensing images with low-resolution multispectral remote sensing images to generate multispectral images with high spatial resolution and high spectral fidelity.

[0003] Traditional multispectral image fusion methods such as component substitution and multi-resolution analysis methods have encountered a balance problem between spatial and spectral distortion in practical applications. In recent years, the progress of deep learning technology has brought new opportunities for image fusion, especially CNN (Convolutional Neural Networks) has shown superior performance in image feature extraction and fusion. However, CNN models usually cannot effectively capture global information in images, especially when processing large-scale remote sensing images, it is easy to ignore long-distance dependencies.

[0004] In order to overcome this limitation, the Transformer model is introduced into the field of image processing due to its global modeling ability of self-attention mechanism. Although the Transformer has shown strong performance in processing image fusion tasks, its computational complexity is still a major challenge, especially when processing high-resolution images, the computational and memory overheads increase significantly. Therefore, how to reduce the computational complexity while maintaining the advantages of global modeling of the Transformer has become the focus of current image fusion research. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a remote sensing image pan-sharpening method and system based on bilateral adaptive evolution Transformer, which overcomes the limitations of existing Transformer framework in computational complexity and memory overhead by introducing bilateral attention calculation and adaptive attention evolution, reduces repeated calculations and optimizes computational efficiency.

[0006] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0007] A remote sensing image pan-sharpening method based on bilateral adaptive evolution Transformer, comprising the following steps:

[0008] S1, processing the original remote sensing image to obtain a low-resolution multispectral remote sensing image and a high-resolution panchromatic remote sensing image, and dividing the original remote sensing image and the obtained remote sensing image into a training set, a validation set and a test set to form a remote sensing image dataset.

[0009] S2, performing horizontal and vertical dimension self-attention calculation on the images in the remote sensing image dataset respectively, updating the weights according to an adaptive evolution mechanism, and constructing a network model based on bilateral adaptive evolution Transformer.

[0010] S3, training the network model based on bilateral adaptive evolution Transformer using the training set in the remote sensing image dataset to obtain a trained network model, the network parameters corresponding to the model being globally optimal network parameters; using the trained network model, obtaining a super-resolution remote sensing image, and completing the super-resolution of the remote sensing image.

[0011] Further, in step S1, forming the remote sensing image dataset includes the following contents:

[0012] S101, performing fuzzy processing on the original remote sensing image GT∈R H×W×B using the modulation transfer function of the satellite, and obtaining a low-resolution multispectral remote sensing image MS∈R H / r×W / r×B and a high-resolution panchromatic remote sensing image PAN∈R H×W×1 using the nearest neighbor interpolation method.

[0013] Wherein, H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image.

[0014] S102, cutting the size of the low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image to 64x64x1 and 16x16x8 respectively, and cutting the size of the original remote sensing image to 64x64x8 pixels.

[0015] S103, dividing the processed remote sensing image into a training set, a validation set and a test set according to a ratio of 7:2:1.

[0016] Further, in step S2, constructing the network model based on bilateral adaptive evolution Transformer includes the following contents:

[0017] S201, first upsample the low-resolution multispectral remote sensing image to the same size as the high-resolution panchromatic remote sensing image by inter23tap interpolation, and the obtained remote sensing image is denoted as MS U , and MS Uand project the high-resolution panchromatic remote sensing image into high-dimensional space to obtain the corresponding high-dimensional features, and the specific formula is:

[0018]

[0019] F P = Conv PAN (PAN)

[0020] wherein Conv() represents a convolution layer, F M represents the features of the low-resolution multispectral remote sensing image, and F P represents the features of the high-resolution panchromatic remote sensing image.

[0021] S202, using the horizontal dimension multi-head self-attention mechanism to process the features obtained in step S201 to obtain the horizontal direction attention weight graph A H , and the specific formula is:

[0022]

[0023] wherein, represents the first-level query matrix of the high-resolution panchromatic remote sensing image, represents the first-level answer matrix of the low-resolution multispectral remote sensing image, and Softmax represents a softmax function.

[0024] S203, using the vertical dimension multi-head self-attention mechanism to process the features obtained in step S201 to obtain the vertical direction attention weight graph A V , and the specific formula is:

[0025]

[0026] wherein, represents the second-level query matrix of the high-resolution panchromatic remote sensing image, represents the second-level answer matrix of the low-resolution multispectral remote sensing image.

[0027] S204, according to A H and A V , the horizontal direction cross-modal aggregation features Z H and the vertical direction cross-modal aggregation features Z V are obtained, and the specific formula is:

[0028]

[0029] wherein, represents the first-level value matrix of the low-resolution multispectral remote sensing image, represents the second-level value matrix of the low-resolution multispectral remote sensing image,

[0030] S205、based on Z H and Z V , the total feature Z is obtained by using a convolutional neural network, and a specific formula is as follows:

[0031] Z = g(Z H , Z V )

[0032] wherein g(·,·) represents a convolutional neural network.

[0033] S206, A H and A V are updated respectively through an adaptive evolution mechanism, specifically, an update kernel is generated through a corresponding position of an attention weight map, and then the original weight is convolved to complete the update, and a formula is as follows:

[0034] A' i,j ij = f(U(A ij i,j )) * U(A ij i,j )

[0035] wherein A' i,j ij represents an updated attention weight map, A i,j ij represents an attention weight map, i∈H,j∈W; f(·) represents a neural network; and U(A i,j ij ) represents a neighborhood of A i,j ij .

[0036] S207, based on the updated attention weight map and the total feature Z, new horizontal direction cross-modal aggregation features Z' h H and new vertical direction cross-modal aggregation features Z' v V are obtained, and a specific formula is as follows:

[0037] Z' h H = A' h H * Z

[0038] Z' v V = A' v V * Z.

[0039] S208, based on Z' h H and Z' v V , a final total feature Z' is calculated by using a convolutional neural network, and a specific formula is as follows:

[0040] Z' = g(Z' h H , Z' v V ).

[0041] S209, repeating steps S206-S208 until a set number of times is reached to stop, completing modeling of high-dimensional features, obtaining a network model based on bilateral adaptive evolution Transformer.

[0042] Further, in step S3, obtaining the trained bilateral adaptive evolution Transformer network model includes the following contents:

[0043] S301, the low-resolution multispectral remote sensing image with a size of 16x16x8 pixels in the training set is obtained by inter23tap interpolation to obtain an up-sampled low-resolution multispectral remote sensing image LRMS∈R H×W×B , and the LRMS and the high-resolution panchromatic remote sensing image with a size of 64x64x1 pixel in the training set are simultaneously input into the network model based on bilateral adaptive evolution Transformer to obtain a high-resolution multispectral remote sensing image X∈R H×W×B .

[0044] S302, the difference between the high-resolution multispectral remote sensing image and the original remote sensing image is calculated using a loss function, and the loss function expression is:

[0045] L total =L rec +L str

[0046] wherein, L total represents the loss function; L rec represents the reconstruction loss function, L rec =‖X-GT‖1; L str represents the structure loss function, L str =‖1-SSIM(X,GT)‖1; ‖·‖1 represents L1 norm; SSIM(·,·) represents structural similarity index.

[0047] S303, the gradient of the network parameters in the bilateral adaptive evolution Transformer network model is calculated using the loss function back propagation, and the parameters are updated, and the specific formula is:

[0048]

[0049] wherein, θ' represents the updated network parameters, θ represents the network parameters, lr represents the hyperparameter learning rate, represents the gradient of the network parameters.

[0050] S304, repeating steps S301-S303 until the loss function converges, at which time the training is completed, and a network model based on bilateral adaptive evolution Transformer is obtained.

[0051] S305. Input the test set from the remote sensing image dataset into the trained bilateral adaptive evolutionary Transformer-based network model to obtain the generalized sharpened remote sensing image Y∈R. H×W×B .

[0052] Furthermore, step S4 is included: evaluating the quality of the obtained oversharpened remote sensing image using evaluation metrics.

[0053] Furthermore, in step S4, evaluating the quality of the remote sensing image includes the following:

[0054] (1) The SAM (Spectral Angle Mapper) index, ERGAS (ErreurRelative Globale Adimensionnellede Synthèse) index, spatial correlation coefficient SCC, and Q index were used to evaluate the generalized sharpened remote sensing image. The specific calculations are as follows:

[0055] 1) SAM indicator:

[0056] The angle between corresponding pixels in the generalized sharpened remote sensing image and the original remote sensing image is calculated. This metric treats each spectral channel as a coordinate axis to calculate the phase angle, setting Y... {i} =[Y {i},1 ,…,Y {i},B Let ] be the i-th pixel vector of a generalized sharpened remote sensing image with B spectral channels, and GT {i} =[GT {i},1 ,…,GT {i},B Let Y be the i-th pixel vector of the original remote sensing image with B spectral channels. {i} and GT {i} The SAM index between them is:

[0057]

[0058] Here, arccos(·) denotes the inverse cosine function, <·,·> denotes the inner product, and ||·|| denotes the l2 norm.

[0059] The global SAM value of the entire remote sensing image is obtained by averaging the SAM indices of all pixel vectors.

[0060] 2) ERGAS indicator:

[0061]

[0062] Mean() represents the inner product.

[0063]

[0064] where GT b denotes the original remote sensing image of the bth spectral channel, Y b denotes the sharpened remote sensing image of the bth spectral channel.

[0065] 3) The spatial correlation coefficient SCC is used to describe the degree of geometric distortion, and the specific expression is:

[0066]

[0067] where SCC denotes the spatial correlation coefficient; CCS denotes the cross correlation between single remote sensing images,

[0068] 4) The Q index:

[0069] The Y {i} is modeled as a quaternion, and the specific expression is:

[0070] Y {i} = Y + k1Y {i},2 + k2Y {i},3 + k3Y {i},4

[0071] where k1, k2, and k3 denote the linear combination coefficients of the corresponding channels.

[0072] The calculation formula of the Q4 index is:

[0073]

[0074] where and denote the mean of GT {i} and Y {i} , respectively, denotes the covariance of GT {i} and Y {i} , and denote the standard deviation of GT {i} and Y {i} , respectively.

[0075] The Q4 index is further generalized to the case where the number of channels is greater than 4, obtaining the Q index.

[0076] (2) The no-reference quality evaluation index QNR, the spectral distortion index D λ , and the spatial distortion index D s are used to evaluate the sharpened remote sensing image, and the specific formula is:

[0077] QNR=(1-D λ ) α (1-D S ) β

[0078]

[0079] wherein, α and β represent coefficients; d b,c (MS,Y) represents the similarity of the low-resolution multispectral remote sensing image and the panchromatic remote sensing image after panchromatic, d b,c (MS,Y) = Q(MS b ,MS)-Q(Y b ,Y c ), MS b represents the low-resolution multispectral remote sensing image of the bth spectral channel, Y c represents the panchromatic remote sensing image after panchromatic of the cth spectral channel, Q() represents the Q index; p represents the order of the square root of D λ ; q represents the order of the square root of D S ; P LP represents the low-pass part of the high-resolution panchromatic remote sensing image.

[0080] Further, the application also proposes a remote sensing image panchromatic system based on bilateral adaptive evolution Transformer, comprising:

[0081] A data set construction module is used to process the original remote sensing image to obtain a low-resolution multispectral remote sensing image and a high-resolution panchromatic remote sensing image, and divide the original remote sensing image and the obtained remote sensing image into a training set, a validation set and a test set to form a remote sensing image data set.

[0082] A network model construction module is used to perform horizontal and vertical dimension self-attention calculation on the images in the remote sensing image data set, update the weights according to the adaptive evolution mechanism, and construct a network model based on bilateral adaptive evolution Transformer.

[0083] A network model training module is used to train the network model based on bilateral adaptive evolution Transformer using the training set in the remote sensing image data set to obtain a network model with the minimum loss function value, that is, a trained network model, at which time the training is completed, and the network parameters corresponding to the model are globally optimal network parameters.

[0084] A remote sensing image panchromatic module is used to calculate the panchromatic remote sensing image using the trained network model in the network model training module to complete the panchromatic of the remote sensing image.

[0085] A quality evaluation module is used to evaluate the quality of the obtained panchromatic remote sensing image using evaluation indexes.

[0086] Further, the present application also proposes an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps of the remote sensing image pan-sharpening method based on bilateral adaptive evolution Transformer.

[0087] Further, the present application also proposes a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to perform the remote sensing image pan-sharpening method based on bilateral adaptive evolution Transformer.

[0088] Compared with the prior art, the present application has the following technical effects:

[0089] (1) The method proposed by the present application utilizes the adaptive evolution mechanism, significantly reduces the redundant operations in multi-head self-attention calculation, thereby greatly reduces the computational complexity, while maintaining the global modeling ability of the Transformer model.

[0090] (2) The method proposed by the present application adopts the bilateral self-attention mechanism, through the fusion of horizontal and vertical dimensions, the spectral information is maintained while the spatial details of the image are improved, and the generated high-resolution multispectral image can also achieve excellent performance in multiple remote sensing data sets. BRIEF DESCRIPTION OF DRAWINGS

[0091] Figure 1 is the overall implementation flowchart of the present application.

[0092] Figure 2 is the schematic diagram of the low-resolution multispectral remote sensing image, the high-resolution panchromatic remote sensing image and the high-resolution multispectral remote sensing image of the embodiment of the present application.

[0093] Figure 3 is the model architecture schematic diagram of the bilateral adaptive evolution Transformer of the embodiment of the present application.

[0094] Figure 4 is the schematic diagram of the pan-sharpened remote sensing image of the embodiment of the present application. DETAILED DESCRIPTION

[0095] The present application will be further described below in conjunction with the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.

[0096] To achieve the above-mentioned purpose, the present application proposes a remote sensing image pan-sharpening method based on bilateral adaptive evolution Transformer, as shown inFigure 1 The specific steps are as follows:

[0097] S1, processing the original remote sensing image to obtain a low-resolution multispectral remote sensing image and a high-resolution panchromatic remote sensing image, and dividing the original remote sensing image and the obtained remote sensing image into a training set, a validation set and a test set to form a remote sensing image dataset; the specific content is:

[0098] S101, using the satellite corresponding modulation transfer function to perform fuzzy processing on the original remote sensing image GT∈R H×W×B , and using the nearest neighbor interpolation method to obtain a low-resolution multispectral remote sensing image MS∈R H / r×W / r×B and a high-resolution panchromatic remote sensing image PAN∈R H×W×1 , as shown in Figure 2 .

[0099] Wherein, H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image.

[0100] S102, the size of the low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image is respectively cropped to 64x64x1, 16x16x8, which is used as network input; the size of the original remote sensing image is cropped to 64x64x8 pixels, which is used as the reference image of the model to evaluate the quality of the generated image.

[0101] S103, the processed data is divided into a training set, a validation set and a test set according to 7:2:1.

[0102] In this embodiment, the value of r is 4.

[0103] S2, the images in the remote sensing image dataset are respectively subjected to horizontal and vertical dimension self-attention calculation, the weights are updated according to the adaptive evolution mechanism, and a network model based on bilateral adaptive evolution Transformer is constructed; the specific content is:

[0104] S201, first upsample the low-resolution multispectral remote sensing image to the same size as the high-resolution panchromatic remote sensing image by inter23tap interpolation, and the obtained remote sensing image is denoted as MS U , and use convolution calculation to project MS U and the high-resolution panchromatic remote sensing image into a high-dimensional space to obtain the corresponding high-dimensional features, and the specific formula is:

[0105]

[0106] F P =Conv PAN (PAN)

[0107] wherein, Conv() represents a convolution layer, F M represents a feature of a low-resolution multispectral remote sensing image, F P represents a feature of a high-resolution panchromatic remote sensing image.

[0108] S202, processing the feature obtained in step S201 by using a multi-head self-attention mechanism in a horizontal dimension to obtain an attention weight map A H , and the specific formula is as follows:

[0109]

[0110] wherein, represents a first-level query matrix of a high-resolution panchromatic remote sensing image, represents a first-level answer matrix of a low-resolution multispectral remote sensing image, and Softmax represents a softmax function.

[0111] S203, processing the feature obtained in step S201 by using a multi-head self-attention mechanism in a vertical dimension to obtain an attention weight map A V , and the specific formula is as follows:

[0112]

[0113]

[0114] wherein, represents a second-level query matrix of a high-resolution panchromatic remote sensing image, represents a second-level answer matrix of a low-resolution multispectral remote sensing image.

[0115] S204, obtaining a horizontal cross-modal aggregated feature Z H and a vertical cross-modal aggregated feature Z V based on A H and A V , and the specific formula is as follows:

[0116]

[0117] represents a first-level value matrix of a low-resolution multispectral remote sensing image, represents a second-level value matrix of a low-resolution multispectral remote sensing image,

[0118] S205, obtaining a total feature Z by using a convolutional neural network based on Z H and Z V , and the specific formula is as follows:

[0119] Z = g(Z H ,Z V )

[0120] where g(·,·) denotes a convolutional neural network.

[0121] S206, update A H and A V respectively through an adaptive evolution mechanism, specifically, generate an update kernel through the corresponding position of the attention weight map, and then complete the update by convolution with the original weight, the formula is as follows:

[0122] A′ ij = f(U(A ij ))*U(A ij )

[0123] where A′ ij denotes the updated attention weight map, A ij denotes the attention weight map, ij = H, V; f(·) denotes a neural network, which includes a convolutional layer, a nonlinear activation layer, a pooling layer and a linear layer; U(A ij ) denotes the neighborhood of A ij , the neighborhood size is controlled by a hyperparameter, and the neighborhood direction is consistent with the direction of the attention weight map.

[0124] S207, based on the updated attention weight map and the total feature Z, obtain new horizontal direction cross-modal aggregation feature Z′ H and new vertical direction cross-modal aggregation feature Z′ V , the specific formula is:

[0125] Z′ H = A′ H ×Z

[0126] Z′ V = A′ V ×Z.

[0127] S208, based on Z′ H and Z′ V , use a convolutional neural network to calculate the final total feature Z′, the specific formula is:

[0128] Z′ = g(Z′ H ,Z′ V ).

[0129] S209, repeat steps S206-S208 until the set number of times is reached to stop, complete the modeling of high-dimensional features, and obtain a network model based on bilateral adaptive evolution Transformer in a pytorch environment using python language; for example Figure 3As shown, the model includes a horizontal cross-attention module HCA, a horizontal adaptive evolution module HAE, a vertical cross-attention module VCA, a vertical adaptive evolution module VAE, and a bilateral feature aggregation module BFIA, which all include basic components of neural networks such as convolutional layers, linear layers, activation functions, etc. The BAEFormer Block is composed of HAE, VAE, BFIA, and layer normalization, and is a repeated structure of the network model.

[0130] S3, training the network model based on the bilateral adaptive evolution Transformer using the training set in the remote sensing image data set, obtaining the network model with the minimum loss function value, that is, the trained network model, at which time the training is completed, and the network parameters corresponding to the model are the globally optimal network parameters; the specific content is:

[0131] S301, the low-resolution multispectral remote sensing image with a size of 16x16x8 pixels in the training set is obtained by inter23tap interpolation to obtain an up-sampled low-resolution multispectral remote sensing image LRMS∈R H×W×B , and the high-resolution panchromatic remote sensing image with a size of 64x64x1 pixel in the training set is simultaneously input into the network model based on the bilateral adaptive evolution Transformer to obtain a high-resolution multispectral remote sensing image X∈R H×W×B .

[0132] S302, the parameters in the network model based on the bilateral adaptive evolution Transformer are randomly given initial values by Gaussian distribution. The difference between the high-resolution multispectral remote sensing image and the original remote sensing image is calculated using the loss function, and the specific formula is:

[0133] The loss function expression is:

[0134] L total =L rec +L str

[0135] Wherein, L total represents the loss function; L rec represents the reconstruction loss function, L rec =‖X-GT‖1; L str represents the structure loss function, L str =‖1-SSIM(X,GT)‖1; ‖·‖1 represents the L1 norm; SSIM(·,·) represents the structural similarity index.

[0136] S303, the gradient of the network parameters in the bilateral adaptive evolution Transformer network model is calculated by back propagation using the loss function, and the parameters are updated, and the specific formula is:

[0137]

[0138] wherein θ' represents the updated network parameters, θ represents the network parameters, lr represents the hyperparameter learning rate, represents the gradient of the network parameters.

[0139] S304, repeat steps S301-S303 until the loss function converges, that is, the number of iterations reaches the hyperparameter I MAX At this time, the training is completed, and the network model based on bilateral adaptive evolution Transformer is obtained.

[0140] S4, input the test set in the remote sensing image data set into the network model based on bilateral adaptive evolution Transformer which is trained, to obtain the panned remote sensing image Y H×W×B , and the panning of the remote sensing image is completed.

[0141] S5, evaluate the quality of the obtained panned remote sensing image by using the evaluation index; the specific content is:

[0142] (1) in the simulation experiment, the SAM (Spectral Angle Mapper, spectral angle mapper) index, the ERGAS (Erreur Relative Globale Adimensionnelle de Synthèse, relative global error) index, the spatial correlation coefficient SCC and the Q index are used to evaluate the panned remote sensing image, and the specific calculation method of each index is as follows:

[0143] 1) SAM index:

[0144] The angle between the panned remote sensing image and the original remote sensing image corresponding to each pixel is calculated, the index regards each spectral channel as a coordinate axis to calculate the phase angle, and Y {i} = [Y {i},1 ,…,Y {i},B ] is the i-th pixel vector of the panned remote sensing image with B spectral channels, GT {i} = [GT {i},1 ,…,GT {i},B ] is the i-th pixel vector of the original remote sensing image with B spectral channels, and the SAM index between Y {i} and GT {i} is:

[0145]

[0146] where arccos() denotes the inverse cosine function, <·,·> denotes the inner product, and ||·|| denotes the l2 norm.

[0147] The global SAM value of the whole remote sensing image is obtained by averaging the SAM index of all pixel vectors. The optimal value of the SAM index is 0.

[0148] 2) ERGAS index:

[0149]

[0150] where Mean() denotes the inner product.

[0151] RMSE (Root Mean Square Error) is a common index for evaluating the error between two images, and reaches the optimal value 0 only when the two images are completely equal.

[0152]

[0153] where GT b denotes the original remote sensing image of the b-th spectral channel, Y b denotes the sharpened remote sensing image of the b-th spectral channel.

[0154] ERGAS comprehensively considers the relative error of all channels and is a multi-modal index, and its optimal value is 0.

[0155] 3) The spatial correlation coefficient SCC can be used to describe the degree of geometric deformation, and the specific expression is:

[0156]

[0157] where SCC denotes the spatial correlation coefficient, and CCS denotes the cross correlation between single remote sensing images,

[0158] 4) Q index:

[0159] Q2 m is a multi-band extension of the universal remote sensing image quality index, and Y {i} is modeled as a quaternion, and the specific expression is:

[0160] Y {i} = Y + k1Y {i},2 + k2Y {i},3 + k3Y {i},4

[0161] where k1, k2, and k3 denote the linear combination coefficients of the corresponding channels.

[0162] The calculation formula of the Q4 index is:

[0163]

[0164] wherein, and denote the mean of GT {i} and Y {i} , denotes the covariance of GT {i} and Y {i} , and denote the standard deviation of GT {i} and Y {i} , respectively.

[0165] The Q4 index is further generalized to the case of more than 4 channels to obtain the Q index.

[0166] (2) On real data experiments, the no-reference quality evaluation index QNR, the spectral distortion index D λ and the spatial distortion index D s are used to evaluate the panned remote sensing images, and the specific formula is:

[0167] QNR = (1 - D λ ) α (1 - D S ) β

[0168]

[0169] wherein, α and β denote coefficients; d b,c (MS, Y) denotes the similarity of the low-resolution multispectral remote sensing image and the panned remote sensing image, d b,c (MS, Y) = Q(MS b , MS) - Q(Y b , Y c ), MS b denotes the low-resolution multispectral remote sensing image of the bth spectral channel, Y c denotes the panned remote sensing image of the cth spectral channel, Q() denotes the Q index for calculating the difference between two channels; p denotes the order of the square root of D λ , and is set to 1; q denotes the order of the square root of D S , and is set to 1; P LP denotes the low-pass part of the high-resolution panchromatic remote sensing image.

[0170] The higher the QNR index, the better the fusion quality; when D λ and D s are both equal to 0, the QNR index reaches the maximum theoretical value 1.

[0171] The average value of each evaluation indicator is: QNR / D λ / D S =0.9515 / 0.0158 / 0.0332.

[0172] In addition, a comparison of visual effects was made, such as Figure 4 As shown in the figure, the experimental results of panchromatic general sharpening of the proposed method and other general sharpening methods on the WorldView-3 real dataset are presented. It can be seen from the figure that the network model based on bilateral adaptive evolution Transformer designed by the present invention exhibits better visual effects than other general sharpening networks.

[0173] Through the above embodiments and simulation results, this invention achieves efficient and accurate multispectral image fusion using a bilateral adaptive evolutionary Transformer model, outperforming existing methods in multiple evaluation metrics. Experimental results demonstrate that this invention exhibits excellent performance in spectral fidelity, spatial resolution, and computational efficiency.

[0174] This invention also proposes a remote sensing image generalization sharpening system based on a bilateral adaptive evolutionary Transformer, including a dataset construction module, a network model construction module, a network model training module, a remote sensing image generalization sharpening module, a quality assessment module, and a computer program that can run on a processor. It should be noted that each module in the above system corresponds to a specific step of the method provided in this invention embodiment, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention embodiment.

[0175] This invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. It should be noted that when the processor executes the computer program, it corresponds to the specific steps of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0176] This invention also proposes a computer-readable storage medium storing a computer program. It should be noted that when the computer program is executed by a processor, it corresponds to the specific steps of the method provided in this invention, possessing the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in this invention.

[0177] The above merely describes the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A remote sensing image pan-sharpening method based on bilateral adaptive evolution Transformer, characterized in that, Comprise: S1, processing the original remote sensing image to obtain a low-resolution multispectral remote sensing image and a high-resolution panchromatic remote sensing image, and dividing the original remote sensing image and the obtained remote sensing image into a training set, a validation set and a test set to form a remote sensing image dataset; S2, performing horizontal and vertical dimension self-attention calculation on the images in the remote sensing image dataset respectively, updating the weights according to the adaptive evolution mechanism, and constructing a network model based on bilateral adaptive evolution Transformer; Specifically: S201. First, the low-resolution multispectral remote sensing image is upsampled to the same size as the high-resolution panchromatic remote sensing image using inter23tap interpolation. The resulting remote sensing image is denoted as MS. U Using convolution calculation to convert MS U By projecting high-resolution panchromatic remote sensing images into a high-dimensional space, the corresponding high-dimensional features are obtained. The specific formula is as follows: F P = Conv PAN (PAN) wherein, Conv() represents a convolution layer, F M represents the features of low-resolution multispectral remote sensing images, F P represents the features of high-resolution panchromatic remote sensing images; S202, use the multi-head self-attention mechanism of horizontal dimension to process the features obtained in step S201 to obtain the attention weight map A in the horizontal direction H The specific formula is: wherein, denotes a first level query matrix of high resolution panchromatic remote sensing imagery, denotes a first level answer matrix of low resolution multispectral remote sensing imagery, Softmax denotes a softmax function; S203, utilize the vertical dimension multi-head self-attention mechanism to process the feature obtained in step S201, obtain the attention weight graph A in the vertical direction V The specific formula is: wherein, a second level query matrix representing a high resolution panchromatic remote sensing image, a second level answer matrix representing a low resolution multispectral remote sensing image; S204、According to A H and A V , to obtain the horizontal cross-modal aggregated feature Z H and the vertical cross-modal aggregated feature Z V , and the specific formula is: wherein, denotes a first level value matrix of a low resolution multispectral remote sensing image, denotes a second level value matrix of a low resolution multispectral remote sensing image, S205、based on Z H and Z V Total features Z are obtained by using a convolutional neural network, and the specific formula is as follows: Z = g(Z H ,Z V ) Wherein, g(·,·) represents a convolutional neural network; S206, update A and A respectively through adaptive evolution mechanism H and A V Specifically, an updated kernel is generated at the corresponding position of the attention weight map, and then the original weight is convolved to complete the update. The formula is as follows: A′ ij = f(U(A ij ))*U(A ij ) where A' = A + ΔA ij represents the updated attention weight map, A ij represents the attention weight map, i∈H,j∈W; f(·) represents a neural network; U(A ij ) represents the neighborhood of A ij . S207、based on the updated attention weight map and the total feature Z, obtain a new horizontal direction cross-modal aggregation feature Z' H and a new vertical direction cross-modal aggregation feature Z' V The specific formula is: Z' H = A' H x Z Z' V = A' V x Z; S208、based on Z′ H and Z′ V The final total feature Z′ is calculated using a convolutional neural network, and the specific formula is: Z' = g(Z' H ,Z' V ); S209, repeating steps S206-S208 until a set number of times is reached to stop, completing the modeling of the high-dimensional features, and obtaining a network model based on bilateral adaptive evolution Transformer, which includes a horizontal cross-attention module, a horizontal adaptive evolution module, a vertical cross-attention module, a vertical adaptive evolution module and a bilateral feature aggregation module; S3, training the network model based on bilateral adaptive evolution Transformer using the training set in the remote sensing image dataset to obtain a trained network model, the network parameters of which are globally optimal; using the trained network model, obtaining a panchromatic remote sensing image after panchromatic, and completing the panchromatic of the remote sensing image.

2. The bilateral self-adaptive evolution Transformer-based remote sensing image pan-sharpening method according to claim 1, characterized in that, In step S1, the remote sensing image dataset comprises the following contents: S101, using the satellite corresponding to the modulation transfer function of the original remote sensing image GT∈R H×W×B Fuzzy processing is carried out, and the nearest neighbor interpolation method is used to obtain low-resolution multispectral remote sensing image MS∈R H / r×W / r×B And high-resolution panchromatic remote sensing image PAN∈R H ×W×1 ; Wherein, H represents the height of the remote sensing image, W represents the width of the remote sensing image, r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to the low-resolution multispectral remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image; S102, the size of the low-resolution multispectral remote sensing image and the high-resolution panchromatic remote sensing image is respectively cropped to 64x64x1, 16x16x8, and the size of the original remote sensing image is cropped to 64x64x8 pixels; S103, the processed remote sensing image is divided into a training set, a validation set and a test set according to 7:2:

1.

3. The bi-lateral self-adaptive evolving Transformer based remote sensing image pan-sharpening method of claim 1, wherein, In step S3, obtaining the trained bilateral adaptive evolution Transformer network model comprises the following contents: S301, the low-resolution multispectral remote sensing image with a size of 16x16x8 pixels in the training set is obtained by inter23tap interpolation to obtain an up-sampled low-resolution multispectral remote sensing image LRMS e R H×W×B , and the LRMS and the high-resolution panchromatic remote sensing image with a size of 64x64x1 pixel in the training set are simultaneously input into the network model based on the bilateral adaptive evolution Transformer to obtain a high-resolution multispectral remote sensing image X e R H ×W×B ; Wherein, H represents the height of the remote sensing image, W represents the width of the remote sensing image, and B represents the total number of spectral channels of the multispectral remote sensing image; S302, calculating the difference between the high-resolution multispectral remote sensing image and the original remote sensing image using a loss function, and the loss function expression is: L total = L rec + L str wherein L total represents a loss function; L rec represents a reconstruction loss function, L rec =‖X-GT‖1; L str represents a structure loss function, L str =‖1-SSIM(X,GT)‖1;‖·‖1 represents an L1 norm; SSIM(·,·) represents a structural similarity index; S303, calculating the gradient of the network parameters in the bilateral adaptive evolution Transformer network model using the loss function back propagation, and updating the parameters, and the specific formula is: wherein θ' denotes the updated network parameters, θ denotes the network parameters, lr denotes the hyperparameter learning rate, denotes the gradient of the network parameters; S304, repeating steps S301-S303 until the loss function converges, at which time the training is completed, and a trained network model based on bilateral adaptive evolution Transformer is obtained; S305, input the test set in the remote sensing image data set into the trained network model based on bilateral adaptive evolution Transformer, to obtain the pan-sharpened remote sensing image Y e R H×W×B .

4. The bilateral self-adaptive evolving Transformer based remote sensing image pan-sharpening method of claim 1, wherein, Further comprising step S4: evaluating the quality of the obtained panchromatic remote sensing image using evaluation indexes.

5. The remote sensing image pan-sharpening method based on bilateral self-adaptive evolving Transformer according to claim 4, characterized in that, In step S4, the quality of the remote sensing image is evaluated as follows: (1) The spectral angle mapping index, the relative global error index, the spatial correlation coefficient SCC and the Q index are used to evaluate the panchromatic remote sensing image, and the specific calculation is: 1) Spectral Angle Mapper (SAM) index: The angle between the corresponding pixels of the panned remote sensing image and the original remote sensing image is calculated, the index takes each spectral channel as a coordinate axis to calculate the phase angle, set Y {i} = [Y {i},1 ,…,Y {i},B ] as the i-th pixel vector of the panned remote sensing image with B spectral channels, GT {i} = [GT {i},1 ,…,GT {i},B ] as the i-th pixel vector of the original remote sensing image with B spectral channels, then the SAM index between Y {i} and GT {i} is: where arccos(·) denotes the inverse cosine function, <·,·> denotes the inner product, ||·|| denotes the l2 norm, and Y denotes the super-resolution remote sensing image; The global SAM value of the whole remote sensing image is obtained by averaging the SAM index of all pixel vectors; 2) ERGAS index: where Mean() denotes the inner product; where GT b represents the original remote sensing image of the bth spectral channel, Y b represents the remote sensing image after unsharp masking of the bth spectral channel, and r represents the ratio of the spatial resolution of the high-resolution panchromatic remote sensing image to that of the low-resolution multispectral remote sensing image. 3) Spatial Correlation Coefficient (SCC) is used to describe the degree of geometric distortion, and the specific expression is: wherein SCC represents a spatial correlation coefficient; CCS represents a cross correlation between single remote sensing images, 4) Q index: Y is modeled as a quaternion with the specific expression: {i} Y is modeled as a quaternion with the specific expression: Y {i} = Y + k1Y {i},2 + k2Y {i},3 + k3Y {i},4 where k1, k2, and k3 denote the linear combination coefficients of the corresponding channels; The calculation formula of the Q4 index is: wherein and respectively denote the mean of GT {i} and Y {i} , denotes the covariance of GT {i} and Y {i} , and respectively denote the standard deviation of GT {i} and Y {i} . The Q index is further generalized to the case where the number of channels is greater than 4; (2) using the no-reference quality metric QNR, the spectral distortion metric D λ and the spatial distortion metric D s The remote sensing image after the unsharp masking is evaluated, and the specific formula is: QNR = (1 - D λ ) α (1 - D S ) β wherein α and β both represent coefficients; d b,c (MS, Y) represents the similarity between the low-resolution multispectral remote sensing image and the panchromatic remote sensing image, d b,c (MS, Y) = Q(MS b , MS) - Q(Y b , Y c ), MS b represents the low-resolution multispectral remote sensing image of the bth spectral channel, Y c represents the panchromatic remote sensing image of the cth spectral channel, Q() represents the Q index; p represents the order of the square root of D λ ; q represents the order of the square root of D S ; P LP represents the low-pass part of the high-resolution panchromatic remote sensing image.

6. The system for the bilateral self-adaptive evolution Transformer-based remote sensing image pan-sharpening method of claim 1, wherein It includes: A data set construction module for processing the original remote sensing image to obtain a low-resolution multispectral remote sensing image and a high-resolution panchromatic remote sensing image, and dividing the original remote sensing image and the obtained remote sensing image into a training set, a validation set, and a test set to form a remote sensing image data set; A network model construction module for performing horizontal and vertical dimension self-attention calculations on the images in the remote sensing image data set, updating the weights according to the adaptive evolution mechanism, and constructing a network model based on bilateral adaptive evolution Transformer; A network model training module for training the network model based on bilateral adaptive evolution Transformer using the training set in the remote sensing image data set to obtain a network model with the smallest loss function value, which is the trained network model, at which point the training is complete, and the network parameters corresponding to the model are the globally optimal network parameters; A super-resolution module for remote sensing images for calculating the super-resolution remote sensing image using the trained network model in the network model training module to complete the super-resolution of the remote sensing image; A quality evaluation module for evaluating the quality of the obtained super-resolution remote sensing image using evaluation indexes.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the bilateral adaptive evolution Transformer-based remote sensing image super-resolution method of any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium being characterized by, The computer program is run by the processor to execute the bilateral adaptive evolution Transformer-based remote sensing image super-resolution method of any one of claims 1 to 5.

Images