Remote sensing image panchromatic sharpening implementation method based on multi-modal deep learning model

Abstract

The invention relates to the field of panchromatic sharpening of remote sensing images, in particular to a remote sensing image panchromatic sharpening implementation method based on a multi-modal deep learning model. The method comprises the following steps: constructing an LTMNet for generating a high-resolution multispectral image; lGAConv is used in LTMNet, a kernel space of a standard convolution kernel is approximated by using a linear combination of two reference kernels, redundant parameters in convolution operation are remarkably reduced, a global feature harmony mechanism is introduced, and global information is fused into local features, so that inconsistency in local processing is reduced, and overall harmony of an image processing result is improved; in the LTMNet, a Transform global information optimization module is used to capture global information of an image to improve the quality of a fused image, and a feature fusion module based on an attention mechanism is used to integrate features from images of different modalities to obtain final output features. The problem that local details and global information are difficult to capture at the same time in the panchromatic sharpening process of the remote sensing image through a traditional convolutional network is solved.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image panchromatic sharpening, and in particular to a remote sensing image panchromatic sharpening implementation method based on a multimodal deep learning model. Background Art

[0002] Limited by the hardware conditions of satellite sensors, optical remote sensing satellites can only capture two different modal images at the same time. One is a single-channel panchromatic image with rich spatial details but lacking spectral information, and the other is a multispectral image with rich spectral information but low spatial resolution. These two images carry important spatial information and spectral information respectively. In order to make full use of the advantages of these two images and achieve high-quality remote sensing data analysis, panchromatic sharpening technology came into being. Panchromatic sharpening of remote sensing images is to fuse panchromatic images and multispectral images to obtain multispectral images with high spatial resolution, which is helpful for the application of remote sensing images in the fields of land object recognition and land monitoring.

[0003] Although the existing pan-sharpening methods have made significant progress, they still have some limitations. First, most of the pan-sharpening networks mentioned above usually use a single-branch network to uniformly extract spatial and spectral features, failing to fully consider the unique properties of images of different modalities, resulting in unsatisfactory quality of the fused image; second, due to the limitation of its receptive field, the convolutional neural network (CNN)-based methods mainly focus on the local feature information of the image and ignore the global dependency of the image. In order to learn the global information in the image, a Transformer architecture based on the self-attention mechanism has been widely used. However, when processing large-scale remote sensing data, Transformer-based models usually face the problems of high computational cost, large number of parameters, and insufficient local information extraction ability. Summary of the invention

[0004] In order to solve the problems of high computational cost, large number of parameters and insufficient local information extraction capability, the present invention provides a method for implementing panchromatic sharpening of remote sensing images based on a multimodal deep learning model, which mainly includes:

[0005] S1: Construct LTMNet, which includes PAN branch and Fusion branch. PAN branch is used to process full-color images and extract spatial features from PAN images. Fusion branch integrates features from LRMS and PAN branches to generate high-resolution multispectral images SR.

[0006] S2: In LTMNet, a lightweight global-aware convolution (LGAConv) that fuses global information is used to approximate the kernel space of the standard convolution kernel by using a linear combination of two benchmark kernels to reduce redundant parameters in the convolution operation. A global feature reconciliation mechanism is introduced in LTMNet to integrate global information into local features, thereby reducing inconsistencies in local processing and improving the overall coordination of image processing results.

[0007] S3: The Transformer global information optimization module is used in LTMNet to capture the global information of the image to improve the quality of the fused image, and the feature fusion module based on the attention mechanism is used to integrate the features from different modal images to obtain the final output features and realize full color sharpening of remote sensing images.

[0008] A computer device includes a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method for implementing panchromatic sharpening of remote sensing images based on a multimodal deep learning model.

[0009] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for implementing panchromatic sharpening of remote sensing images based on a multimodal deep learning model.

[0010] A computer program product includes a computer program or instructions. When the program or instructions are executed by a processor, the steps of the method for implementing panchromatic sharpening of remote sensing images based on a multimodal deep learning model are implemented.

[0011] The technical solution provided by the present invention has the following beneficial effects: the present invention proposes a dual-branch multi-scale feature extraction network of LTMNet for extracting feature information of images of different modalities. The network can effectively capture the local features and global information of the image. At the same time, a lightweight global-aware convolution (LGAConv) is also proposed to improve the model performance by fusing global information. The convolution approximates the kernel space of the standard convolution kernel by using a linear combination of two benchmark kernels, significantly reducing the computational cost and parameter amount of the model and improving the processing efficiency. At the same time, the fusion of local and global information is effectively balanced by using the Global Feature Harmonization (GFH) mechanism, further enhancing the integrity and consistency of feature representation. The proposed efficient Transformer global information optimization module captures the global information of the image, improves the quality of the fused image, and uses a feature fusion module based on the attention mechanism to integrate features from images of different modalities to reduce feature redundancy between the two images and enhance the complementarity between these features. By combining the lightweight globally aware convolution (LGAConv) and the Transformer global information optimization module, the problem that traditional convolutional networks are difficult to capture both local details and global information during the pan-sharpening process of remote sensing images is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0013] Figure 1 It is a structural diagram of LTMNet in an embodiment of the present invention;

[0014] Figure 2 is a schematic diagram of using LGAConv that fuses global information in a network in an embodiment of the present invention;

[0015] Figure 3 is a schematic diagram of using a Transformer global information optimization module in a network in an embodiment of the present invention;

[0016] Figure 4 Schematic diagram of the representation range of the first four principal components of the standard convolution kernel after principal component analysis (PCA) in an embodiment of the present invention;

[0017] Figure 5 is a schematic diagram of qualitative comparison results of simulation samples on the WV3 data set in an embodiment of the present invention;

[0018] Figure 6 is a schematic diagram of qualitative comparison results of different methods in embodiments of the present invention on real samples of the WV3 dataset;

[0019] Figure 7is a schematic diagram of qualitative comparison results of simulation samples on the QB data set in an embodiment of the present invention;

[0020] Figure 8 is a schematic diagram of qualitative comparison results of simulation samples on the WV2 data set in an embodiment of the present invention;

[0021] Fig. 9 Schematic diagram of qualitative results of ablation experiments on simulated samples on the WV3 dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0023] Example 1

[0024] The embodiment of the present invention provides a method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model, which specifically includes:

[0025] S1: Establish a multimodal fusion network based on lightweight global perception convolution and Transformer (Multimodal Fusion Network with LGAConv and Transformer, LTMNet). The network structure of LTMNet is as follows: Figure 1 As shown in the figure, LTMNet consists of two branches: the PAN branch with rich spatial information and the Fusion branch with rich spectral information and spatial information. The PAN branch is used to process panchromatic images, extract spatial features from PAN images, and focus on extracting high-resolution spatial information from panchromatic images PAN; the fusion branch processes low-resolution multispectral images MS, which are used to extract spectral information and fuse it with the spatial information of the PAN branch, integrating the features from the LRMS and PAN branches to generate a high-resolution multispectral image, which retains sufficient spatial details while maintaining accurate spectral information. Among them, LRMS represents a low-resolution multispectral image, MS∈R h×w×C represents a low-resolution multispectral image, h and w represent the height and width of the low-resolution multispectral image, respectively, and C represents the number of channels of the multispectral image; PAN∈R H×W represents a full-color image, H and W represent the height and width of the full-color image respectively, and h=H / 4, w=W / 4; SR∈R H×W×C Stands for Pan-sharpened image. Figure 1In the figure, Fs Block represents the feature fusion module, RS Block represents the reconstruction module, LGABlock represents the feature extraction module composed of LGAConv, LT Encoder represents the encoder, LTDecoder represents the decoder, Pixel Shuffle (Sub-Pixel Convolutional Neural Network) represents the upsampling operation, LGAconv represents the lightweight global-aware convolution that fuses global information, Leaky Relu represents the activation function, Conv represents the convolution, straide represents the stride, Transposed Conv represents the transposed convolution, Addition represents the element-by-element addition, Skip connection represents the skip connection, and Transformer Block represents the Transformer global information optimization module.

[0026] Depend on Figure 1 It can be seen that in order to capture enough deep information under limited network parameters, the feature maps are processed from three different scales, namely H×W×n, Where n represents the number of channels of the input feature map. At the same time, in order to assist the feature recovery process and retain more feature detail information, feature maps with the same size and number of channels are connected through jump connections. In the PAN branch and the Fusion branch, encoder-decoder modules (LT Encoder and LT Decoder) based on the LGAConv-Transformer architecture are introduced respectively. Each encoder and decoder contains LGA Block and Transformer Block. Among them, LGA Block contains two 3×3 LGAConvs and uses Leaky Relu function as the activation function. At the same time, in order to avoid gradient disappearance, a jump connection is established between the input and output. In addition, the encoder contains a convolution downsampling operation and the decoder contains a transposed convolution upsampling operation to process features at different scales respectively. In these modules, LGA Block is used to extract local features, while Transformer Block is used to capture global features to ensure that important global information is not lost when extracting spatial and spectral features, thereby obtaining multi-scale features that take into account both local details and global spatial information.

[0027] Taking the Fusion branch as an example, first, the multispectral image MS is upsampled by PixelShuffle (Sub-Pixel Convolutional Neural Network) to obtain an upsampled multispectral image with the same spatial resolution as PAN. Then, a 3×3 LGAConv is used to extract features and its channel size is changed to n to obtain the first feature map of the Fusion branch. After the feature extraction is completed, the multi-scale features from the PAN branch and the Fusion branch are fused through the feature fusion module (FSBlock) based on the attention mechanism, and the fused feature map is input into the Fusion branch for feature extraction. FS Block is responsible for integrating feature information from different modalities at multiple scales to ensure the complementarity of the panchromatic image and the multispectral image, thereby generating a feature map containing rich spectral information and spatial details. After the fused features are processed by a series of LT Encoders and LT Decoders, the network further enhances the fused high-dimensional features through an efficient RS Block (reconstruction module). RS Block consists of 1 Fs Block, 2 LGAConv and 1 Leaky ReLU activation function. Among them, Fs Block is responsible for further fusing features, reducing feature redundancy and increasing complementarity. Subsequently, the two LGAConvs gradually reduce the number of channels and adjust the features to match the number of channels C of the multispectral image. Finally, the feature map obtained by Fs Block is compared with the upsampled Perform element-by-element addition to generate the final high-resolution multispectral image SR.

[0028] In the PAN branch and the Fusion branch, the features of the PAN image and the MS image are first extracted respectively. The specific steps are as follows:

[0029] In order to capture enough deep information with limited network parameters, the model processes feature maps at three different scales, namely H×W×n, Where n represents the number of channels of the input feature map;

[0030] In order to assist feature recovery and retain more detailed information, skip connections are used to connect feature maps with the same size and number of channels;

[0031] In LTMNet, a lightweight global-aware convolution (LGAConv) that integrates global information is used. The LGAConv is integrated in the encoder (LT Encoder) and decoder (LT Decoder). Each encoder and decoder consists of an LGABlock and a Transformer Block to achieve efficient feature extraction in collaboration.

[0032] After feature extraction, the multi-scale features from the PAN branch and the Fusion branch are fused through the feature fusion module FS Block based on the attention mechanism;

[0033] The fused features are processed by a series of LT Encoders and LT Decoders;

[0034] The fused high-dimensional features are further enhanced through an efficient reconstruction module RS Block, which consists of 1 Fs Block, 2 LGAConvs, and 1 Leaky ReLU activation function. Specifically, Fs Block is responsible for further fusing features, reducing feature redundancy and increasing complementarity, and then 2 LGAConvs gradually reduce the number of channels and adjust the feature map to match the number of channels of the multispectral image;

[0035] The feature map obtained by RS Block is combined with the upsampled multispectral image Perform element-by-element addition to obtain the final high-resolution multispectral image SR.

[0036] S2: Figure 2 As shown in the figure, a lightweight global-aware convolution LGAConv that fuses global information is used in LTMNet. By using a linear combination of two benchmark kernels to approximate the kernel space of the standard convolution kernel, the redundant parameters in the convolution operation are significantly reduced. A global feature reconciliation mechanism is introduced in LTMNet to integrate global information into local features to reduce the inconsistency in local processing and improve the overall coordination of image processing results. Figure 2 In the figure, GAP (Global Average Pooling) represents global average pooling, Convolution represents convolution, and Add represents element-by-element addition.

[0037] Perform principal component analysis on the convolution kernels in the standard convolution, and treat each convolution kernel as a vector to form a convolution kernel matrix. Use PCA to decompose the convolution kernel matrix and extract the principal components with the largest variance. It is found that the first two principal components have strong representation capabilities, while the remaining components have weak capabilities, indicating that the standard convolution independently assigns a convolution kernel to each channel, resulting in a large amount of redundant information. Figure 4As shown in the figure, the representation range of the first four principal components is shown. The horizontal axis represents the representation range of each principal component, and the vertical axis represents the first four principal components after the standard convolution kernel PCA. The principal component with a larger representation range means that it occupies a larger proportion in the overall representation of the convolution kernel and has a stronger representation ability. This analysis result shows that the first two principal components have strong representation capabilities, while the representation capabilities of the remaining components are relatively weak. This result further shows that the standard convolution kernel is highly correlated between channel dimensions, and it may not require all convolution kernel parameters to capture the key features of the input data, which provides the possibility of reducing model parameters. According to the above PCA analysis results, it can be obtained that the standard convolution independently assigns a convolution kernel to each channel, which will result in a large amount of redundant information. Therefore, by introducing learnable parameters, the two single-channel benchmark kernels are linearly combined and extended to all channels, thereby approximating the kernel space of the standard convolution kernel. This method reduces parameter redundancy and improves the computational efficiency of the model. The specific generation process is as follows. Figure 2 As shown in (a), is the convolution kernel obtained by expanding two base kernels, C in is the number of input feature map channels, C out is the number of output feature map channels, and k×k is the size of the convolution kernel. In order to reduce redundancy, LGAConv first generates two benchmark kernels B1 and B2, which are defined as follows:

[0038] B1,B2∈R 1×k×k

[0039] Then, the learnable parameter α of each channel is i and β i , a linear combination of these two base kernels generates the convolution kernel K for each channel i :

[0040] K i =α i B1+β i B2,i=1,2,...,C in

[0041] Among them, i is the channel index, α i and β i is the learnable weight for each channel;

[0042] The global feature harmonization mechanism (GFH) is introduced in LTMNet. The specific implementation steps are as follows: Figure 2 As shown in (b):

[0043] First, the input feature map After the global average pooling operation, the global feature vector is obtained This vector captures the overall characteristic information of the image;

[0044] Then, the global feature vector G is passed through a fully connected layer to generate the global feature harmonization bias term Then add it to the convolution output to introduce global information:

[0045]

[0046] in, represents the convolution operation, Represents local features, It is C out convolution kernels generated by the base kernel, each with a size of C in ×k×k,O ij is the feature obtained after the initial convolution, B is the global feature harmonization bias term, and Y ij is the local feature F ij Feature map obtained after LGAConv.

[0047] S3: Figure 3 As shown in the figure, an efficient Transformer global information optimization module is used in the network to capture the global information of the image and improve the quality of the fused image. A feature fusion module based on the attention mechanism is used to integrate features from images of different modalities to obtain the final output features and realize full-color sharpening of remote sensing images.

[0048] A Transformer global information optimization module is used, such as Figure 3 As shown in Figure 5, the module consists of two key parts: multi-depth convolutional head transposed attention (MDTA) and gated deep convolutional feedforward network (GDFN). Figure 3 In the figure, Layer Norm represents layer normalization, Softmax represents Softmax function, Transposed-Attention Map represents transposed attention matrix, Reshape represents dimension transformation, GELU represents GELU activation function, Layer Normalization represents layer normalization, Matrix Mutiplication represents matrix multiplication, Element-wiseAddition represents element-wise addition, Element-wise Mutiplication represents element-wise multiplication, and Depth-wise Convolution represents depth convolution.

[0049] like Figure 3 As shown in (a) in the figure, the specific steps of MDTA are as follows:

[0050] (1) First, given the layer-normalized input feature map M L∈R H×W×C , (where C is the number of channels, H and W are the height and width respectively);

[0051] (2) Then, pixel-level cross-channel context information is aggregated through convolution with a kernel size of 1x1;

[0052] (3) Finally, channel-level spatial context information is encoded through a deep convolution with a kernel size of 3x3. This process generates a projection of the query Q, key K, and value V:

[0053]

[0054] in, represents a convolution with a kernel size of 1x1, Represents a depthwise convolution with a kernel size of 3x3. The convolution kernels of size 1x1 represent the query Q, key K, and value V, respectively. Depthwise convolution with kernel size of 3x3 representing query Q, key K and value V respectively; M L Represents the input feature map after layer normalization (LayerNorm).

[0055] Next, the query Q after dimension transformation R and key K R Perform dot product (·) to generate the transposed attention matrix A∈R C×C , then perform element-wise multiplication of the attention matrix with the projection generated in step (3) and add the original input feature M to obtain the final output feature map

[0056] The overall MDTA can be expressed as:

[0057] Attention(Q R ,K R ,V R )=V R Softmax(K R Q R / α),

[0058]

[0059] Among them, Attention represents the attention mechanism, which is R H×W×C The original tensor is reshaped to get Q R ∈R HW ×C , K R ∈R C×HW ,V R ∈R HW×C, α is a learnable scaling parameter used to adjust K before applying the softmax function R and V R The dot product size of W is used to control the attention distribution and avoid excessive or small dot product values ​​affecting gradient updates. p It is a convolution with a kernel size of 1×1, which is used to transform the weighted projection into the final output feature map. The design of MDTA avoids the computational bottleneck of standard self-attention in the spatial dimension and ensures the effective capture of global information by combining local context information.

[0060] GDFN improves the traditional feedforward network by introducing gating mechanism and deep convolution, thereby enhancing the control ability of information flow. By processing the features generated by MDTA, it dynamically filters irrelevant information and enhances useful feature information, and enhances the spatial context information of the features, ensuring that the model can more effectively capture the correlation between local information and global information. The specific framework is as follows: Figure 3 The characteristics obtained by MDTA are shown in (b). After layer normalization, two parallel linear transformation layers are used to calculate the gating signal, and then a convolution with a kernel size of 1×1 is used to adjust the number of channels of the feature map and combine it with the feature map. Add together to get the final output feature Y∈R H×W×C :

[0061]

[0062] Among them, ⊙ represents the element-by-element product, is the GELU activation function, LN represents layer normalization, and Gating represents the gating mechanism. represents the depthwise convolution with a kernel size of 3x3 in the first linear transformation layer. represents the convolution with a kernel size of 1×1 in the first linear transformation layer. represents the depthwise convolution with a kernel size of 3x3 in the second linear transformation layer. represents the convolution with a kernel size of 1×1 in the second linear transformation layer. The convolution kernel size that adjusts the number of feature map channels is 1×1. This design ensures that the features can automatically select important information flows when passing through the network, reducing the transmission of useless features and further improving the network's expressiveness.

[0063] To demonstrate the effectiveness of LTMNet, experiments were conducted on multiple datasets and LTMNet was compared with multiple state-of-the-art pan-sharpening methods. Specifically, four traditional methods were selected: BT-H, TV, MTF-GLP-FS, and BDSD-PC, as well as eight representative deep learning-based advanced methods: PNN, DiCNN, MSDCNN, BDPN, FusionNet, LAGNet, PanDiff, and BIMPan.

[0064] To ensure the fairness of the experiment, all deep learning-based methods use the same dataset and are trained on the same NVIDIATITAN XP GPU and the same PyTorch environment. The hyperparameter settings of each method are consistent with their original literature. For the LTMNet proposed in this invention, the initial learning rate is set to 0.001, the training is performed for 500 epochs, and the batch size is set to 16. At the same time, the Adam optimizer is used in the training process, and the learning rate is halved every 100 epochs.

[0065] Experiments were conducted on datasets obtained from three remote sensing satellites: WorldView-3 (WV3, 8-bands), WorldView-2 (WV2, 8-bands), and QuickBird (QB, 4-bands). Due to the lack of ground truth images as references in the original satellite data, the Wald protocol was used to generate the datasets required for the experiment.

[0066] First, the images collected by the three remote sensing satellites are cropped into smaller areas and divided into training and test sets. The training set consists of PAN / LRMS / GT image pairs generated by downsampling, with image sizes of 64×64, 16×16×C (C is the number of bands), and 64×64×C, respectively.

[0067] The training set of WV3 contains about 10,000 pairs of eight-band images, while the training set of QB contains about 17,000 pairs and 20,000 pairs of four-band images, respectively. Among them, the WV2 dataset is mainly used to verify the generalization ability of the model, so the model trained on the WV3 dataset is directly applied to the WV2 dataset for testing. For the simulation test set, 20 pairs of representative ground coverage images are prepared for each satellite. The sizes of these downsampled PAN / LRMS / GT images are 256×256, 64×64×C, and 256×256×C, respectively. In addition, the real test set includes 20 pairs of original PAN / LRMS images, with image sizes of 512×512 and 128×128, respectively.

[0068] Figure 5The figure shows the qualitative comparison results of the simulated samples on the WV3 dataset, where the first and third rows show the pan-sharpened images predicted by each method, and the second and fourth rows show the absolute error maps between the pan-sharpened images predicted by each method and the ground truth image (GT). Figure 6 The following are the qualitative comparison results of different methods on real samples of the WV3 dataset. Figure 7 The figure shows the qualitative comparison results of the simulated samples on the QB dataset, where the first and third rows show the pan-sharpened images predicted by each method, and the second and fourth rows show the absolute error maps between the pan-sharpened images predicted by each method and the ground truth image (GT). Figure 8 Qualitative comparison results of simulated samples on the WV2 dataset, where the first and third rows show the pan-sharpened images predicted by each method, and the second and fourth rows show the absolute error maps between the pan-sharpened images predicted by each method and the ground truth image (GT). Fig. 9 Qualitative results of ablation experiments on simulated samples on the WV3 dataset.

[0069] Following the research standards of panchromatic sharpening, the present invention uses spectral angle mapping (SAM), global relative error (ERGAS), universal image quality index (Q2 n ) and spatial correlation coefficient (SCC) are used to evaluate the experimental results in the simulation data set. For the real data set, the spectral distortion index D is used. λ , spatial distortion index D s The experimental results are evaluated by three evaluation indicators: quality with no reference (QNR).

[0070] By combining the lightweight global perception convolution (LGAConv) and the Transformer global information optimization module, the problem that traditional convolutional networks are difficult to capture local details and global information simultaneously during the pan-sharpening of remote sensing images is solved. Specifically, LTMNet uses a dual-branch multi-scale feature extraction network to capture the complex features of images of different modalities. LGAConv can reduce the number of convolution parameters from C to 1. in ×k×k×C out Lower to C out ×(3C in +2k 2 ), the parameter sizes of the two convolutions are shown in Table 1:

[0071] Table 1 Parameter size of two convolutions

[0072]

[0073] Table 2 shows the mean and standard deviation (std) of quantitative indicators of 20 simulated samples and 20 real samples of traditional methods and deep learning-based methods on the WV3 dataset. Table 3 shows the mean and standard deviation (std) of quantitative indicators of 20 simulated samples and 20 real samples of traditional methods and deep learning-based methods on the QB dataset. Table 4 shows the mean and standard deviation (std) of quantitative indicators of 20 simulated samples and 20 real samples of some deep learning-based methods on the WV2 dataset. Table 5 shows the quantitative results of ablation experiments on 20 simulated samples on the WV3 dataset. Table 6 shows the quantitative results of ablation experiments on 20 simulated samples on the WV3 dataset.

[0074] Table 2

[0075]

[0076]

[0077] Table 3

[0078]

[0079] Table 4

[0080]

[0081]

[0082] Table 5

[0083]

[0084] Table 6

[0085]

[0086] Among them, the bold part in Table 2-6 represents the optimal one, and the underlined part represents the suboptimal one.

[0087] Experimental results analysis

[0088] (1) Experimental analysis of simulation data sets

[0089] The purpose of the simulation dataset evaluation is to measure the difference between the fused pan-sharpened image (SR) and the ground truth image (GT). Table 2 shows the quantitative results of all the comparison methods and the proposed LTMNet model on 20 simulated samples of the WV3 dataset. The experimental results show that LTMNet outperforms other methods in all indicators, which fully proves the effectiveness of the method. On the one hand, by introducing the LGAConv module, LTMNet makes full use of the correlation between the convolution kernel channels and reduces redundant parameters. While significantly reducing the computational complexity, it enhances the ability to extract local details; on the other hand, the global information optimization module based on Restormer effectively captures global dependencies and contextual information, thereby improving the expression ability of multi-scale features. In order to intuitively demonstrate the performance of each method, Figure 5 The pseudo-color RGB images of the pan-sharpened images generated by different methods and their absolute error maps with GT are shown. The darker the absolute error map, the closer it is to GT. It can be seen that the pan-sharpened image generated by LTMNet is closest to the real image. By observing the two local magnified areas of the pseudo-color image, it can be seen that the LTMNet method is clearer in details than other methods, which is consistent with the performance of the quantitative results. The above experimental results are sufficient to show that LTMNet is superior to the current advanced pan-sharpening methods. In addition, in order to verify the wide applicability of the proposed method, comparative experiments were also carried out on the QB dataset. Table 3 and Figure 7 The quantitative and qualitative experimental results of the comparison methods are shown respectively. As expected, LTMNet performs best in overall performance, which further proves the effectiveness of the proposed method.

[0090] (2) Experimental analysis of real datasets

[0091] The goal of panchromatic sharpening is to achieve practical applications. To verify the practical applicability of the proposed method, experiments were conducted on real data samples to evaluate the generalization ability of the simulation dataset results without ground truth images. Specifically, 20 WV3 real samples and 20 QB real samples were selected for testing. Table 2 shows the quantitative experimental results of each comparison method on the WV3 real dataset. The results are consistent with the simulation dataset. LTMNet has achieved the best performance in all indicators. In order to more intuitively show the performance differences of each method, Figure 6 The visualization results of some samples in the WV3 real dataset are shown. It can be observed that the full-color sharpened images generated by LTMNet are superior to other methods in visual effects and fidelity. For the QB dataset containing 4 bands, the quantitative experimental results of each comparison method are shown in Table 3. It can be seen from Table 3 that LTMNet performs well in all indicators. The above experimental results fully demonstrate the potential value and applicability of LTMNet in practical application scenarios.

[0092] (3) Generalization experiment

[0093] In order to further verify the generalization ability of the proposed method, 20 simulated samples and 20 real samples on the WV2 dataset are used to test LTMNet and other deep learning pan-sharpening methods trained on the WV3 dataset. The quantitative evaluation results are shown in Table 4. It can be seen that LTMNet has achieved the best performance in almost all indicators. In addition, the qualitative experimental results of a simulated sample are shown in Table 4. Figure 8 As shown in the figure, by comparing the absolute error map with the pseudo-color image, it can be seen that the full-color sharpened image generated by LTMNet is visually closest to the ground truth image, and the fused image is clearer. This result is consistent with the quantitative analysis results, indicating that the proposed method has excellent generalization ability.

[0094] (4) Ablation experiment

[0095] To verify the effectiveness of the proposed method, the present invention designs four variant models of LTMNet for ablation experiments. All variant models are tested on 20 simulation samples in the WV3 dataset.

[0096] First, to verify the effectiveness of the LGAConv module, the present invention uses the standard convolutional layer to replace LGAConv to construct the first variant model (V1) while keeping the other network structures of LTMNet unchanged. In addition, to explore the role of the global feature harmonization mechanism (GFH), the GFH mechanism in LGAConv is removed to obtain the second variant model (V2). The quantitative results in Table 5 show that V2 performs lower than V1 and LTMNet in all indicators, while LTMNet, due to the combination of the advantages of GFH and LGAConv, is superior to V1 and V2 in all indicators, and its parameter volume is lower than V1. Fig. 9 The qualitative experimental results of the ablation experiment are shown. Fig. 9 It can be seen that the quality of the full-color sharpened image generated by LTMNet is better than that of V1 and V2. The above experimental results verify that the proposed LGAConv module can effectively extract feature information while reducing parameter redundancy, and the global feature harmonization mechanism (GFH) further enhances the overall consistency and quality of the image.

[0097] Secondly, to further verify the effectiveness of the model design, the present invention uses a single-branch network to uniformly extract spatial and spectral features while keeping other structures unchanged, and constructs the third variant model (V3). In addition, the Transformer global information optimization module is removed, and for the sake of parameter fairness, the module is replaced by a standard convolutional layer to obtain the fourth variant model (V4). The quantitative indicators in Table 6 are similar to Fig. 9The visual effects show that the proposed LTMNet is superior to V3 and V4 in terms of various quantitative indicators and fused image quality. Experimental results show that the dual-branch network structure can more effectively capture the feature information of multispectral and panchromatic images, avoiding the limitations of single-branch networks in feature extraction. At the same time, the Transformer global information optimization module effectively enhances the model's ability to capture long-range dependencies and global information, further improving the quality of fused images.

[0098] The dual-branch multi-scale feature extraction network of LTMNet proposed in the present invention combines lightweight global perception convolution (LGAConv) and Transformer global information optimization module to solve the problem that traditional convolutional networks are difficult to capture local details and global information simultaneously during the pan-sharpening process of remote sensing images. Specifically, LTMNet uses a dual-branch multi-scale feature extraction network to capture the complex features of images of different modalities. The network effectively reduces parameter redundancy and improves the efficiency of local feature extraction through LGAConv, and further improves the global consistency and detail performance of the image through the global feature harmony mechanism (GFH); in addition, the Transformer global information optimization module captures the long-range dependencies in the image and enhances the global information expression of the fused image. Experimental results show that the performance of LTMNet on multiple remote sensing datasets is better than that of the current advanced pan-sharpening methods, and it shows obvious performance advantages in terms of both spatial resolution improvement and spectral information fidelity. Ablation experiments further verify the role of LGAConv, Transformer global information optimization module and dual-branch multi-scale feature extraction network in improving model performance. In general, LTMNet not only achieves excellent panchromatic sharpening effects on simulated datasets, but also demonstrates good generalization ability in real data tests, providing an effective solution for panchromatic sharpening of remote sensing images. Extensive experiments were conducted on different datasets obtained from various satellite sensors, and the results show that the proposed method is superior to the most advanced methods in terms of visual effects and various indicators, reaching the state-of-the-art (SOTA) performance level, which strongly verifies the effectiveness of the method.

[0099] Example 2

[0100] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the above method steps.

[0101] Example 3

[0102] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above method steps.

[0103] Example 4

[0104] A computer program product includes a computer program or instructions, and when the program or instructions are executed by a processor, the above method steps are implemented.

[0105] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing panchromatic sharpening of remote sensing images based on a multimodal deep learning model, characterized in that: include: S1: Construct LTMNet, which includes PAN branch and Fusion branch. PAN branch is used to process full-color images and extract spatial features from PAN images. Fusion branch integrates features from LRMS and PAN branches to generate high-resolution multispectral images SR. S2: In LTMNet, a lightweight global-aware convolution (LGAConv) that fuses global information is used to approximate the kernel space of the standard convolution kernel by using a linear combination of two benchmark kernels to reduce redundant parameters in the convolution operation. A global feature reconciliation mechanism is introduced in LTMNet to integrate global information into local features, thereby reducing inconsistencies in local processing and improving the overall coordination of image processing results. S3: The Transformer global information optimization module is used in LTMNet to capture the global information of the image to improve the quality of the fused image, and the feature fusion module based on the attention mechanism is used to integrate the features from different modal images to obtain the final output features and realize full color sharpening of remote sensing images.

2. The method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model according to claim 1, characterized in that: In S1, in the PAN branch and the Fusion branch, feature extraction is performed on the PAN image and the MS image respectively. The specific steps are as follows: In order to capture enough deep information under limited network parameters, LTMNet processes feature maps at three different scales, namely H×W×n, Where n represents the number of channels of the input feature map; H and W are the height and width respectively; In order to assist feature recovery and retain more detailed information, skip connections are used to connect feature maps with the same size and number of channels; LGAConv is used in LTMNet. LGAConv is integrated in the encoder and decoder. Both the encoder and decoder are composed of LGABlock and Transformer Block to extract high-dimensional features. After feature extraction, the feature fusion module FS Block based on the attention mechanism fuses the multi-scale features from the PAN branch and the Fusion branch; The fused high-dimensional features are further enhanced through a reconstruction module RS Block. RS Block consists of 1 FsBlock, 2 LGAConv and 1 Leaky ReLU activation function. Fs Block is responsible for further fusing features, reducing feature redundancy and increasing complementarity. Then 2 LGAConv are used to gradually reduce the number of channels and adjust the feature map to match the number of channels of the multispectral image. The feature map obtained by RS Block is combined with the upsampled multispectral image Perform element-by-element addition to obtain the final high-resolution multispectral image SR.

3. The method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model according to claim 1, characterized in that: In S2, the convolution kernel in the standard convolution is subjected to principal component analysis, and each convolution kernel is regarded as a vector to form a convolution kernel matrix. The convolution kernel matrix is ​​decomposed using PCA to extract the principal component with the largest variance. To reduce redundancy, LGAConv first generates two benchmark kernels B1 and B2, which are defined as follows: B1,B2∈R 1×k×k Among them, k×k is the size of the convolution kernel; Then, the learnable parameter α of each channel is i and β i , a linear combination of these two base kernels generates the convolution kernel K for each channel i : K i =α i B1+β i B2,i=1,2,...,C in Among them, i is the channel index, α i and β i is the learnable weight for each channel, C in is the number of input feature map channels; C out is the number of output feature map channels; The global feature reconciliation mechanism is introduced in LTMNet. The specific steps are as follows: First, the input feature map After the global average pooling operation, the global feature vector is obtained Used to capture the overall characteristic information of the image; Then, the global feature vector G is passed through a fully connected layer to generate the global feature harmonization bias term Then add the global feature harmonization bias term to the convolution output to introduce global information: in, represents the convolution operation, Represents local features, It is C out convolution kernels generated by the base kernel, each with a size of C in ×k×k,O ij is the feature obtained after the initial convolution, B is the global feature harmonization bias term, and Y ij is the local feature F ij Feature map obtained after LGAConv.

4. The method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model according to claim 1, characterized in that: In S3, the Transformer global information optimization module consists of a multi-depth convolutional head transposed attention MDTA and a gated deep convolutional feedforward network GDFN.

5. The method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model as claimed in claim 4, characterized in that: In S3, the specific steps of MDTA are as follows: (1) Given a layer-normalized input feature map M L ∈R H×W×C , where C is the number of channels, H and W are the height and width respectively; (2) Aggregate pixel-level cross-channel contextual information through convolution with a kernel size of 1x1; (3) Channel-level spatial context information is encoded through a deep convolution with a kernel size of 3x3. This process generates projections Q′, K′, V′ of the query Q, key K, and value V: in, The convolution kernels of size 1x1 represent the query Q, key K, and value V, respectively. Depthwise convolution with kernel size of 3x3 representing query Q, key K and value V respectively; M L Represents the input feature map after layer normalization; The query Q after dimension transformation R and key K R Perform dot product (·) to generate the transposed attention matrix A∈R C×C , then perform element-wise multiplication of the attention matrix with the projection generated in step (3) and add the original input feature M to obtain the output feature 6. The method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model according to claim 5, characterized in that: MDTA is expressed as: Attention(Q R ,K R ,V R )=V R ·Softmax(K R ·Q R / α), Among them, Attention represents the attention mechanism, which is R H×W×C The original tensor is reshaped to get Q R ∈R HW×C , K R ∈R C×HW ,V R ∈R HW×C , α is a learnable scaling parameter used to adjust K before applying the softmax function R and V R The dot product size of W controls the attention distribution. p It is a convolution with a kernel size of 1×1, which is used to transform the weighted projection into the final output feature map, and M represents the original input feature.

7. The method for implementing pan-sharpening of remote sensing images based on a multimodal deep learning model according to claim 6, characterized in that: GDFN improves the traditional feedforward network by introducing gating mechanism and deep convolution, thereby enhancing the control ability of information flow. Processing is performed to dynamically filter irrelevant information and enhance useful feature information, and to enhance the spatial context information of features, ensuring that LTMNet can more effectively capture the correlation between local and global information; Characteristics of MDTA After layer normalization, two parallel linear transformation layers are used to calculate the gating signal, and then a convolution with a kernel size of 1×1 is used to adjust the number of channels of the feature map and combine it with the feature Add together to get the final output feature Y∈R H×W×C : Among them, ⊙ represents the element-by-element product, is the GELU activation function, LN represents layer normalization, and Gating represents the gating mechanism. represents the depthwise convolution with a kernel size of 3x3 in the first linear transformation layer. represents the convolution with a kernel size of 1×1 in the first linear transformation layer. represents the depthwise convolution with a kernel size of 3x3 in the second linear transformation layer. represents the convolution with a kernel size of 1×1 in the second linear transformation layer. Indicates that the convolution kernel size for the final adjustment of the number of feature map channels is 1×1.

8. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method for implementing full-color sharpening of remote sensing images based on a multimodal deep learning model as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the program is executed by a processor, the steps of the method for implementing full-color sharpening of remote sensing images based on a multimodal deep learning model as described in any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that It includes a computer program or instruction, which, when executed by a processor, implements the steps of the method for implementing full-color sharpening of remote sensing images based on a multimodal deep learning model as described in any one of claims 1 to 7.

Images