A remote sensing image fusion method based on combination of supervised and unsupervised learning

Abstract

The application provides a remote sensing image fusion method based on the combination of supervised learning and unsupervised learning, to solve the problem of unclear fusion images caused by the weak generalization of neural networks under different satellites in the field of remote sensing image fusion. We use a supervised learning model constructed by combining Unet network and Transformer network to learn the prior mapping relationship between panchromatic and multispectral images. Then in the switching satellite scene, the model parameters in the supervised learning are loaded to obtain a preliminary fusion image, and the loss function is used to drive the adjustment of the spectrum and space, overcoming the weak generalization of the neural network trained on a certain satellite under other remote sensing satellites. Through the combination of supervised learning and unsupervised learning, the general prior mapping relationship of panchromatic and multispectral images can be utilized, and the adaptive adjustment of space and spectrum can also be utilized. Therefore, the spatial details of the adaptively adjusted multispectral image are clear.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of remote sensing image fusion, and relates to a remote sensing image fusion method combining supervised learning and unsupervised learning, which is suitable for solving the application scenario that the weak generalization of neural networks under different satellites leads to unclear fused images. BACKGROUND

[0002] Remote sensing image fusion technology is to obtain high spatial resolution multispectral images by fusing panchromatic images with high spatial resolution and low spectral resolution and multispectral images with high spectral resolution and low spatial resolution. By combining the complementary information of panchromatic and multispectral images, higher quality remote sensing images can be obtained and applied in the fields of agriculture, military, geographical exploration, etc.

[0003] Existing remote sensing image fusion technologies can be divided into component substitution (CS), multi-resolution analysis (MRA), model-based methods and deep learning (DL) based methods. The component substitution method obtains a fused image by replacing the spatial components in the multispectral image with the panchromatic image. However, the component substitution method often causes spectral distortion. The multi-resolution analysis method only injects the high-frequency domain information of the panchromatic image into the multispectral image, which can better preserve the spectral domain information but often causes loss of spatial information. The model-based method establishes an optimization model by constructing spatial and spectral constraints, but generally has high computational cost.

[0004] At present, due to the powerful feature extraction capability of neural networks, deep learning based remote sensing image fusion methods are a research hotspot in the field of remote sensing. However, due to the weak generalization problem of deep learning methods, due to the problems of shooting light, angle, discrimination accuracy, etc. between different satellites, the neural network trained on one satellite often causes serious spatial distortion when applied to another satellite, resulting in a fused image with low image quality. This limits the application of deep learning in the field of remote sensing image fusion, and therefore how to design a deep learning framework to improve the generalization of the deep learning network so that the model trained on one satellite can be better applied to other satellites to obtain multispectral images with rich spatial details and small spectral distortion is an important problem in the field of remote sensing image fusion. SUMMARY

[0005] In order to solve the problems existing in the prior art, the present application provides a remote sensing image fusion method based on the combination of supervised learning and unsupervised learning.

[0006] The present application provides a remote sensing image fusion method based on the combination of supervised learning and unsupervised learning, comprising the following steps:

[0007] Step 1, construct a detail difference extraction module for extracting the detail difference between multispectral and panchromatic images;

[0008] Step 2, construct a Unetformer module to adjust the injected details through supervised learning and inject details into multispectral images;

[0009] Step 3, construct a supervised objective function to train an image fusion model to learn the spectral and spatial prior knowledge of remote sensing images, and the image fusion model includes a detail difference extraction module and a Unetformer module;

[0010] Step 4, construct an adaptive adjustment network to adjust the space and spectrum of the image driven by the space and spectrum loss function in an unsupervised manner;

[0011] Step 5, use the simulation data of one satellite to train the above image fusion model, and use the trained fusion model to adjust the image through the adaptive adjustment network on other satellites.

[0012] Further, the specific implementation of step 1 is as follows:

[0013] Step 1.1, copy the single-channel panchromatic image into a multi-channel image along the channel direction, and the number of channels is the same as that of the multispectral image;

[0014] Step 1.2, upsample the multispectral image to match the size of the panchromatic image;

[0015] Step 1.3, subtract the copied panchromatic image and the upsampled multispectral image to extract the detail difference between the multispectral and panchromatic images.

[0016] Further, the Unetformer module in step 2 includes an encoder, a Transformer block, and a decoder, and the input is the detail difference in step 1, and the specific implementation is as follows:

[0017] Step 2.1, the encoder uses 4 convolutional layers, and the number of convolutional kernels increases layer by layer to extract multi-dimensional features. The first three convolutional layers are followed by an average pooling operation to reduce the image size, and the receptive field of the convolutional kernel can cover more parts of the feature map to increase the size of the receptive field. Finally, the encoder outputs Z∈R C,H,W , C represents the number of channels;

[0018] Step 2.2, the block embedding module in the Transformer block divides the feature image Z into n image blocks and flattens them into an image block sequence:

[0019] X=[x1,x2,…,x n ]

[0020] Where X∈Rn,E , E = h x w x C, E is the number of feature channels, H, W are the image size of the image, h, w are the image block size;

[0021] Step 2.3, use three matrices W Q , W K , W V to multiply the image block sequence X, and map to query vector Q, key vector K, value vector V, Q / K / V ∈ R n , E Split the number of feature channels E in Q, K, V into multiple attention heads:

[0022] Q = [Q1, Q2,..., Q h ]

[0023] K = [K1, K2,, K h ]

[0024] V = [V1, V2,..., V n ]

[0025] Q i / K i / V i ∈ R n,d , where E = h x d, h is the number of attention heads, d is the number of feature channels of each attention head, W Q , W K , W V These three matrices are learnable parameters, and the initialization is randomly generated;

[0026] Step 2.4, for each attention head, calculate the feature similarity of Q i and K i on d feature channels:

[0027]

[0028] Where, S i ∈ R n,n , S i represents the self-similarity between two image blocks in the image block sequence on the i-th attention head; Step 2.5, normalize the correlation between the image block sequences obtained in step 2.4 through the softmax function:

[0029] g i = softmax(S i )

[0030] Step 2.6, for the calculated self-similarity g i , the value vector V iThe output vector of the i-th attention head is calculated as follows:

[0031] O i i V i

[0032] Step 2.7, by self-attention, the output vector of each attention head is obtained, and finally the output vectors of all attention heads are spliced to obtain the final output vector:

[0033] O=[O1,O2,...,O h )

[0034] Where, O∈R h,n,d ;

[0035] Step 2.8, the output vector is folded into a feature image through dimension transformation, and the output feature image is obtained;

[0036] Step 2.9, the decoder adopts 4 convolutional layers, in the first 3 convolutional layers, the feature image is first increased in size through deconvolution, then concatenated with the output of the corresponding layer of the encoder, and then passed through a convolutional layer, and finally a convolutional operation with 4 convolutional kernels in the fourth layer maps the image from the feature domain back to the image domain;

[0037] Step 2.10, through a skip connection, the decoder output is added to the upsampled multispectral image, and the spatial details are injected into the multispectral image to obtain the final fusion image.

[0038] Further, in step 2.1, the number of convolutional kernels changes from 32, 64, 128 to 256, and each convolutional layer consists of two two-dimensional convolutions + LeakyReLu activation functions.

[0039] Further, in step 2.9, the number of convolutional kernels in the first 3 layers changes from 128, 64, to 32, the first 3 convolutional layers consist of two two-dimensional convolutions + LeakyReLu activation functions, and the fourth convolutional layer is a two-dimensional convolution.

[0040] Further, the loss function constructed in step 3 is as follows:

[0041]

[0042] Where, F n and G n represent the fusion image and the reference image respectively, and b is the batch size.

[0043] Further, the specific implementation of step 4 is as follows:

[0044] ​Step 4.1, the fused image is fused through two convolutional layers, and the number of convolutional kernels is increased layer by layer to extract image features;

[0045] Step 4.2, the image features in step 4.1 are obtained through two channel attentions to obtain multi-channel feature images to focus on more important image features;

[0046] Step 4.3, two convolutional layers are used to remap the multi-channel feature images into four-channel images, and the original input fused image is added to obtain the reconstructed fused image.

[0047] Step 4.4, a spatial-spectral joint loss function is constructed to drive the unsupervised model.

[0048]

[0049] Wherein, HF and HP represent the high-frequency information of the gray fused image and the panchromatic image respectively, FB represents the blurred fused image, and MS represents the up-sampled multispectral image.

[0050] Further, in step 4.1, the channels of the convolutional layer are 32 and 64, and the size of the convolutional kernel is 3*3.

[0051] Further, in step 4.2, the channel attention is obtained by averaging and maximum pooling of the module input, then performing two convolutions and adding them, and then normalizing through the Sigmoid activation function to obtain the attention weight, and the output is obtained by multiplying the module input and the attention weight.

[0052] Further, in step 4.4, the gray fused image is obtained by adding the channels of the fused image and taking the average, and the high-frequency information of the gray fused image and the panchromatic image is obtained by a high-pass filter, the high-pass filter is obtained by subtracting the low-frequency content of the original image from the original image, the average filtering is realized through a global pooling layer, and the blurred fused image is obtained by up-sampling and then down-sampling the original fused image.

[0053] Further, in step 5, the image fusion model is trained by using the simulation data of one satellite, and the trained fusion model is used to adjust the image through adaptive adjustment network on other satellites, and the test results are compared with existing algorithms through subjective vision and objective evaluation indexes, and the objective evaluation index includes peak signal-to-noise ratio.

[0054] Compared with the prior art, the advantages and beneficial effects of the present application are:

[0055] The present application firstly learns the prior knowledge of remote sensing image fusion through supervised learning. In supervised learning, the panchromatic image is copied along the channel direction, and the injected details are obtained by subtracting the copied panchromatic image from the up-sampled multi-spectral image. Then the injected details are sent into the Unetformer module to adjust the injected details, and then the injected details are added to the multi-spectral image through the jump connection to obtain the fusion image. When switching to other satellites, the present application firstly obtains the preliminary fusion image by loading the model parameters in the supervised learning, and then adjusts the spatial and spectral of the fusion image unsupervisedly by setting the spatial and spectral loss function, and reduces the spatial distortion of the fusion image as much as possible under the condition of the least spectral distortion. In the supervised model, the Unetformer combines the Unet and the Transformer, which not only retains the advantage of the U-shaped structure that can combine high-dimensional and low-dimensional information, but also makes the large receptive field feature image output by the encoder pass through the Transformer block to better focus on global information and enhance the long-distance modeling capability, so that the adjustment of the injected spatial details is more accurate. Compared with the traditional supervised learning, the method combining supervised and unsupervised learning can make the fusion model have strong generalization ability on other satellites, dynamically adjust the space and spectrum, and improve the spatial detail and spectral fidelity, so that the multi-spectral image obtained through adaptive adjustment is clearer and the spectrum is well preserved. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is the overall framework diagram of the remote sensing image fusion network based on the combination of supervised and unsupervised learning of the embodiment.

[0057] Figure 2 is the network architecture diagram of the Unetformer module and the adaptive adjustment network of the embodiment.

[0058] Figure 3 is the test result diagram of the simulation data of the embodiment, wherein (a) is a low-resolution multi-spectral image, (b) is the result of PNN, (c) is the result of DiCNN, (d) is the result of FusionNet, (e) is the result of the method proposed in the present application, and (f) is a reference image. DETAILED DESCRIPTION

[0059] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the embodiments described herein are only used to explain the present application and do not limit the present application.

[0060] The application mainly aims to solve the application requirement of the weak generalization of neural networks under different satellites in the field of remote sensing image fusion leading to unclear fusion images. We use a model combining Unet network and Transformer network to learn the prior mapping relationship between panchromatic and multispectral images, and then load the model parameters in supervised learning to obtain a preliminary fusion image when switching satellites. The spatial and spectral of the fusion image are adjusted by an adaptive adjustment network composed of channel attention, and the spatial and spectral loss functions are set to unsupervisedly drive the adaptive adjustment network. Thus, the multispectral fusion image obtained on the cross-satellite dataset is rich in spatial details and has minimal spectral distortion.

[0061] Figure 1 is an embodiment of a whole framework diagram of a remote sensing image fusion network based on the combination of supervised and unsupervised learning. Figure 1 Figure 2 is an embodiment of a network architecture diagram of a Unetformer module and an adaptive adjustment network. The embodiment provides a remote sensing image fusion method based on the combination of supervised and unsupervised learning to solve the problem of unclear fusion images caused by the weak generalization of neural networks under different satellites in the field of remote sensing image fusion, which specifically includes the following steps: Figure 2

[0062] Step 1: Construct a detail difference extraction module for extracting the detail difference between multispectral and panchromatic images. The specific implementation includes the following sub-steps:

[0063] Step 1.1: Copy the single-channel panchromatic image into a multi-channel image along the channel direction, and the number of channels is the same as that of the multispectral image.

[0064] Step 1.2: Upsample the multispectral image to match the size of the panchromatic image.

[0065] Step 1.3: Subtract the copied panchromatic image from the upsampled multispectral image to extract the detail difference between the multispectral and panchromatic images.

[0066] Step 2: Construct a Unetformer module to adjust the injected details through nonlinear parameter learning and inject the details into the multispectral image to obtain a fusion image. The specific implementation includes the following sub-steps:

[0067] Step 2.1: The encoder adopts 4 layers of convolutional layers, each of which is composed of two two-dimensional convolution + LeakyReLu activation function. The number of convolution kernels increases layer by layer, and the number of convolution kernels changes from 32, 64, 128 to 256 to extract multi-dimensional features. An average pooling operation is followed after the first 3 layers of convolutional layers, so that the image size is reduced, and the receptive field of the convolution kernel can cover more parts of the feature map, increasing the size of the receptive field. Finally, the encoder outputs Z C,H,W ​In the present embodiment, C = 256, H = 256, W = 256.

[0068] Step 2.2: The block embedding module in the Transformer block splits the feature image Z into n image blocks and flattens it into an image block sequence:

[0069] x = [x1, x2, …, xn] n ]

[0070] where X ∈ R n,E , E = h x w x C, E is the number of feature channels, H, W are the image size of the image, h, w are the image block size. In the present embodiment, h = 4, w = 4, n = 4096, E = 4096.

[0071] Step 2.3, multiply the image block sequence X with three matrices W Q , W K , W V to get the query vector Q, the key vector K, and the value vector V, Q / K / V ∈ R n,E , split the number of feature channels E in Q, K, V into multiple attention heads:

[0072] Q = [Q1, Q2, …, Q h l

[0073] K = [K1, K2, …, K h ]

[0074] V = [V1, V2, …, V n ]

[0075] Q i / K i / V i ∈ R n , d where E = h x d, h is the number of attention heads, d is the number of feature channels of each attention head. In the present embodiment, h = 16, d = 256.

[0076] Step 2.4, for each attention head, calculate the feature similarity of Q i and K i on the feature channel d:

[0077]

[0078] where S i ∈ R n,n , S ig represents the self-similarity between two image blocks in the i-th attention head. Step 2.5, the correlation between the image block sequences obtained in step 2.4 is normalized by a softmax function:

[0079] g i = softmax(S i )

[0080] Step 2.6, for the calculated self-similarity g i , the weighted sum of all image block sequences in the value vector V i is calculated to obtain the output vector of the i-th attention head, which is calculated as follows:

[0081] O i = g i V i

[0082] Step 2.7, through self-attention, the output vectors of each attention head are obtained simultaneously, and finally the output vectors of all attention heads are spliced to obtain the final output vector:

[0083] O = [O1, O2,..., O h ]

[0084] where O ∈ R h,n,d .

[0085] Step 2.8, the output vector is folded into a feature image through dimension transformation to obtain the output feature image

[0086] Step 2.9, the decoder adopts 4 convolutional layers, in the first 3 convolutional layers, the feature image is first increased in size through deconvolution, then concatenated with the output of the corresponding layer of the encoder, and then passed through a convolutional layer; the fourth layer is a convolutional operation with 4 convolutional kernels that maps the image from the feature domain back to the image domain. The number of convolutional kernels in the first 3 layers changes from 128, 64, to 32, and the first 3 convolutional layers consist of two two-dimensional convolutions + LeakyReLu activation functions, and the fourth convolutional layer is a two-dimensional convolution.

[0087] Step 2.10, through a skip connection, the decoder output is added to the upsampled multispectral image to inject spatial details into the multispectral image, obtaining the final fusion image.

[0088] Step 3: Construct the model objective function of image fusion to drive model training. The specific implementation includes the following sub-steps:

[0089] Step 3.1: Construct the loss function. Construct the loss function based on L2:

[0090]

[0091] where F n and G n represent the fused image and the reference image respectively, and b is the batch size. In this embodiment, b = 8.

[0092] Step 3.2: Randomly select b data inputs from the training set to complete an iteration and adjust the network parameters.

[0093] Step 4: Build an adaptive adjustment network to adjust the space and spectrum of the image unsupervised by the spatial and spectral loss function. The specific implementation includes the following sub-steps:

[0094] Step 4.1, the fused image passes through two convolutional layers, and the number of convolutional kernels increases layer by layer. The number of channels of the convolutional layer is 32, 64, and the size of the convolutional kernel is 3x3 to extract image features;

[0095] Step 4.2, the feature image output by step 4.1 passes through two channel attentions to focus on more important image features. The channel attention is obtained by averaging and maximum pooling the module input, then performing two convolutions and adding them, and then normalizing through the Sigmoid activation function. The output is obtained by multiplying the module input and the attention weight;

[0096] Step 4.3, two convolutional layers are used to remap the multi-channel feature image (output of step 4.2) to a four-channel image, and the original input fused image is added to obtain the reconstructed fused image.

[0097] Step 4.4, build a spatial and spectral joint loss function to drive the adaptive adjustment network, i.e. steps 4.1-4.3, the loss function is as follows:

[0098]

[0099] where HF, HP represent the high-frequency information of the gray-scale fused image and the panchromatic image respectively, FB represents the blurred fused image, and MS represents the up-sampled multispectral image. The gray-scale fused image is obtained by adding the channels of the fused image and taking the average, and the high-frequency information HF, HP of the gray-scale fused image and the panchromatic image is obtained by a high-pass filter. The high-pass filter is realized by subtracting the low-frequency content of the original image from the original image, and the average filtering is realized by a global pooling layer. The blurred fused image is obtained by up-sampling and then down-sampling the original fused image.

[0100] Step 5: Use the simulation data of one satellite to train the above image fusion model, and use the trained fusion model to adjust the image through the adaptive adjustment network on other satellites. The specific implementation includes the following sub-steps:

[0101] Step 5.1: Training the network by supervised learning using simulation data on one satellite. In this embodiment, 90% of the images in GF-2 satellite are used for training, and 10% of the images are used for validation. The reference image is the original multispectral image with resolution 256x256, and the network input is the multispectral image down-sampled by 4 times and the panchromatic image with resolution 256x256. In the network training, the initial learning rate is 0.0003, the batch size is 8, the network is optimized by Adam optimizer, and the training rounds are 105 rounds.

[0102] Step 5.2: Loading the fusion model trained on GF-2 satellite, fixing the parameters of the fusion model, and fine-tuning the adaptive adjustment network on each image on the QuickBird satellite. The test results are compared with the comparison methods in terms of visual and objective evaluation indicators. The simulation test image size is 512x512, the adaptive adjustment network training learning rate is 9e-5, the training rounds are 2000 rounds, and the optimizer is Adam optimizer. In order to verify the effectiveness of the proposed method, the proposed method is compared with the supervised learning method. The supervised learning methods include PNN, DiCNN, and FusionNet. The visual comparison results are shown in the accompanying drawings. Figure 3 The objective evaluation index is the peak signal-to-noise ratio (PSNR), and the average results on the simulation test set are shown in Table 1.

[0103] Table 1 Comparison of average PSNR (dB) of different methods of simulation data (ideal value: +∞)

[0104]

[0105] It can be seen that the proposed method first extracts the prior mapping relationship between the panchromatic and multispectral images through supervised learning, and then adjusts the spatial and spectral of the fusion image through unsupervised learning, which can effectively alleviate the problem of unclear image caused by weak generalization of neural network on different satellites.

[0106] It should be understood that parts not described in detail in the specification are all prior art.

[0107] It should be understood that the above description of the embodiments is relatively detailed, and therefore should not be considered as limiting the scope of patent protection of the present application. Those skilled in the art can make substitutions or modifications without departing from the scope of the claims, and all fall within the scope of protection of the present application. The scope of protection of the present application should be subject to the appended claims.

Claims

1. A remote sensing image fusion method based on a combination of supervised and unsupervised learning, characterized in that, The method comprises the following steps: Step 1, constructing a detail difference extraction module for extracting the detail difference between the multispectral and panchromatic images; The specific implementation of step 1 is as follows: Step 1.1, copying the single-channel panchromatic image into a multi-channel image along the channel direction, and the number of channels is the same as that of the multispectral image; Step 1.2, up-sampling the multispectral image to match the size of the panchromatic image; Step 1.3, subtracting the copied panchromatic image from the up-sampled multispectral image to extract the detail difference between the multispectral and panchromatic images; Step 2, constructing a Unetformer module to adjust the injected details through supervised learning and inject the details into the multispectral image; The Unetformer module in step 2 comprises an encoder, a Transformer block and a decoder, and the input of the Unetformer module is the detail difference in step 1, and the specific implementation is as follows: Step 2.1, the encoder adopts 4 layers of convolutional layers, the number of convolutional kernels increases layer by layer to extract multi-dimensional features, and the first 3 layers of convolutional layers are followed by an average pooling operation to reduce the image size, so that the receptive field of the convolutional kernel can cover more parts of the feature map to increase the size of the receptive field, and finally the encoder outputs , C represents the number of channels; Step 2.2, the block embedding module in the Transformer block embeds the feature image into image patches, and flattens it into a sequence of image patches: wherein, , , , is the number of feature channels, H, W are the image size of the image, h, w are the image block size; Step 2.

3. Using three matrices , , Multiplying with the sequence of image patches to map to query vectors , key vectors , value vectors , Splitting the number of feature channels in Q, K, V into multiple attention heads: ​ where , is the number of attention heads, is the number of feature channels for each attention head, , , These three matrices are learnable parameters, initialized randomly; Step 2.

4. For each attention head, compute and On feature similarity on the individual feature channels: in, , Indicates the first Self-similarity between pairs of image patches within a sequence of image patches at attention; Step 2.

5. The correlation between the image block sequences obtained in step 2.4 is normalized by the function: Softmax ​ Step 2.

6. Compute the self-similarity degree for the computed value vector Step 2.

7. Compute the weighted sum of all image block sequences of the value vector to get the output vector of the i-th attention head as follows: Step 2.7, obtaining the output vectors of each attention head through self-attention, and finally concatenating the output vectors of all attention heads to obtain the final output vector: wherein ; Step 2.8, folding the output vector into a feature image through dimension transformation to obtain an output feature image; Step 2.9, the decoder adopts 4 convolutional layers, in the first 3 convolutional layers, the feature image is first increased in size through deconvolution, then concatenated with the output of the corresponding layer of the encoder, and then passed through a convolutional layer, and finally a convolutional operation with 4 convolutional kernels is performed on the fourth layer to map the image from the feature domain back to the image domain; Step 2.10, adding the output of the decoder and the up-sampled multispectral image through a skip connection to inject spatial details into the multispectral image to obtain the final fusion image; Step 3, constructing a supervised objective function to train the image fusion model to learn the spectral and spatial prior knowledge of the remote sensing image to obtain the fusion image, wherein the image fusion model comprises a detail difference extraction module and a Unetformer module; Step 4, constructing an adaptive adjustment network to obtain a reconstructed fusion image, and using a spatial and spectral loss function to unsupervisedly drive the image to adjust the spatial and spectral information; Step 5, training the above image fusion model using the simulation data of one satellite, and using the trained image fusion model to adjust the image through the adaptive adjustment network on other satellites. 2.The method of claim 1, wherein the method comprises: In step 2.1, the number of convolutional kernels of the 4 convolutional layers changes from 32, 64, 128 to 256, and each convolutional layer consists of two two-dimensional convolutions and a LeakyReLu activation function. 3.The method of claim 1, wherein: In step 2.9, the number of convolutional kernels of the first 3 layers changes from 128, 64 to 32, the first 3 convolutional layers consist of two two-dimensional convolutions and a LeakyReLu activation function, and the fourth convolutional layer is a two-dimensional convolution.

4. The method of claim 1, wherein the method is based on a combination of supervised and unsupervised learning. The loss function constructed in step 3 is as follows: wherein, and represent a fused image and a reference image, respectively, is a batch size.

5. The method of claim 1, wherein the method is based on a combination of supervised and unsupervised learning. The specific implementation of step 4 is as follows: Step 4.1, the fusion image is passed through two convolutional layers with increasing number of convolutional kernels to extract image features; Step 4.2, the image features in step 4.1 are passed through two channel attention layers to obtain multi-channel feature images to focus on more important image features; Step 4.3, the multi-channel feature image is remapped to a four-channel image by using two convolution layers, and the original input fusion image is added to obtain the reconstructed fusion image; Step 4.4, a spatial-spectral joint loss function is constructed, and the loss function is as follows: Wherein, HF, HP represent the high-frequency information of the gray fusion image and the panchromatic image respectively, FB represents the blurred fusion image, and MS represents the up-sampled multispectral image.

6. The method of claim 5, wherein the method is based on a combination of supervised and unsupervised learning. The number of channels of the convolution layer in step 4.1 is , 64, and the size of the convolution kernel is ; In step 4.2, the channel attention is obtained by averaging and maximum pooling of the module input, followed by two convolutions, addition, and normalization by the Sigmoid activation function. The output is obtained by multiplying the module input and the attention weight.

7. The method of claim 5, wherein the method is based on a combination of supervised and unsupervised learning. In step 4.4, the gray fusion image is obtained by adding the average of each channel of the fusion image, and the high-frequency information of the gray fusion image and the panchromatic image is obtained by a high-pass filter. The high-pass filter is realized by subtracting the low-frequency content of the original image from the original image, which is obtained by averaging filtering. The average filtering is realized by a global pooling layer. The blurred fusion image is obtained by up-sampling and then down-sampling the original fusion image.

8. The method of claim 1, wherein the method is based on a combination of supervised and unsupervised learning. In step 5, the parameters of the trained fusion model are fixed, and the test results of image adjustment on other satellites are obtained by adaptive adjustment of the network. Then, the test results are compared with existing remote sensing fusion methods through subjective visual and objective evaluation indicators. The objective evaluation indicators include peak signal-to-noise ratio.

Images