Remote sensing image panchromatic sharpening method based on multi-scale dual-domain information fusion technology

Abstract

The present application relates to the remote sensing image panchromatic sharpening method based on multiscale dual-domain information fusion technology, compared with prior art, the defects of easy to produce false image, color distortion, poor generalization of fusion image are solved.The present application includes the following steps: remote sensing image acquisition and pretreatment;building panchromatic sharpening model;training of panchromatic sharpening model;acquisition of the remote sensing image to be sharpened;obtaining of the panchromatic sharpening result of remote sensing image.Compared with the panchromatic sharpening method using spatial domain features, the precision and color performance of fusion image are improved, which is superior to the existing method in various reference indicators, and the image quality is higher.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image processing technology, specifically to a method for panchromatic sharpening of remote sensing images based on multi-scale dual-domain information fusion technology. Background Technology

[0002] In current remote sensing systems, due to the limitations of satellite sensors, it is difficult for a single sensor to obtain high-resolution multispectral images. Therefore, satellites typically carry two sensors to obtain panchromatic (PAN) and multispectral (MS) images respectively.

[0003] Multispectral images contain rich spectral information but have relatively low resolution, while panchromatic images possess fine image texture information but only contain spectral information from a single channel. To obtain high-resolution multispectral images, a common method is to fuse the MS and PAN images; this method is also known as panchromatic sharpening. Because this method is fundamental to many downstream tasks, such as remote sensing image segmentation and object detection, it has received considerable attention.

[0004] Currently used methods can be mainly divided into two categories: traditional methods based on manually designed features and deep learning-based methods. Traditional methods include component substitution, multi-resolution analysis, and variational optimization. Manually designed features cannot fully utilize the information in an image; therefore, deep learning-based methods currently dominate the market.

[0005] However, these methods also suffer from artifacts and color distortion in their output. This is because these methods fuse features at a single scale, which fails to fully extract features from remotely sensed ground objects with large size variations. In addition, these methods focus on processing information in the spatial domain and ignore the information implied in the frequency domain. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies in image fusion, such as artifacts, color distortion, and poor generalization, by providing a remote sensing image panchromatic sharpening method based on multi-scale dual-domain information fusion technology.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A panchromatic sharpening method for remote sensing images based on multi-scale dual-domain information fusion technology includes the following steps:

[0009] 11) Acquisition and preprocessing of remote sensing images: Acquire panchromatic and multispectral images, crop them according to a set size, and downsample the cropped panchromatic and multispectral images by 4 times to generate sample pairs.<MS、PAN、G> Where MS and PAN are the downsampled multispectral and panchromatic images, and G is the multispectral image before downsampling;

[0010] 12) Constructing a full-color sharpening model: Based on the spatial domain information fusion subnetwork and the frequency domain information fusion subnetwork, multi-scale dual-domain information fusion is performed to construct a full-color sharpening model;

[0011] 13) Training the panchromatic sharpening model: Input the preprocessed panchromatic image and multispectral image into the panchromatic sharpening model for training;

[0012] 14) Acquisition of remote sensing images to be sharpened: Acquire the panchromatic and multispectral images to be sharpened and perform preprocessing;

[0013] 15) Obtaining the panchromatic sharpening result of remote sensing image: Input the preprocessed panchromatic image and multispectral image into the trained panchromatic sharpening model to obtain the panchromatic sharpening result of remote sensing image.

[0014] The construction of the full-color sharpening model includes the following steps:

[0015] 21) Construct a multi-scale network based on U-Net, which includes two sub-networks: a spatial information fusion sub-network and a frequency domain information fusion sub-network;

[0016] U-Net is defined as consisting of a shrinking subnetwork and an expanding subnetwork. In the shrinking subnetwork, multi-scale feature information of the image is extracted through convolution and downsampling operations, while in the expanding subnetwork, feature reconstruction is performed through upsampling and feature concatenation techniques. The shrinking subnetwork is defined as a spatial domain information fusion subnetwork, and the expanding subnetwork is defined as a frequency domain information fusion subnetwork.

[0017] 22) Construct a spatial information fusion sub-network, including: convolutional projection module 1, convolutional projection module 2, convolutional feature extraction module 1, convolutional feature extraction module 2, convolutional downsampling module 1, convolutional downsampling module 2, convolutional downsampling module 3, convolutional downsampling module 4, spatial information fusion module 1, spatial information fusion module 2, and spatial information fusion module 3; wherein, the convolutional projection module projects the input image onto the feature space, the convolutional feature extraction module is used to extract image features, the convolutional downsampling module is used to extract multi-scale spatial feature information, and the spatial information fusion module fuses the panchromatic image feature map and the multispectral image feature map at the same scale;

[0018] 23) Construct a frequency domain information fusion subnetwork, which includes upsampling module 1, upsampling module 2, frequency domain information fusion module 1, frequency domain information fusion module 2, frequency domain information fusion module 3, and channel transformation module; wherein, the upsampling module upsamples the output of the previous subnetwork and inputs the upsampled result into the frequency domain information fusion module; the upsampling module is a bicubic interpolation operation, which enlarges the size of the input feature map by a factor of two, and the channel transformation module maps the feature map from the feature space to the output image.

[0019] The training of the full-color sharpening model includes the following steps:

[0020] 31) Define and establish the spatial loss function

[0021]

[0022] Frequency domain loss function and

[0023]

[0024] Overall loss function

[0025]

[0026] Where A and P are the amplitude and phase components after Fourier transform, respectively, Y and G correspond to the network output and the real image, and λ is a hyperparameter.

[0027] 32) Input the preprocessed multispectral image and panchromatic image into the network. The features are fused in the spatial domain fusion sub-network and the frequency domain fusion sub-network respectively. The final result is added to the input multispectral image to obtain the final result: that is, LRMS and PAN are passed through the network to obtain the fusion result F. ms7 , will F ms7 Adding it to LRMS yields the reconstructed output image;

[0028] 321) Read training set image pairs<ms,pan,G> These correspond to low-resolution hyperspectral images, panchromatic images, and high-resolution multispectral images, respectively.

[0029] 322) Input ms and pan into the pan-sharpening model to obtain the output result Y;

[0030] 323) Calculate the loss functions of Y and G based on the spatial domain loss function and the frequency domain loss function;

[0031] 324) Optimize the network weights according to the loss function of the training set and the gradient descent algorithm until the set total number of iterations is reached.

[0032] The construction of the spatial information fusion subnetwork includes the following steps:

[0033] 41) Set the convolutional projection module 1 to consist of 3x3 convolutions, set stride=1, its input is a multispectral image, and its output is the corresponding feature F. ms The convolutional projection module 2 is configured as a 3x3 convolution with stride = 1. Its input is a panchromatic image, and its output is the corresponding feature F. pan ;

[0034] 42) The spatial information fusion module 1 is configured to contain two branches, which interact through a feature extraction unit; the feature extraction unit consists of an HIN module, which takes as input the features F of the MS and PAN images. ms and F pan After processing through two branches, the final result is fused using a 1x1 convolution, producing the fused result F. ms1 Its expression is as follows:

[0035] Given input F ms and F pan The formula definition for the spatial information fusion module is as follows:

[0036] F g =HIN(F pan )

[0037] F′ ms =F ms ×F g

[0038] F″ ms =F′ ms +F g

[0039] F l =HIN(F″ ms )

[0040] F′ pan =F pan ×F l

[0041] F″ pan =F′ pan +F l

[0042] F ms1 =Conv 1×1 (Concat(F″ pan ,F″ ms )),

[0043] In the formula, Conv 1×1Corresponding to a convolution operation of size 1, Concat corresponds to concatenation along the channel dimension, and HIN is the feature extraction unit;

[0044] When input F pan and F ms At that time, first put F pan The extracted features F are obtained through the HIN module. g , will F ms With F g Multiply and add F g We obtain F′ ms ; F″ ms The extracted features F are obtained through the HIN module. l , will F l With F pan Multiply and add F l We obtain F′ pan ; Transform F″ ms With F″ pan The channel dimensions are concatenated, and then a 1x1 convolution is performed to obtain the fused feature F. ms1 ;

[0045] Given input F in the spatial information fusion module pan The formulas for the HIN module are defined as follows:

[0046] F mid =Conv 3×3 (F pan )

[0047]

[0048] F res =Conv 3×3 (R out )

[0049] F g =F res +F pan

[0050] In the formula, ChannelSplit represents copying and evenly dividing along the channel dimension, Concat represents concatenating along the channel dimension, In represents Instance Normalization, and Conv represents Instance Normalization. 3×3 This corresponds to a convolution operation with a kernel size of 3x3;

[0051] When input F pan First, a 3x3 convolution operation is performed to extract features. Then, the extracted features are copied along the channel dimension, and the copied features are evenly divided to obtain... and right After performing Instance Normalization, the two features are concatenated along their channel dimensions and then fused using a 3x3 convolution operation. Finally, the original input features are added to obtain the transformed feature F. g F g That is, F in the spatial information fusion module pan Features extracted by the HIN module;

[0052] Given input F in the spatial information fusion module ms The formulas for the HIN module are defined as follows:

[0053] F mid′ =Conv 3×3 (F ms )

[0054]

[0055] F res′ =Conv 3×3 (R out′ )

[0056] F l =F res′ +F ms

[0057] In the formula, ChannelSplit represents copying and evenly dividing along the channel dimension, Concat represents concatenating along the channel dimension, In represents Instance Normalization, and Conv represents... 3×3 This corresponds to a convolution operation with a kernel size of 3x3;

[0058] When input F ms First, a 3x3 convolution operation is performed to extract features. Then, the extracted features are copied along the channel dimension, and the copied features are evenly divided to obtain... and right After performing Instance Normalization, the two features are concatenated along their channel dimensions and then fused using a 3x3 convolution operation. Finally, the original input features are added to obtain the transformed feature F. l F l That is, F in the spatial information fusion module ms Features extracted by the HIN module;

[0059] 43) Set convolutional downsampling module 1 to consist of 3x3 convolutions, set stride = 2, and the input of convolutional downsampling module 1 is F. ms1 The output is the feature map F.ms1′ The size has been reduced to half of its original size;

[0060] 44) Set the convolutional feature extraction module 1 to consist of 3x3 convolutions, set stride=1, and its input is F. pan The output is F pan′ ;

[0061] 45) Set convolutional downsampling module 2 to consist of 3x3 convolutions, set stride = 2, and the input of convolutional downsampling module 2 is F. pan′ The output is the feature map F. pan1 The size has been reduced to half of its original size;

[0062] 46) Set up airspace information fusion module 2, which has the same structure as airspace information fusion module 1, and its input is F. ms1′ and F pan1 The output F is obtained. ms2 ;

[0063] 47) Set the convolutional downsampling module 3 to consist of 3x3 convolutions, set stride = 2, and the input of the convolutional downsampling module 3 is F. ms2 The output is the feature map F. ms2 The size is reduced to half of its original size.

[0064] 48) Set the convolutional feature extraction module 2 to consist of 3x3 convolutions, set stride=1, and its input is F. pan1 The output is F pan1′ ;

[0065] 49) Set convolutional downsampling module 4 to consist of 3x3 convolutions, set stride = 2, and the input of convolutional downsampling module 2 is F. pan1′ The output is the feature map F. pan2 The size has been reduced to half of its original size;

[0066] 410) Set up the airspace information fusion module 3, which has the same structure as the airspace information fusion module 1, and its input is F. ms2′ and F pan2 The output F is obtained. ms3 ;

[0067] 411) The first part of the spatial information fusion subnetwork is defined as convolutional projection module 1 and convolutional projection module 2; the second part is a sequentially cascaded stack of feature extraction module 1, convolutional downsampling module 2, feature extraction module 2, and convolutional downsampling module 4; the third part is a sequentially cascaded stack of spatial information fusion module 1, convolutional downsampling module 1, spatial information fusion module 2, convolutional downsampling module 3, and spatial information fusion module 3.

[0068] The output F of the first partms and F pan Input the second part and the third part respectively, and output the second part F. pan1 and F pan2 Input the third part, and connect the three parts to complete the construction of the spatial information fusion subnetwork.

[0069] The frequency domain information fusion subnetwork includes the following steps:

[0070] 51) Set up sampling module 1 to perform bicubic interpolation operation, and the input of upsampling module 1 is F. ms3 The output is the feature map F. ms3′ The size is doubled;

[0071] 52) The frequency domain information fusion module 1 is configured to include a Fourier transform, an inverse Fourier transform function, a feature fusion unit, and a feature extraction unit, wherein the feature fusion unit is implemented by a 1x1 convolution and the feature extraction unit is implemented by a 3x3 convolution.

[0072] The input is F ms3′ and F pan1′ In the frequency domain information fusion module, F ms3′ and F pan1′ Perform Fourier transforms separately to generate corresponding amplitude and spectral components. Then fuse these components and perform an inverse Fourier transform to obtain the spatial features. Finally, compare these spatial features with the Fourier transform. ms3′ Perform splicing and fusion;

[0073] Its expression is as follows:

[0074] Given input F pan1′ and F ms3′ The formula for the frequency domain information fusion module is defined as follows:

[0075]

[0076] F mid′ =Conv 3×3 (F ms3′ )

[0077] R out′ =Conv 1×1 (Cat([F mid′ ,F fre′ ])),

[0078] F ms4 =R out′ +F ms3′ ,

[0079] In the formula Corresponding to the Fourier transform, This represents the inverse Fourier transform, with Cat indicating the concatenation of channel dimensions. These are the amplitude and phase components after Fourier transform, respectively. 1×1 Conv 3×3 These correspond to convolution operations of sizes 1x1 and 3x3, respectively;

[0080] When input F ms3′ and F pan1′ At that time, the corresponding phase and amplitude components are generated by Fourier transform, and then fused by convolution operation to generate the fused amplitude. Phase components The fused result is then subjected to inverse Fourier transform to obtain the fused feature map.

[0081] Take the original input F ms3′ Perform convolution operations to extract features and obtain F. mid′ and the feature F fused with the frequency domain fre′ The data is fused to facilitate information exchange between the two domains, and the final result F is obtained through residual connections. ms4 ;

[0082] 53) Set up sampling module 2 to perform bicubic interpolation operation, and the input of upsampling module 2 is F. ms4 The output is the feature map F. mx4′ The size is doubled;

[0083] 54) Configure frequency domain information fusion module 2, following the same operation as frequency domain information fusion module 1, where the input is F. ms4′ and F pan′ The output is F ms5 ;

[0084] 55) Configure frequency domain information fusion module 3, following the same operation as frequency domain information fusion module 1, where the input is F. ms5 and F pan The output is F ms6 ;

[0085] 56) Set the channel transformation module to consist of 3x3 convolution operations, stride=1, and output F. ms7 Adjust the feature map channels to 4;

[0086] 57) Cascade and stack the upsampling module 1, frequency domain information fusion module 1, upsampling module 2, frequency domain information fusion module 2, frequency domain information fusion module 3, and channel conversion module in sequence to complete the construction of the frequency domain information fusion sub-network.

[0087] Beneficial effects

[0088] The remote sensing image pancolor sharpening method based on multi-scale dual-domain information fusion technology of the present invention improves the accuracy and color performance of the fused image compared with the pancolor sharpening method using spatial domain features. It outperforms existing methods in a variety of reference indicators and produces higher quality images.

[0089] This invention addresses the characteristic that frequency domain information can improve model performance in super-resolution tasks. It introduces frequency domain information into the pancolor sharpening method, enabling the use of frequency domain information to improve model performance. Through frequency domain convolution operations and dual-domain fusion networks, the model's performance is enhanced. Attached Figure Description

[0090] Figure 1 This is a sequence diagram of the method of the present invention;

[0091] Figure 2 This is an overall structural diagram of the full-color sharpening model involved in this invention;

[0092] Figure 3 This is a structural diagram of the frequency domain information fusion module and the spatial domain information fusion module involved in this invention;

[0093] Figure 4 This is a diagram illustrating the fusion effect of the present invention. Detailed Implementation

[0094] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0095] like Figure 1 As shown, the remote sensing image panchromatic sharpening method based on multi-scale dual-domain information fusion technology of the present invention includes the following steps:

[0096] The first step is the acquisition and preprocessing of remote sensing images: acquire panchromatic and multispectral images, crop them according to a set size, and then downsample the cropped panchromatic and multispectral images by a factor of 4 to generate sample pairs.<MS、PAN、G> Where MS and PAN are the downsampled multispectral and panchromatic images, respectively, and G is the multispectral image before downsampling.

[0097] The second step is to construct a full-color sharpening model: based on the spatial domain information fusion subnetwork and the frequency domain information fusion subnetwork, multi-scale dual-domain information fusion is performed to construct a full-color sharpening model.

[0098] Existing conventional methods mostly focus on spatial image processing. While this produces considerable output, it also has certain problems. Because spatial modeling of image information can only be concentrated in a small area, it struggles to handle remote sensing images with large scale variations, resulting in blurry details and unclear textures. Simultaneously, due to the underutilization of frequency domain information, the generated images also exhibit spectral distortion. Existing research shows that introducing frequency domain information allows models to better learn high-frequency features and possess long-range modeling capabilities. Therefore, the method described in this invention integrates frequency domain information with multi-scale modeling, enabling the acquisition of global image information. Simultaneously, it utilizes frequency domain features to improve model performance, resulting in images with clearer details and more accurate spectra. It includes the following steps:

[0099] (1) A multi-scale network based on U-Net is constructed, which includes two sub-networks: a spatial information fusion sub-network and a frequency domain information fusion sub-network. Thanks to its ability to extract multi-scale features, U-Net is widely used in the field of image restoration. U-Net is defined as including a shrinking sub-network and an expanding sub-network. In the shrinking sub-network, multi-scale feature information of the image is extracted through convolution and downsampling operations. In the expanding sub-network, feature reconstruction is performed through upsampling and feature concatenation techniques. The shrinking sub-network is defined as the spatial information fusion sub-network and the expanding sub-network as the frequency domain information fusion sub-network.

[0100] (2) Figure 2 As shown, a spatial information fusion subnetwork is constructed, including: convolutional projection module 1, convolutional projection module 2, convolutional feature extraction module 1, convolutional feature extraction module 2, convolutional downsampling module 1, convolutional downsampling module 2, convolutional downsampling module 3, convolutional downsampling module 4, spatial information fusion module 1, spatial information fusion module 2, and spatial information fusion module 3. Among them, the convolutional projection module projects the input image onto the feature space, the convolutional feature extraction module is used to extract image features, the convolutional downsampling module is used to extract multi-scale spatial feature information, and the spatial information fusion module fuses the panchromatic image feature map and the multispectral image feature map of the same scale.

[0101] In this spatial information subnetwork, this invention employs common methods of spatial domain information image fusion to improve image quality. This method preserves key image information while merging the advantages of multiple images, saving spatial details and spectral information. Furthermore, to address ground features with significant scale variations in remote sensing images, this invention designs a downsampling module for modeling multi-scale image features. Reducing the image scale also helps decrease the computational complexity of the model. By using the spatial information fusion subnetwork, we can fuse images at multiple scales, better preserving spatial details.

[0102] A1) Set convolutional projection module 1 to consist of 3x3 convolutions, set stride=1, its input is a multispectral image, and its output is the corresponding feature F. ms The convolutional projection module 2 is configured as a 3x3 convolution with stride = 1. Its input is a panchromatic image, and its output is the corresponding feature F. pan ;

[0103] A2) The spatial information fusion module 1 is configured to contain two branches, which interact through a feature extraction unit; the feature extraction unit consists of an HIN module, which takes as input the features F of the MS and PAN images. ms and F pan After processing through two branches, the final result is fused using a 1x1 convolution, producing the fused result F. ms1 Its expression is as follows:

[0104] Given input F ms and F pan The formula definition for the spatial information fusion module is as follows:

[0105] F g =HIN(F pan )

[0106] F′ ms =F ms ×F g

[0107] F″ ms =F′ ms +F g

[0108] F l =HIN(F″ ms )

[0109] F′ pan =F pan ×F l

[0110] F″ pan =F′ pan +F l

[0111] F ms1 =Conv 1×1 (Concat(F″ pan ,F″ ms )),

[0112] In the formula, Conv 1×1 Corresponding to a convolution operation of size 1, Concat corresponds to concatenation along the channel dimension, and HIN is the feature extraction unit;

[0113] When input Fpan and F ms At that time, first put F pan The extracted features F are obtained through the HIN module. g , will F ms With F g Multiply and add F g We obtain F′ ms ; F″ ms The extracted features F are obtained through the HIN module. l , will F l With F pan Multiply and add F l We obtain F′ pan ; Transform F″ ms With F″ pan The channel dimensions are concatenated, and then a 1x1 convolution is performed to obtain the fused feature F. ms1 ;

[0114] Given input F in the spatial information fusion module pan The formulas for the HIN module are defined as follows:

[0115] F mid =Conv 3×3 (F pan )

[0116]

[0117] F res =Conv 3×3 (R out )

[0118] F g =F res +F pan

[0119] In the formula, ChannelSplit represents copying and evenly dividing along the channel dimension, Concat represents concatenating along the channel dimension, In represents Instance Normalization, and Conv represents Instance Normalization. 3×3 This corresponds to a convolution operation with a kernel size of 3x3;

[0120] When input F pan First, a 3x3 convolution operation is performed to extract features. Then, the extracted features are copied along the channel dimension, and the copied features are evenly divided to obtain... and right After performing Instance Normalization, the two features are concatenated along their channel dimensions and then fused using a 3x3 convolution operation. Finally, the original input features are added to obtain the transformed feature F. g F g That is, F in the spatial information fusion module pan Features extracted by the HIN module;

[0121] Given input F in the spatial information fusion module ms The formulas for the HIN module are defined as follows:

[0122] F mid′ =Conv 3×3 (F ms )

[0123]

[0124] F res′ =Conv 3×3 (R out′ )

[0125] F l =F res′ +F ms

[0126] In the formula, ChannelSplit represents copying and evenly dividing along the channel dimension, Concat represents concatenating along the channel dimension, In represents Instance Normalization, and Conv represents... 3×3 This corresponds to a convolution operation with a kernel size of 3x3;

[0127] When input F ms First, a 3x3 convolution operation is performed to extract features. Then, the extracted features are copied along the channel dimension, and the copied features are evenly divided to obtain... and right After performing Instance Normalization, the two features are concatenated along their channel dimensions and then fused using a 3x3 convolution operation. Finally, the original input features are added to obtain the transformed feature F. l F l That is, F in the spatial information fusion module ms Features extracted by the HIN module;

[0128] A3) Set convolutional downsampling module 1 to consist of 3x3 convolutions, set stride = 2, and the input of convolutional downsampling module 1 is F. ms1 The output is the feature map F. ms1′The size has been reduced to half of its original size;

[0129] A4) Configure convolutional feature extraction module 1 as a 3x3 convolution, setting stride = 1, and its input is F. pan The output is F pan′ ;

[0130] A5) Set convolutional downsampling module 2 to consist of 3x3 convolutions, set stride = 2, and the input of convolutional downsampling module 2 is F. pan′ The output is the feature map F. pan1 The size has been reduced to half of its original size;

[0131] A6) Set up the airspace information fusion module 2, which has the same structure as the airspace information fusion module 1, and its input is F. ms1′ and F pan1 The output F is obtained. ms2 ;

[0132] A7) Set the convolutional downsampling module 3 to consist of 3x3 convolutions, set stride = 2, and the input of the convolutional downsampling module 3 is F. ms2 The output is the feature map F. ms2 The size is reduced to half of its original size.

[0133] A8) Set the convolutional feature extraction module 2 to consist of 3x3 convolutions, set stride = 1, and its input is F. pan1 The output is F pan1′ ;

[0134] A9) Set convolutional downsampling module 4 to consist of 3x3 convolutions, set stride = 2, and the input of convolutional downsampling module 2 is F. pan1′ The output is the feature map F. pan2 The size has been reduced to half of its original size;

[0135] A10) Set up the airspace information fusion module 3, which has the same structure as the airspace information fusion module 1, and its input is F. ms2′ and F pan2 The output F is obtained. ms3 ;

[0136] A11) The first part of the spatial information fusion subnetwork is defined as convolutional projection module 1 and convolutional projection module 2; the second part is a sequentially cascaded stack of feature extraction module 1, convolutional downsampling module 2, feature extraction module 2, and convolutional downsampling module 4; the third part is a sequentially cascaded stack of spatial information fusion module 1, convolutional downsampling module 1, spatial information fusion module 2, convolutional downsampling module 3, and spatial information fusion module 3.

[0137] The output F of the first part ms and Fpsn Input the second part and the third part respectively, and output the second part F. pan1 and F pan2 Input the third part, and connect the three parts to complete the construction of the spatial information fusion subnetwork.

[0138] (3) Construct a frequency domain information fusion subnetwork, which includes an upsampling module 1, an upsampling module 2, a frequency domain information fusion module 1, a frequency domain information fusion module 2, a frequency domain information fusion module 3, and a channel transformation module; wherein, the upsampling module upsamples the output of the previous subnetwork and inputs the upsampled result into the frequency domain information fusion module; the upsampling module is a bicubic interpolation operation, which enlarges the size of the input feature map by a factor of two, and the channel transformation module maps the feature map from the feature space to the output image.

[0139] After initial information fusion in the spatial domain, frequency domain information will be introduced for frequency domain fusion. Spectral and phase analysis after Fourier transform reveals that, in terms of amplitude, the PAN image, compared to the high-resolution multispectral image, primarily lacks low-frequency information, while the LRMS image, compared to the high-resolution multispectral image, lacks information in both high and low frequencies. However, in terms of phase, the PAN image retains most of the key information compared to the high-resolution multispectral image, while the LRMS image suffers from severe phase loss. Therefore, we can fuse the frequency domain information from PAN features and the frequency domain information from LRMS features to facilitate image reconstruction.

[0140] like Figure 3 As shown, this invention employs Fourier transform to separate phase and amplitude information, which are then fused separately to supplement the frequency domain information of the PAN features into the frequency domain information of the LRMS features. The specific steps are as follows:

[0141] B1) Set up sampling module 1 to perform bicubic interpolation, and the input of upsampling module 1 is F. ms3 The output is the feature map F. ms3′ The size is doubled;

[0142] B2) The frequency domain information fusion module 1 is configured to include a Fourier transform, an inverse Fourier transform function, a feature fusion unit, and a feature extraction unit, wherein the feature fusion unit is implemented by a 1x1 convolution and the feature extraction unit is implemented by a 3x3 convolution.

[0143] The input is F ms3′ and F pan1′ In the frequency domain information fusion module, F ms3′ and F pan1′Perform Fourier transforms separately to generate corresponding amplitude and spectral components. Then fuse these components and perform an inverse Fourier transform to obtain the spatial features. Finally, compare these spatial features with the Fourier transform. ms3′ Perform splicing and fusion;

[0144] Its expression is as follows:

[0145] Given input F pan1′ and F ms3′ The formula for the frequency domain information fusion module is defined as follows:

[0146]

[0147] F mid′ =Conv 3×3 (F ms3′ )

[0148] R oit′ =Conv 1×1 (Cat([F mid′ ,F fre′ ])),

[0149] F ms4 =R out′ +F ms3′ ,

[0150] In the formula Corresponding to the Fourier transform, This represents the inverse Fourier transform, with Cat indicating the concatenation of channel dimensions. These are the amplitude and phase components after Fourier transform, respectively. 1×1 Conv 3×3 These correspond to convolution operations of sizes 1x1 and 3x3, respectively;

[0151] When input F ms3′ and F pan1′ At that time, the corresponding phase and amplitude components are generated by Fourier transform, and then fused by convolution operation to generate the fused amplitude. Phase components The fused result is then subjected to inverse Fourier transform to obtain the fused feature map.

[0152] Take the original input F ms3′ Perform convolution operations to extract features and obtain F. mid′ and the feature F fused with the frequency domain fre′ The data is fused to facilitate information exchange between the two domains, and the final result F is obtained through residual connections. ms4 ;

[0153] B3) Set upsampling module 2 to perform bicubic interpolation, and the input of upsampling module 2 is F. ms4 The output is the feature map F. ms4′ The size is doubled;

[0154] B4) Configure frequency domain information fusion module 2, following the same operation as frequency domain information fusion module 1, where the input is F. ms4′ and F pan′ The output is F ms5 ;

[0155] B5) Configure frequency domain information fusion module 3, following the same operation as frequency domain information fusion module 1, where the input is F. ms5 and F pan The output is F ms6 ;

[0156] B6) Set the channel transformation module, which consists of 3x3 convolution operations, stride=1, and the output is F. ms7 Adjust the feature map channels to 4;

[0157] B7) Cascade and stack the upsampling module 1, frequency domain information fusion module 1, upsampling module 2, frequency domain information fusion module 2, frequency domain information fusion module 3, and channel conversion module in sequence to complete the construction of the frequency domain information fusion sub-network.

[0158] The third step is training the panchromatic sharpening model: input the preprocessed panchromatic image and multispectral image into the panchromatic sharpening model for training.

[0159] (1) Define and establish the spatial loss function

[0160]

[0161] Frequency domain loss function and

[0162]

[0163] Overall loss function

[0164]

[0165] Where A and P are the amplitude and phase components after Fourier transform, respectively, Y and G correspond to the network output and the real image, and λ is a hyperparameter.

[0166] (2) Input the preprocessed multispectral image and panchromatic image into the network. The features are fused in the spatial domain fusion subnetwork and the frequency domain fusion subnetwork respectively. The final result is added to the input multispectral image to obtain the final result: that is, LRMS and PAN are passed through the network to obtain the fusion result F. ms7 , will F ms7 Adding it to LRMS yields the reconstructed output image;

[0167] C1) Read training set images<ms,pan,G> These correspond to low-resolution hyperspectral images, panchromatic images, and high-resolution multispectral images, respectively.

[0168] C2) Input ms and pan into the pan-sharpening model to obtain the output result Y;

[0169] C3) Calculate the loss functions of Y and G based on the spatial domain loss function and the frequency domain loss function;

[0170] C4) Optimize the network weights using the gradient descent algorithm based on the loss function of the training set until the set total number of iterations is reached.

[0171] The fourth step is to acquire the remote sensing image to be sharpened: acquire the panchromatic image and multispectral image to be sharpened and perform preprocessing.

[0172] Step 5: Obtaining the panchromatic sharpening result of the remote sensing image: Input the preprocessed panchromatic image and multispectral image into the trained panchromatic sharpening model to obtain the panchromatic sharpening result of the remote sensing image.

[0173] Here, using PAN and LRMS as shown in Figure 4, we obtain the following: Figure 4 The image fusion effect shown.

[0174] Using the original multispectral MS image as a reference, the quality of the fused image is evaluated using evaluation metrics. For the true fused image, the fused image is compared with the original MS or panchromatic PAN image, and the corresponding evaluation metrics are calculated to assess the quality of the fused image. Figure 4 As can be seen from the image, the method in this patent has the clearest details and the most realistic spectral features compared to other methods. It can also be seen from the residual plot in the last row of the image that the residual plot of this invention has the least brightness, indicating that the fusion result has the best quality.

[0175] In simulation experiments, this invention uses the World-View II, World-View III, and Gaofen-2 datasets to evaluate the model. Tables 1, 2, and 3 show the evaluation metrics of the model. The evaluation metrics used are described below:

[0176] Spectral Angle Mapper (SAM): Measures the spectral similarity between the fused image and the corresponding reference image. Its value is a positive number greater than or equal to 0. The smaller the value, the higher the spectral similarity between the two images.

[0177] ERGAS (Erreur Relative Globale Adimensionnellede Synthèse): measures the global spectral quality of the fused image. Its value is a positive number greater than or equal to 0. The smaller the value, the smaller the spectral difference between the two images.

[0178] Relative Average Spectral Error (RASE): Reflects the global spectral quality of the fused image. Its value is a positive number greater than or equal to 0. The smaller the value, the lower the degree of spectral distortion of the fused image.

[0179] Spatial Correlation Coefficient (SCC): Measures the spatial detail similarity between the fused image and the reference image. Its value is between 0 and 1. The closer the value is to 1, the stronger the spatial correlation between the two images and the higher the spatial quality of the fused image.

[0180] The general image quality metric Q measures the correlation, average brightness similarity, and contrast similarity between the fused image and the reference image. Its value is between 0 and 1. The closer the Q value is to 1, the higher the quality of the fused image.

[0181] Structural Similarity (SSIM): This measure globally evaluates the similarity between two images in three aspects: luminance, contrast, and structure. The value ranges from 0 to 1, with a higher similarity between the two images being closer to 1.

[0182] Peak signal-to-noise ratio (PSNR) is a logarithmic value that measures the mean square error between the fused image and the reference image. A value greater than 0 indicates that the fused image is closer to the reference image.

[0183] Dλ spectral distortion index, a measure of the degree of spectral distortion.

[0184] Ds Space Quality Index

[0185] The quality of the fusion results obtained by the network was evaluated using the above seven indicators and compared with a variety of existing advanced full-color sharpening algorithms. Tables 1, 2 and 3 show the comparison results of World-View II, Gaofen-2 and World-View III, with the best results marked in bold.

[0186] Table 1: Quantitative Results Based on the WorldView-II Dataset

[0187] Methods PSNR SSIM SAM ERGAS SCC Q Dλ DS QNR SFIM 34.1297 0.8975 0.0439 2.3449 0.9079 0.6064 0.0915 0.1277 0.7942 GS 35.6376 0.9176 0.0423 1.8774 0.9225 0.6307 0.0607 0.1285 0.8195 Brovey 35.8646 0.9216 0.0403 1.8238 0.8913 0.6163 0.077 0.136 0.7977 HIS 35.2962 0.9027 0.0461 2.0278 0.8534 0.5704 0.0774 0.1578 0.777 GFPCA 33.4558 0.9038 0.0488 2.1401 0.8924 0.4665 0.1016 0.1656 0.7508 PNN 40.755 0.9624 0.0259 1.0646 0.9677 0.7426 0.065 0.1186 0.825 PANNet 40.8176 0.9626 0.0257 1.0557 0.968 0.7437 0.0645 0.1189 0.8252 MSDCNN 41.3355 0.9664 0.0242 0.994 0.9721 0.7577 0.0635 0.1172 0.8276 SRPPNN 41.4538 0.9679 0.0233 0.9899 0.9729 0.7691 0.0637 0.1164 0.8281 GPPNN 41.1622 0.9684 0.0244 1.0315 0.9722 0.7627 0.0642 0.1163 0.8278 This patent model 41.8435 0.9711 0.0222 0.9478 0.9757 0.777 0.0631 0.1144 0.8305

[0188] Table 2: Quantitative results based on the Gaofen-II dataset

[0189] Methods PSNR SSIM SAM ERGAS SCC Q Dλ DS QNR SFIM 36.906 0.8882 0.0318 1.7398 0.8128 0.4349 0.0691 0.1312 0.8109 GS 37.226 0.9034 0.0309 1.6736 0.7851 0.4211 0.0397 0.1214 0.8445 Brovey 37.7974 0.9026 0.0218 1.372 0.6446 0.3857 0.0905 0.1443 0.779 HIS 38.1754 0.91 0.0243 1.5336 0.6738 0.3682 0.0418 0.1345 0.8301 GFPCA 37.9443 0.9204 0.0314 1.5604 0.8032 0.3236 0.0898 0.1815 0.7445 PNN 43.1208 0.9704 0.0172 0.8528 0.94 0.739 0.0387 0.1162 0.8494 PANNet 43.0659 0.9685 0.0178 0.8577 0.9402 0.7309 0.0369 0.1219 0.8455 MSDCNN 45.6874 0.9827 0.0135 0.6389 0.9526 0.7759 0.0368 0.1112 0.856 SRPPNN 47.1998 0.9877 0.0106 0.5586 0.9564 0.79 0.0364 0.1087 0.8588 GPPNN 44.2145 0.9815 0.0137 0.7361 0.951 0.7721 0.0362 0.1078 0.8612 This patent model 47.4101 0.9895 0.0101 0.5414 0.9618 0.8035 0.0346 0.1057 0.8633

[0190] Table 3: Quantitative Results Based on the WorldView-III Dataset

[0191] Methods PSNR SSIM SAM ERGAS SCC Q Dλ DS QNR SFIM 21.8212 0.5457 0.1208 8.973 0.6952 0.4531 0.0448 0.1265 0.8347 GS 22.5608 0.547 0.1217 8.2433 0.7131 0.4411 0.035 0.2011 0.7695 Brovey 22.506 0.5466 0.1159 8.2331 0.7033 0.4394 0.0481 0.2006 0.7603 HIS 22.5579 0.5354 0.1266 8.3616 0.6994 0.4301 0.0356 0.2073 0.7634 GFPCA 22.34 0.4826 0.1294 8.3964 0.6987 0.3115 0.0528 0.1214 0.8325 PNN 29.9418 0.9121 0.0824 3.3206 0.954 0.8679 0.046 0.0933 0.8654 PANNet 29.684 0.9072 0.0851 3.4263 0.9512 0.8631 0.0474 0.0942 0.8634 MSDCNN 30.3038 0.9184 0.0782 3.1884 0.9577 0.8763 0.0432 0.0877 0.8732 SRPPNN 30.4346 0.9202 0.077 3.1553 0.9581 0.8776 0.0414 0.0909 0.8719 GPPNN 30.1785 0.9175 0.0776 3.2593 0.9569 0.8739 0.0438 0.0936 0.8671 This patent model 30.8645 0.9258 0.0757 2.9851 0.9624 0.8856 0.0411 0.0899 0.8732

[0192] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for panchromatic sharpening of remote sensing images based on multi-scale dual-domain information fusion technology, characterized in that, Includes the following steps: 11) Acquisition and preprocessing of remote sensing images: Acquire panchromatic and multispectral images, crop them according to a set size, and downsample the cropped panchromatic and multispectral images by 4 times to generate sample pairs.<MS、PAN、G> Where MS and PAN are the downsampled multispectral and panchromatic images, and G is the multispectral image before downsampling; 12) Constructing a full-color sharpening model: Based on the spatial domain information fusion subnetwork and the frequency domain information fusion subnetwork, multi-scale dual-domain information fusion is performed to construct a full-color sharpening model; The construction of the full-color sharpening model includes the following steps: 121) Construct a multi-scale network based on U-Net, which includes two sub-networks: a spatial information fusion sub-network and a frequency domain information fusion sub-network; U-Net is defined as consisting of a shrinking subnetwork and an expanding subnetwork. In the shrinking subnetwork, multi-scale feature information of the image is extracted through convolution and downsampling operations, while in the expanding subnetwork, feature reconstruction is performed through upsampling and feature concatenation techniques. The shrinking subnetwork is defined as a spatial domain information fusion subnetwork, and the expanding subnetwork is defined as a frequency domain information fusion subnetwork. 122) Construct a spatial information fusion sub-network, including: convolutional projection module 1, convolutional projection module 2, convolutional feature extraction module 1, convolutional feature extraction module 2, convolutional downsampling module 1, convolutional downsampling module 2, convolutional downsampling module 3, convolutional downsampling module 4, spatial information fusion module 1, spatial information fusion module 2, and spatial information fusion module 3; wherein, the convolutional projection module projects the input image onto the feature space, the convolutional feature extraction module is used to extract image features, the convolutional downsampling module is used to extract multi-scale spatial feature information, and the spatial information fusion module fuses the panchromatic image feature map and the multispectral image feature map at the same scale; 123) Construct a frequency domain information fusion subnetwork, which includes upsampling module 1, upsampling module 2, frequency domain information fusion module 1, frequency domain information fusion module 2, frequency domain information fusion module 3, and channel transformation module; wherein, the upsampling module upsamples the output of the previous subnetwork and inputs the upsampled result into the frequency domain information fusion module; the upsampling module is a bicubic interpolation operation, which enlarges the size of the input feature map by a factor of two, and the channel transformation module maps the feature map from the feature space to the output image; 13) Training the panchromatic sharpening model: Input the preprocessed panchromatic image and multispectral image into the panchromatic sharpening model for training; 14) Acquisition of remote sensing images to be sharpened: Acquire the panchromatic and multispectral images to be sharpened and perform preprocessing; 15) Obtaining the panchromatic sharpening result of remote sensing image: Input the preprocessed panchromatic image and multispectral image into the trained panchromatic sharpening model to obtain the panchromatic sharpening result of remote sensing image.

2. The method according to claim 1, characterized in that, The training of the full-color sharpening model includes the following steps: 21) Set up the airspace loss function , ， Frequency domain loss function and : ， ， Overall loss function : ， Where A and P are the amplitude and phase components after Fourier transform, respectively, Y and G correspond to the network output and the real image, and λ is a hyperparameter. 22) input the preprocessed multispectral image and panchromatic image into the network, respectively fused in the spatial domain fusion subnetwork and the frequency domain fusion subnetwork, and finally add the input multispectral image to the obtained result to obtain the final result: that is, the LRMS and the PAN are input through the network to obtain the fusion result , the output image after reconstruction is obtained by adding the LRMS. 221) Read training set image pairs<ms,pan,G> These correspond to low-resolution hyperspectral images, panchromatic images, and high-resolution multispectral images, respectively. 222) Input ms and pan into the pan-sharpening model to obtain the output result Y; 223) Calculate the loss functions of Y and G based on the spatial domain loss function and the frequency domain loss function; 224) Optimize the network weights according to the loss function of the training set and the gradient descent algorithm until the set total number of iterations is reached.

3. The method of claim 1, wherein the method is based on a multi-scale dual-domain information fusion technique. The construction of the spatial information fusion subnetwork includes the following steps: 31) Set the convolutional projection module 1 consists of 3x3 convolution, set stride = 1, its input is the multispectral image, the output is the corresponding feature Set the convolutional projection module 2 consists of 3x3 convolution, set stride = 1, its input is the panchromatic image, the output is the corresponding feature , 32) Set airspace information fusion module 1 contains two branches, two branches interact through feature extraction unit; wherein the feature extraction unit is composed of HIN module, the features of input MS and PAN image and , after two branch processing, finally fused by 1x1 convolution, produce fusion results ; Its expression is as follows: Given input and The formula definition of the spatial information fusion module is as follows ， ， ， ， ， ， ， In the formula, corresponding to a convolution operation with a size of 1, corresponding to concatenation of the channel dimension, is a feature extraction unit; When input and , first pass through the module to obtain the extracted features , multiply and , and add to obtain ; pass through the module to obtain the extracted features , multiply and , and add to obtain ; and concatenate the channel dimensions of the transformed and , and pass through a 1x1 convolution to obtain the fused features ; Given input in the spatial information fusion module ,in The formula for the module is defined as follows: ， ， ， ， ， In the formula, To replicate and evenly distribute along the channel dimension, This is for splicing along the channel dimension. For Instance Normalization, This corresponds to a convolution operation with a kernel size of 3x3; When input First, a 3x3 convolution operation is performed to extract features. Then, the extracted features are copied along the channel dimension, and the copied features are evenly divided to obtain... and ,right After performing Instance Normalization, the two features are concatenated along their channel dimensions and then fused using a 3x3 convolution operation. Finally, the original input features are added to obtain the transformed features. , That is, in the spatial information fusion module go through Features extracted by the module; Given input in the spatial information fusion module ,in The formula for the module is defined as follows: ， ， ， ， ， in the formula To replicate and evenly distribute along the channel dimension, This is for splicing along the channel dimension. For Instance Normalization, This corresponds to a convolution operation with a kernel size of 3x3; When input First, a 3x3 convolution operation is performed to extract features. Then, the extracted features are copied along the channel dimension, and the copied features are evenly divided to obtain... and ,right After performing Instance Normalization, the two features are concatenated along their channel dimensions and then fused using a 3x3 convolution operation. Finally, the original input features are added to obtain the transformed features. , That is, in the spatial information fusion module go through Features extracted by the module; 33) Set convolutional downsampling module 1 to consist of 3x3 convolutions, set stride=2, and the input of convolutional downsampling module 1 is... The output is a feature map. The size has been reduced to half of its original size; 34) Set the convolutional feature extraction module 1 to consist of 3x3 convolutions, set stride=1, and its input is... The output is , 35) Set convolutional downsampling module 2 to consist of 3x3 convolutions, set stride=2, and the input of convolutional downsampling module 2 is... The output is a feature map. The size has been reduced to half of its original size; 36) Set up airspace information fusion module 2, which has the same structure as airspace information fusion module 1, and its input is... and , get output ; 37) Set the convolutional downsampling module 3 to consist of 3x3 convolutions, set stride=2, and the input of the convolutional downsampling module 3 is... The output is a feature map. The size has been reduced to half of its original size. 38) Set convolutional feature extraction module 2 to consist of 3x3 convolutions, set stride=1, and its input is... The output is ; 39) Set convolutional downsampling module 4 to consist of 3x3 convolutions, set stride=2, and the input of convolutional downsampling module 2 is... The output is a feature map. The size has been reduced to half of its original size; 310) Set up the airspace information fusion module 3, which has the same structure as the airspace information fusion module 1, and its input is... and , get output , 311) The first part of the spatial information fusion subnetwork is defined as convolutional projection module 1 and convolutional projection module 2; the second part is a sequentially cascaded stack of feature extraction module 1, convolutional downsampling module 2, feature extraction module 2, and convolutional downsampling module 4; the third part is a sequentially cascaded stack of spatial information fusion module 1, convolutional downsampling module 1, spatial information fusion module 2, convolutional downsampling module 3, and spatial information fusion module 3. The output of the first part and Input the second part and the third part respectively, and output the second part. and Input the third part, and connect the three parts to complete the construction of the spatial information fusion subnetwork.

4. The remote sensing image panchromatic sharpening method based on multi-scale dual-domain information fusion technology according to claim 1, characterized in that, The frequency domain information fusion subnetwork includes the following steps: 41) Set up sampling module 1 to perform bicubic interpolation operation. The input of upsampling module 1 is... The output is a feature map. The size is doubled; 42) The frequency domain information fusion module 1 is set to include Fourier transform, inverse Fourier transform function, feature fusion unit, and feature extraction unit, wherein the feature fusion unit is implemented by 1x1 convolution and the feature extraction unit is implemented by 3x3 convolution; Input is and In the frequency domain information fusion module, and Perform Fourier transforms separately to generate corresponding amplitude and spectral components. Merge the corresponding components from both, then perform an inverse Fourier transform to obtain the spatial domain features. Finally, combine the spatial domain features with... Perform splicing and fusion; Its expression is as follows: Given input and The formula for the frequency domain information fusion module is defined as follows: ， ， ， ， ， ， ， ， In the formula Corresponding to the Fourier transform, This is the inverse Fourier transform. For channel-dimensional splicing, , These are the amplitude and phase components after Fourier transform, respectively. , These correspond to convolution operations of sizes 1x1 and 3x3, respectively; When input and At that time, the corresponding phase and amplitude components are generated by Fourier transform, and then fused by convolution operation to generate the fused amplitude. Phase components The fused result is then subjected to inverse Fourier transform to obtain the fused feature map. The original input Perform convolution operations to extract features, and obtain and features fused with the frequency domain The data is fused to facilitate information exchange between the two domains, and the final result is obtained through residual connections. ; 43) Set up sampling module 2 to perform bicubic interpolation operation. The input of upsampling module 2 is... The output is a feature map. The size is doubled; 44) Configure frequency domain information fusion module 2, following the same operation as frequency domain information fusion module 1, where the input is... and The output is ; 45) Configure frequency domain information fusion module 3, following the same operation as frequency domain information fusion module 1, where the input is... and The output is , 46) Set the channel transformation module to consist of 3x3 convolution operations, stride=1, and the output is... Adjust the feature map channels to 4; 47) Cascade and stack the upsampling module 1, frequency domain information fusion module 1, upsampling module 2, frequency domain information fusion module 2, frequency domain information fusion module 3, and channel conversion module in sequence to complete the construction of the frequency domain information fusion sub-network.

Images