A dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement.

Abstract

This invention discloses a dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement, comprising: acquiring PAN and LRMS images and preprocessing them; obtaining an LLMS image after preprocessing the LRMS image; constructing a multispectral image super-resolution enhancement sub-network and training it in the low spatial resolution domain; loading the trained multispectral image super-resolution enhancement sub-network and performing super-resolution enhancement on the LRMS image through it to obtain an HLMS image; constructing a frequency detail injection module and inputting the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain an HRMS image. This invention significantly improves the spatial quality of the generated image while ensuring excellent spectral performance, effectively alleviating problems such as spatial detail blurring and object distortion in the generated image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement. Background Technology

[0002] Panchromatic image sharpening is a technique that utilizes the high-resolution characteristics of panchromatic images to fuse them with multispectral images, thereby improving the spatial resolution of the multispectral image while preserving its spectral information. It is widely considered an effective image enhancement method for improving the spatial representation capabilities of remote sensing images. It aims to overcome the limitation of single satellite sensors in acquiring high spatial resolution multispectral images, obtaining remote sensing image data with both high spatial and high spectral resolution without changing the hardware. Due to the differences between panchromatic and multispectral images in terms of space, radiometrics, and spectral properties, and the susceptibility of the imaging process to atmospheric conditions, surface characteristics, and imaging geometry, panchromatic sharpening faces numerous challenges, making it a research hotspot in the field of remote sensing that is both valuable in application and highly challenging.

[0003] Methods for panchromatic sharpening of remote sensing images can be broadly categorized into two types: traditional methods and deep learning methods. Traditional methods consist of component substitution, multi-resolution analysis, and variational optimization. These methods require manual design of image features and filters, and different features and filters are needed for different scenes and data, consuming significant manpower and time. In contrast, deep learning-based panchromatic sharpening methods can automatically learn image features and filters from large datasets, exhibiting better adaptability and universality. Therefore, deep learning-based panchromatic sharpening methods have become a popular research direction in this field and have achieved significant progress.

[0004] However, the images obtained by these deep learning methods still have some shortcomings in terms of quality, such as blurred spatial details and object distortion. The main reason for these problems is that most existing methods use interpolation algorithms when enlarging multispectral images. While this simple interpolation operation can improve the image resolution, it ignores the rich spatial detail information contained in multispectral images, resulting in blurred details and object distortion in pancolor-sharpened images. In addition, these methods focus on training the model on images with reduced resolution and then directly applying the trained model to the pancolor sharpening task of full-resolution images. Although this strategy simplifies the model training process, it ignores the differences in high-frequency detail features between full-resolution and low-resolution images, making it difficult for the trained model to fit the high-frequency detail features of the full-resolution image well, thus affecting the quality of the sharpened image. Summary of the Invention

[0005] To overcome the shortcomings of the above technologies, this invention provides a dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement.

[0006] Terminology Explanation:

[0007] 1. LRMS: Low-resolution Multispectral.

[0008] 2. PAN: Panchromatic, full-color image.

[0009] 3. LLMS: Lower Low Resolution Multispectral, a multispectral image with lower spatial resolution.

[0010] 4. HLMS: High-resolution Low-quality Multispectral.

[0011] 5. HRMS: High-resolution Multispectral, high spatial resolution multispectral image.

[0012] 6. HMS: High-quality multispectral image.

[0013] The technical solution adopted by this invention to overcome its technical problems is:

[0014] A dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement includes the following steps:

[0015] S1. Acquire remote sensing satellite images, which include panchromatic images and low spatial resolution multispectral images taken by the satellite, denoted as PAN images and LRMS images respectively. Preprocess the PAN images and LRMS images, wherein the LRMS images are preprocessed to obtain LLMS images, and the spatial resolution of the LLMS images is lower than that of the LRMS images.

[0016] S2. Construct a multispectral image super-resolution enhancement sub-network and train the multispectral image super-resolution enhancement sub-network in the low spatial resolution domain.

[0017] S3. Load the multispectral image super-resolution enhancement subnetwork trained in step S2, and use the multispectral image super-resolution enhancement subnetwork to perform super-resolution on the LRMS image, thereby obtaining a high spatial resolution low quality multispectral image, i.e., an HLMS image.

[0018] S4. Construct a frequency detail injection module and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, i.e., an HRMS image.

[0019] Further, in step S1, the PAN image and LRMS image are preprocessed, specifically including:

[0020] The PAN image is cropped to a size of H×W with 1 channel, where H and W represent the height and width of the image, respectively.

[0021] The LRMS image is cropped to a size of [size missing]. The number of channels is C, where r represents the image magnification factor; then, the LRMS image is downsampled by a factor of r to obtain the LLMS image, the height of which is... Width The number of channels is C.

[0022] Furthermore, step S2 specifically includes:

[0023] S2.1. Copy the input LLMS image, copying the number of channels of the LLMS image with C channels to C×r. 2 , get one The image; then a pixel rearrangement is obtained. The image; wherein, the two operations of channel copying and pixel rearrangement are both completed by a convolutional neural network, and the convolutional neural network used for channel copying and pixel rearrangement constitutes the multispectral image super-resolution enhancement sub-network; the multispectral image super-resolution enhancement sub-network is expressed by formulas (1) and (2) as follows:

[0024]

[0025] In formulas (1) and (2), M represents the multispectral image; C represents the number of channels in the image; U pc This indicates that channels of the multispectral image M are copied, with a copy factor of r. 2 ;P sf This indicates pixel rearrangement, which enlarges the height and width of the multispectral image M;

[0026] S2.2 Training the multispectral image super-resolution enhancement sub-network by designing a loss function L lspe Implementation, loss function L lspe The following can be expressed using formula (3):

[0027] L lspe =||LRMS-P sf (U pc(LLMS))||1 (3)

[0028] In formula (3), LLMS is the input of the multispectral image super-resolution enhancement subnetwork, and LRMS is the supervision signal for training the multispectral image super-resolution enhancement subnetwork.

[0029] Furthermore, step S3 specifically includes:

[0030] By loading the multispectral image super-resolution enhancement subnetwork trained in step S2, the trained multispectral image super-resolution enhancement subnetwork is used to super-resolution the LRMS image, so that the height and width of the LRMS image are both increased by r times, resulting in a high spatial resolution low quality multispectral image, namely the HLMS image. The size of the HLMS image is H×W and the number of channels is C.

[0031] Further, step S4 includes:

[0032] S4.1 Processing HLMS and PAN images;

[0033] S4.2 Construct and train the frequency detail injection module. Input the features of the HLMS image and PAN image processed in step S4.1 into the trained frequency detail injection module and output the HMS image.

[0034] S4.3. The HMS image and the HLMS image obtained in step S3 are weighted and added together using an unsupervised learning method to obtain a high spatial resolution multispectral image, i.e., the HRMS image.

[0035] Furthermore, step S4.1 specifically includes:

[0036] Two convolutional layers are used to convolve the input HLMS image and PAN image respectively to extract features, so that the number of channels in both the HLMS image and PAN image becomes 32.

[0037] Furthermore, step S4.2 specifically includes:

[0038] A frequency detail injection module is constructed and trained, which incorporates the features F from the HLMS image after feature extraction in step S4.1. HLMS and the features F of the PAN image PAN Input to the frequency detail injection module: First, it undergoes a Fourier transform to obtain F... HLMS and F PAN The amplitude and phase of these two features; then using F PAN The amplitude and phase are respectively subtracted from F HLMSThe amplitude and phase of the HLMS image are analyzed, and the difference is fed into a 1×1 convolutional network for feature extraction. The extracted features are then mapped to a higher dimension using a ReLU activation function. The difference between the amplitude and phase after convolutional network feature extraction and ReLU activation is added to the amplitude and phase of the HLMS image after 1×1 convolutional network feature extraction and ReLU activation. The resulting amplitude and phase are then input into an inverse Fourier transform to convert the frequency domain to the spatial domain. Finally, the image passes through a 3×3 convolutional network to obtain the output of the frequency detail injection module, i.e., the HLMS image. The frequency detail injection module then performs feature extraction on the HLMS image. HLMS and the features F of the PAN image PAN The processing procedure is represented by formulas (4) to (8) as follows:

[0039] A HLMS P HLMS =F(HLMS) (4)

[0040] A PAN P PAN =F(PAN) (5)

[0041] A HMS =f c1 (cat(f f (f c1 (A HLMS )), f f (f c1 (|A PAN -A HLMS |)))) (6)

[0042] P HMS =f c1 (cat(f f (f c1 (P HLMS )), f f (f c1 (|P PAN -P HLMS |)))) (7)

[0043] HMS = f c3 (F -1 (A HMS P HMS (8)

[0044] In formulas (4) to (8), F represents the Fourier transform; F -1 This represents the inverse Fourier transform; A and P represent the amplitude and phase after the Fourier transform, respectively. HLMS P represents the amplitude of the HLMS image after Fourier transform. HLMSA represents the phase of the HLMS image after Fourier transform. PAN P represents the amplitude of the PAN image after Fourier transform. PAN A represents the phase of the PAN image after Fourier transform. HMS P represents the amplitude of the HMS image after Fourier transform. HMS The phase of the HMS image after Fourier transform is represented by f; HMS on the left side of equation (8) represents the HMS image. c1 f represents a 1×1 convolutional network; c3 This represents a 3×3 convolutional network; f f This indicates the ReLU activation function; cat indicates concatenation of channel dimensions.

[0045] Furthermore, step S4.3 specifically includes:

[0046] For LRMS and PAN images, an unsupervised learning inference method is performed by designing a loss function, which is as follows:

[0047] 1) Spectral loss function L in the spatial domain spe It can be expressed by formula (9) as follows:

[0048] L spe =||HRMS-HLMS||1 (9)

[0049] 2) Structural loss function L in the spatial domain spa The following can be expressed by formulas (10) and (11):

[0050] ρ σ (h, w, b) = Cov(PAN) σ (h, w), HRMS σ (h, w, b)) (10)

[0051]

[0052] In formulas (10) and (11), ρ σ (h, w, b) represents the local correlation between the PAN image and the HRMS image, σ represents the size of the local block, (h, w) represents the pixel coordinates of the starting point of the local block, b represents the channel number, and Cov represents the magnitude of the correlation coefficient between the PAN image and the HRMS image. σ (h, w) represents a region of size σ in the PAN image originating at coordinates (h, w); HRMS σ (h, w, b) represents a region of size σ in the HRMS image, originating at coordinates (h, w); ρ σ，refThe local correlation coefficient between the PAN image and the LRMS image is calculated using formula (10) and used as a reference value; σ (h, w, b) represents the value of the structural loss function in the spatial domain between local blocks of size σ with starting pixel (h, w) on channel b of the PAN and HRMS images.

[0053] The final L is obtained by averaging the local loss of all channels and each pixel of the PAN and HRMS images. spa ;

[0054] 3) Frequency domain loss function L fre It can be expressed as follows by formula (12):

[0055] L fre =||A HRMS -A PAN ||1+||P HRMS -P PAN ||1 (12)

[0056] In formula (12), A HRMS P represents the amplitude of the HRMS image after Fourier transform. HRMS This represents the phase of the HRMS image after Fourier transform;

[0057] 4) The complete loss function is expressed by formula (13) as follows:

[0058] L = L fre +λ1L spa +λ2L spe (13)

[0059] In formula (13), λ1 and λ2 represent hyperparameters, both of which are constants.

[0060] The beneficial effects of this invention are:

[0061] The method of this invention improves the performance of panchromatic sharpening of remote sensing images on full-resolution images. The resulting high spatial resolution multispectral image significantly improves the spatial quality of the generated image while ensuring excellent spectral performance, effectively alleviating problems such as blurred spatial details and object distortion in the generated image. Attached Figure Description

[0062] Figure 1 This is a flowchart of the dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement, as described in an embodiment of the present invention.

[0063] Figure 2 This is a general schematic diagram of the neural network used in the embodiments of the present invention.

[0064] Figure 3 This is a schematic diagram of the frequency detail injection module described in an embodiment of the present invention. Detailed Implementation

[0065] To facilitate a better understanding of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following are merely exemplary and do not limit the scope of protection of the present invention.

[0066] This invention discloses a dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement, comprising the following steps:

[0067] S1. Acquire remote sensing satellite images, which include panchromatic images and low spatial resolution multispectral images taken by the satellite, denoted as PAN images and LRMS images respectively. Preprocess the PAN images and LRMS images, wherein the LRMS images are preprocessed to obtain LLMS images, and the spatial resolution of the LLMS images is lower than that of the LRMS images.

[0068] S2. Construct a multispectral image super-resolution enhancement sub-network and train the multispectral image super-resolution enhancement sub-network in the low spatial resolution domain.

[0069] S3. Load the multispectral image super-resolution enhancement subnetwork trained in step S2, and use the multispectral image super-resolution enhancement subnetwork to perform super-resolution on the LRMS image, thereby obtaining a high spatial resolution low quality multispectral image, i.e., an HLMS image.

[0070] S4. Construct a frequency detail injection module and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, i.e., an HRMS image.

[0071] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. These are merely exemplary embodiments of the present invention; however, it should be understood that the present invention can be implemented in various forms and is not limited to the embodiments described herein. These embodiments are provided to enable those skilled in the art to understand the present invention more clearly and thoroughly.

[0072] This embodiment describes a dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement, such as... Figure 1 As shown, the steps include:

[0073] S1. Acquire remote sensing satellite images, which include panchromatic images and low spatial resolution multispectral images taken by the satellite, denoted as PAN images and LRMS images, respectively. Preprocess the PAN images and LRMS images, wherein the LRMS images are preprocessed to obtain LLMS images, and the spatial resolution of the LLMS images is lower than that of the LRMS images.

[0074] In this embodiment, the preprocessing of PAN images and LRMS images specifically includes:

[0075] The PAN image is cropped to a size of 512×512 with 1 channel, meaning the height and width of the cropped PAN image are both 512.

[0076] The LRMS image was cropped to a size of 128×128 with 8 channels. Then, the LRMS image was downsampled by 4 times to obtain the LLMS image, which was 32×32 with 8 channels.

[0077] S2. Construct a multispectral image super-resolution enhancement subnetwork and train the multispectral image super-resolution enhancement subnetwork in the low spatial resolution domain.

[0078] In this embodiment, step S2 specifically includes:

[0079] S2.1. The input LLMS image is copied to increase the number of channels from 8 to 128, resulting in a 32×32×128 image. Then, a 128×128×8 image is obtained through pixel rearrangement. Both the channel copying and pixel rearrangement operations are performed by a convolutional neural network. The convolutional neural network used for channel copying and pixel rearrangement constitutes the multispectral image super-resolution enhancement sub-network. The multispectral image super-resolution enhancement sub-network is represented by formulas (1) and (2) as follows:

[0080]

[0081] In formulas (1) and (2), M represents the multispectral image; C represents the number of channels in the image; U pc This indicates that channels of the multispectral image M are copied, with a copy factor of r. 2 ;P sf This indicates pixel rearrangement, which enlarges the height and width of the multispectral image M;

[0082] S2.2 Training the multispectral image super-resolution enhancement sub-network by designing a loss function L lspe Implementation, loss function L lspe The following can be expressed using formula (3):

[0083] L lspe =||LRMS-P sf (U pc (LLMS))||1 (3)

[0084] In formula (3), LLMS is the input of the multispectral image super-resolution enhancement subnetwork, and LRMS is the supervision signal for training the multispectral image super-resolution enhancement subnetwork.

[0085] S3. Load the multispectral image super-resolution enhancement subnetwork trained in step S2, and use the multispectral image super-resolution enhancement subnetwork to perform super-resolution on the LRMS image, thereby obtaining a high spatial resolution low quality multispectral image, i.e., an HLMS image.

[0086] In this embodiment, as Figure 2 As shown, step S3 specifically includes:

[0087] By loading the multispectral image super-resolution enhancement subnetwork trained in step S2, the trained multispectral image super-resolution enhancement subnetwork is used to super-resolution the LRMS image, so that the height and width of the LRMS image are both increased by 4 times, resulting in a high spatial resolution low quality multispectral image, namely the HLMS image. The size of the HLMS image is 512×512 and the number of channels is 8.

[0088] S4. Construct a frequency detail injection module and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, i.e., an HRMS image.

[0089] In this embodiment, as Figure 2 and Figure 3 As shown, step S4 includes:

[0090] S4.1 Processing the HLMS and PAN images: Use two convolutional layers to convolve the input HLMS and PAN images respectively to extract features, so that the number of channels in both the HLMS and PAN images becomes 32.

[0091] S4.2 Construct and train the frequency detail injection module. Input the features of the HLMS image and PAN image processed in step S4.1 into the trained frequency detail injection module and output the HMS image.

[0092] Specifically, step S4.2 includes:

[0093] A frequency detail injection module is constructed and trained, which incorporates the features F from the HLMS image after feature extraction in step S4.1. HLMS and the features F of the PAN image PANInput to the frequency detail injection module: First, it undergoes a Fourier transform to obtain F... HLMS and F PAN The amplitude and phase of these two features; then using F PAN The amplitude and phase are respectively subtracted from F HLMS The amplitude and phase of the HLMS image are analyzed, and the difference is fed into a 1×1 convolutional network for feature extraction. The extracted features are then mapped to a higher dimension using a ReLU activation function. The difference between the amplitude and phase after convolutional network feature extraction and ReLU activation is added to the amplitude and phase of the HLMS image after 1×1 convolutional network feature extraction and ReLU activation. The resulting amplitude and phase are then input into an inverse Fourier transform to convert the frequency domain to the spatial domain. Finally, the image passes through a 3×3 convolutional network to obtain the output of the frequency detail injection module, i.e., the HLMS image. The frequency detail injection module then performs feature extraction on the HLMS image. HLMS and the features F of the PAN image PAN The processing procedure is represented by formulas (4) to (8) as follows:

[0094] A HLMS P HLMS =F(HLMS) (4)

[0095] A PAN P PAN =F(PAN) (5)

[0096] A HMS =f c1 (cat(f f (f c1 (A HLMS )), f f (f c1 (|A PAN -A HLMS |)))) (6)

[0097] P HMS =f c1 (cat(f f (f c1 (P HLMS )), f f (f c1 (|P PAN -P HLMS |)))) (7)

[0098] HMS = f c3 (F -1 (A HMS P HMS (8)

[0099] In formulas (4) to (8), F represents the Fourier transform; F -1 This represents the inverse Fourier transform; A and P represent the amplitude and phase after the Fourier transform, respectively. HLMS P represents the amplitude of the HLMS image after Fourier transform. HLMS A represents the phase of the HLMS image after Fourier transform. PAN P represents the amplitude of the PAN image after Fourier transform. PAN A represents the phase of the PAN image after Fourier transform. HMS P represents the amplitude of the HMS image after Fourier transform. HMS The phase of the HMS image after Fourier transform is represented by f; HMS on the left side of equation (8) represents the HMS image. c1 f represents a 1×1 convolutional network; c3 This represents a 3×3 convolutional network; f f This indicates the ReLU activation function; cat indicates concatenation of channel dimensions.

[0100] S4.3. The HMS image and the HLMS image obtained in step S3 are weighted and added together using an unsupervised learning method to obtain a high spatial resolution multispectral image, i.e., the HRMS image.

[0101] Specifically, step S4.3 includes:

[0102] For LRMS and PAN images, an unsupervised learning inference method is performed by designing a loss function, which is as follows:

[0103] 1) Spectral loss function L in the spatial domain spe It can be expressed by formula (9) as follows:

[0104] L spe =||HRMS-HLMS||1 (9)

[0105] 2) Structural loss function L in the spatial domain spa The following can be expressed by formulas (10) and (11):

[0106] ρ σ (h, w, b) = Cov(PAN) σ (h, w), HRMS σ (h, w, b)) (10)

[0107]

[0108] In formulas (10) and (11), ρ σ(h, w, b) represents the local correlation between the PAN image and the HRMS image, σ represents the size of the local block, (h, w) represents the pixel coordinates of the starting point of the local block, b represents the channel number, and Cov represents the magnitude of the correlation coefficient between the PAN image and the HRMS image. σ (h, w) represents a region of size σ in the PAN image originating at coordinates (h, w); HRMS σ (h, w, b) represents a region of size σ in the HRMS image, originating at coordinates (h, w); ρ σ，ref The local correlation coefficient between the PAN image and the LRMS image is calculated using formula (10) and used as a reference value; σ (h, w, b) represents the value of the structural loss function in the spatial domain between local blocks of size σ with starting pixel (h, w) on channel b of the PAN and HRMS images.

[0109] The final L is obtained by averaging the local loss of all channels and each pixel of the PAN and HRMS images. spa ;

[0110] 3) Frequency domain loss function L fre It can be expressed as follows by formula (12):

[0111] L fre =||A HRMS -A PAN ||1+||P HRMS -P PAN ||1 (12)

[0112] In formula (12), A HRMS P represents the amplitude of the HRMS image after Fourier transform. HRMS This represents the phase of the HRMS image after Fourier transform;

[0113] 4) The complete loss function is expressed by formula (13) as follows:

[0114] L = L fre +λ1L spa +λ2L spe (13)

[0115] In formula (13), λ1 and λ2 represent hyperparameters, both of which are constants.

[0116] The method of this invention is compared with eight remote sensing image panchromatic sharpening methods: PNN (Pansharpening by Convolutional Neural Network), MSDCNN (Multiscale and Multidepth Convolutional Neural Network), BDPN (Bidirectional Pyramid Network), DRPNN (Deep Residual Pan-sharpening Neural Network), DiCNN (Detail injection based Convolutional Neural Networks), FusionNet (Fusion Network), Ban_aware-TCI (PAN-guided band-aware multi-spectral feature enhancement), and PYDDN (Pyramid Dual Domain Injection Network). Using R-Q2n (Reprojection Universal Image Quality Index), R-SAM (Reprojection Spectral Angle Mapper), and R-ERGAS (Reprojection Relative Dimensionless Global Error in Synthesis) respectively, (Spectral Distortion Index), D ρThe Correlation Distortion Index (R-Q2) was used as an evaluation metric, tested on full-resolution image data captured by the WorldView3, WorldView2, and QuickBird satellites. The objective metric results are shown in Tables 1, 2, and 3, respectively. In the tables, the highest-ranked metric is indicated by a double underline, the second-ranked metric by a dashed line, and the third-ranked metric by a wavy line. It can be seen that the method of this invention achieves the best performance in the spectral index R-Q2. n R-SAM, R-ERGAS The results of the above indicators are not significantly different from the optimal method; however, the method of this invention performs better in terms of spatial evaluation index D. ρ The above method performs best and is far superior to the other eight methods.

[0117] Table 1. Performance comparison of the method of this invention and eight remote sensing image panchromatic sharpening methods on the Worldview3 dataset.

[0118]

[0119] Table 2. Performance comparison of the method of this invention and eight other remote sensing image panchromatic sharpening methods on the Worldview2 dataset.

[0120]

[0121] Table 3. Performance comparison of the method of this invention and eight remote sensing image panchromatic sharpening methods on the QuickBird dataset.

[0122]

[0123] In summary, the method of the present invention improves the performance of panchromatic sharpening of remote sensing images on full-resolution images. The resulting high spatial resolution multispectral image significantly improves the spatial quality of the generated image while ensuring excellent spectral performance, effectively alleviating problems such as blurred spatial details and object distortion in the generated image.

[0124] The above description only outlines the basic principles and preferred embodiments of the present invention. Those skilled in the art can make many changes and modifications based on the above description, and these changes and modifications should fall within the protection scope of the present invention.

Claims

1. A dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement, characterized in that, Including the following steps: S1. Acquire remote sensing satellite images, which include panchromatic images and low spatial resolution multispectral images taken by the satellite, denoted as PAN images and LRMS images respectively. Preprocess the PAN images and LRMS images, wherein the LRMS images are preprocessed to obtain LLMS images, and the spatial resolution of the LLMS images is lower than that of the LRMS images. S2. Construct a multispectral image super-resolution enhancement sub-network and train the multispectral image super-resolution enhancement sub-network in the low spatial resolution domain. S3. Load the multispectral image super-resolution enhancement subnetwork trained in step S2, and use the multispectral image super-resolution enhancement subnetwork to perform super-resolution on the LRMS image, thereby obtaining a high spatial resolution low quality multispectral image, i.e., an HLMS image. S4. Construct a frequency detail injection module and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, i.e., an HRMS image; Step S4 includes: S4.1 Processing HLMS and PAN images; S4.

2. Construct and train the frequency detail injection module. Input the features of the HLMS image and PAN image processed in step S4.1 into the trained frequency detail injection module, and output the HMS image. Step S4.2 specifically includes: A frequency detail injection module is constructed and trained, which incorporates the features F from the HLMS image after feature extraction in step S4.

1. HLMS and the features F of the PAN image PAN Input to the frequency detail injection module: First, it undergoes a Fourier transform to obtain F... HLMS and F PAN The amplitude and phase of these two features; then using F PAN The amplitude and phase are respectively subtracted from F HLMS The amplitude and phase of the frequency domain are obtained, and the difference is fed into a 1×1 convolutional network for feature extraction. The extracted features are mapped to high dimension by the ReLU activation function. The difference between the amplitude and phase after feature extraction by the convolutional network and the ReLU activation function is added to the amplitude and phase of the HLMS image after feature extraction by the 1×1 convolutional network and the ReLU activation function. The obtained amplitude and phase are input into the inverse Fourier transform to transform from the frequency domain to the spatial domain. Then, the output of the frequency detail injection module is obtained by passing through a 3×3 convolutional network, which is the HMS image. S4.

3. The HMS image and the HLMS image obtained in step S3 are weighted and added together using an unsupervised learning method to obtain a high spatial resolution multispectral image, i.e., the HRMS image.

2. The dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement according to claim 1, characterized in that, In step S1, the PAN image and LRMS image are preprocessed, specifically including: The PAN image is cropped to a size of H×W with 1 channel, where H and W represent the height and width of the image, respectively. The LRMS image is cropped to a size of [size missing]. The number of channels is C, where r represents the image magnification factor; then, the LRMS image is downsampled by a factor of r to obtain the LLMS image, the height of which is... Width The number of channels is C.

3. The dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement according to claim 2, characterized in that, Step S2 specifically includes: S2.

1. Copy the input LLMS image, making the number of channels of the LLMS image with C channels C×r. 2 , get one The image; then a pixel rearrangement is used to obtain a The image; wherein, the two operations of channel copying and pixel rearrangement are both completed by a convolutional neural network, and the convolutional neural network used for channel copying and pixel rearrangement constitutes the multispectral image super-resolution enhancement sub-network; the multispectral image super-resolution enhancement sub-network is expressed by formulas (1) and (2) as follows: In formulas (1) and (2), M represents the multispectral image; C represents the number of channels in the image; U pc This indicates that channels of the multispectral image M are copied, with a copy factor of r. 2 ;P sf This indicates pixel rearrangement, which enlarges the height and width of the multispectral image M; S2.2 Training the multispectral image super-resolution enhancement sub-network by designing a loss function L lspe Implementation, loss function L lspe The following can be expressed using formula (3): L lspe =‖LRMS-P sf (U pc (LLMS))‖1 (3) In Equation (3), LLMS is the input to the multispectral image super-resolution enhancement subnetwork.

4. The dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement according to claim 2 or 3, characterized in that, Step S3 specifically includes: By loading the multispectral image super-resolution enhancement subnetwork trained in step S2, the trained multispectral image super-resolution enhancement subnetwork is used to super-resolution the LRMS image, so that the height and width of the LRMS image are both increased by r times, resulting in a high spatial resolution low quality multispectral image, namely the HLMS image. The size of the HLMS image is H×W and the number of channels is C.

5. The dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement according to claim 4, characterized in that, Step S4.1 specifically includes: Two convolutional layers are used to convolve the input HLMS image and PAN image respectively to extract features, so that the number of channels in both the HLMS image and PAN image becomes 32.

6. The dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement according to claim 5, characterized in that, In step S4.2: Frequency detail injection module for HLMS image features F HLMS and the features F of the PAN image PAN The processing procedure is represented by formulas (4) to (8) as follows: A HLMS ,P HLMS =F( HLMS ) (4) A PAN ,P PAN =F(PAN) (5) A HMS =f c1 (cat(f f (f c1 (A HLMS )),f f (f c1 (|A PAN -A HLMS |)))) (6) P HMS =f c1 (cat(f f (f c1 (P HLMS )),f f (f c1 (|P PAN -P HLMS |)))) (7) HMS=f c3 (F -1 (A HMS ,P HMS )) (8) In formulas (4) to (8), F represents the Fourier transform; F -1 This represents the inverse Fourier transform; A and P represent the amplitude and phase after the Fourier transform, respectively. HLMS P represents the amplitude of the HLMS image after Fourier transform. HLMS A represents the phase of the HLMS image after Fourier transform. PAN P represents the amplitude of the PAN image after Fourier transform. PAN A represents the phase of the PAN image after Fourier transform. HMS P represents the amplitude of the HMS image after Fourier transform. HMS The phase of the HMS image after Fourier transform is represented by f; HMS on the left side of equation (8) represents the HMS image. c1 f represents a 1×1 convolutional network; c3 This represents a 3×3 convolutional network; f f This indicates the ReLU activation function; cat indicates concatenation of channel dimensions.

7. The dual-domain unsupervised panchromatic sharpening method for remote sensing images based on super-resolution enhancement according to claim 6, characterized in that, Step S4.3 specifically includes: For LRMS and PAN images, an unsupervised learning inference method is performed by designing a loss function, which is as follows: 1) Spectral loss function L in the spatial domain spe It can be expressed by formula (9) as follows: L spe =‖HRMS-HLMS‖1 (9) 2) Structural loss function L in the spatial domain spa The following can be expressed by formulas (10) and (11): ρ σ (h,w,b)=Cov(PAN σ (h,w),HRMS σ (h,w,b)) (10) In formulas (10) and (11), ρ σ (h,w,b) represents the local correlation between the PAN image and the HRMS image, σ represents the size of the local block, (h,w) represents the pixel coordinates of the starting point of the local block, b represents the channel number, and Cov represents the magnitude of the correlation coefficient between the PAN image and the HRMS image. σ (h,w) represents a region of size σ in the PAN image, originating at coordinates (h,w); HRMS σ (h,w,b) represents a region of size σ in the HRMS image, originating at coordinates (h,w); ρ σ,ref The local correlation coefficient between the PAN image and the LRMS image is calculated using formula (10) and used as a reference value; σ (h,w,b) represents the value of the structural loss function in the spatial domain between local blocks of size σ with starting pixel (h,w) on channel b of the PAN and HRMS images. The final L is obtained by averaging the local loss of all channels and each pixel of the PAN and HRMS images. spa ; 3) Frequency domain loss function L fre It can be expressed as follows by formula (12): L fre =‖A HRMS -A PAN ‖1+‖P HRMS -P PAN ‖1 (12) In formula (12), A HRMS P represents the amplitude of the HRMS image after Fourier transform. HRMS This represents the phase of the HRMS image after Fourier transform; 4) The complete loss function is expressed by formula (13) as follows: L=L fre +λ1L spa +λ2L spe (13) In formula (13), λ1 and λ2 represent hyperparameters, both of which are constants.

Images