Double-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement

Abstract

The invention discloses a double-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement, and the method comprises the steps: obtaining a PAN image and an LRMS image, carrying out the preprocessing of the PAN image and the LRMS image, and obtaining an LLMS image after the preprocessing of the LRMS image; constructing a multi-spectral image super-resolution enhancement sub-network, and training the multi-spectral image super-resolution enhancement sub-network in a low spatial resolution domain; loading the trained multi-spectral image super-resolution enhancement sub-network, and performing super-resolution on the LRMS image through the multi-spectral image super-resolution enhancement sub-network to obtain an HLMS image; and constructing a frequency detail injection module, and inputting the HLMS image and the PAN image into the frequency detail injection module for unsupervised learning to obtain an HRMS image. According to the method, the excellent spectral performance is ensured, the space quality of the generated image is remarkably improved, and the problems of fuzzy space details, object deformation and the like of the generated image are effectively relieved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a dual-domain unsupervised remote sensing image full color sharpening method based on super-resolution enhancement. Background Art

[0002] Remote sensing image panchromatic sharpening technology is a technology that utilizes the high-resolution characteristics of panchromatic images and fuses them with multispectral images to improve the spatial resolution of multispectral images while maintaining their spectral information. It is widely considered to be an image enhancement method that effectively improves the spatial expression ability of remote sensing images. It aims to make up for the limitation that a single satellite sensor cannot obtain multispectral images with high spatial resolution, and obtain remote sensing image data with both high spatial resolution and high spectral resolution without changing the hardware conditions. Due to the differences between panchromatic and multispectral images in space, radiation, spectrum, etc., and the imaging process is easily affected by factors such as atmospheric conditions, surface characteristics, and imaging geometry, panchromatic sharpening faces many challenges and has become a research hotspot in the field of remote sensing that is both valuable and challenging.

[0003] The methods of pan-sharpening remote sensing images can be divided into two categories: traditional methods and deep learning methods. Traditional methods are composed of component substitution method, multi-resolution analysis method and variational optimization method. Traditional pan-sharpening methods manually design image features and filters, and different features and filters are required for different scenes and data, which consumes a lot of manpower and time. The pan-sharpening method based on deep learning can automatically learn image features and filters based on a large amount of data, and has better adaptability and universality. Therefore, the pan-sharpening method based on deep learning has become a hot research direction in this field and has made significant progress.

[0004] However, the images obtained by these deep learning methods still have some shortcomings in quality, such as blurred spatial details and object deformation. The main reason for these problems is that most existing methods use interpolation algorithms to process multispectral images when enlarging them. Although this simple interpolation operation can improve the resolution of the image, it ignores the rich spatial detail information contained in the multispectral image, resulting in the details of the full-color sharpened image being blurred and the objects being deformed. In addition, these methods focus on model training on images with reduced resolution, and then directly apply the trained model to the full-color sharpening task of full-resolution images. Although this strategy can simplify the model training process, it ignores the differences in high-frequency detail features between full-resolution images and low-resolution images, making it difficult for the trained model to fit the high-frequency detail features of the full-resolution image well, thereby affecting the quality of the sharpened image. Summary of the invention

[0005] In order to overcome the deficiencies of the above technologies, the present invention provides a dual-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement.

[0006] Terminology explanation:

[0007] 1. LRMS: Low-resolution Multispectral, low spatial resolution multispectral image.

[0008] 2. PAN: Panchromatic, full-color image.

[0009] 3. LLMS: Lower Low Resolution Multispectral, multispectral image with lower spatial resolution.

[0010] 4. HLMS: High-resolution Low-quality Multispectral, high spatial resolution low-quality multispectral image.

[0011] 5. HRMS: High-resolution Multispectral, high spatial resolution multispectral image.

[0012] 6. HMS: High-quality Multispectral, high-quality multispectral image.

[0013] The technical solution adopted by the present invention to overcome the technical problems is:

[0014] A dual-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement, comprising the steps of:

[0015] S1. Acquire a remote sensing satellite image, wherein the remote sensing satellite image includes a panchromatic image and a low spatial resolution multispectral image taken by a satellite, which are respectively recorded as a PAN image and a LRMS image, and preprocess the PAN image and the LRMS image, wherein the LLMS image is obtained after the LRMS image is preprocessed, and the spatial resolution of the LLMS image is lower than the spatial resolution of the LRMS image;

[0016] S2. Construct a multispectral image super-resolution enhancer network and train the multispectral image super-resolution enhancer network in a low spatial resolution domain;

[0017] S3, loading the multispectral image super-resolution enhancer network trained in step S2, and super-resolving the LRMS image through the multispectral image super-resolution enhancer network, so as to obtain a high spatial resolution and low quality multispectral image, i.e., a HLMS image;

[0018] S4. Build a frequency detail injection module, and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, namely, a HRMS image.

[0019] Furthermore, in step S1, the PAN image and the LRMS image are preprocessed, specifically including:

[0020] The PAN image is cropped to a size of H×W and a channel number of 1, where H and W represent the height and width of the image, respectively;

[0021] The LRMS image is cropped to a size of The number of channels is C, where r represents the magnification of the image. Then, the LRMS image is downsampled r times to obtain the LLMS image, and the height of the LLMS image is Width The number of channels is C.

[0022] Furthermore, step S2 specifically includes:

[0023] S2.1. Copy the input LLMS image and copy the number of channels of the LLMS image with the number of channels C to C×r 2 , get a The image is then rearranged to obtain an The image is obtained by using the convolutional neural network. The convolutional neural network for channel duplication and pixel rearrangement constitutes a multispectral image super-resolution enhancer network. The multispectral image super-resolution enhancer network is expressed as follows by formulas (1) and (2):

[0024]

[0025] In formulas (1) and (2), M represents the multispectral image; C represents the number of channels of the image; U pc Indicates that the channels of the multispectral image M are copied, and the copy multiple is r 2 ;P sf It represents pixel rearrangement, which expands the height and width of the multispectral image M;

[0026] S2.2. Train the multispectral image super-resolution enhancement sub-network by designing the loss function L lspe Implementation, loss function L lspe It is expressed as follows using formula (3):

[0027] L lspe =||LRMS-P sf (U pc(LLMS))||1 (3)

[0028] In formula (3), LLMS is the input of the multispectral image super-resolution enhancer network, and LRMS is the supervisory signal for training the multispectral image super-resolution enhancer network.

[0029] Furthermore, step S3 specifically includes:

[0030] By loading the multispectral image super-resolution enhancer network trained in step S2, the LRMS image is super-resolved using the trained multispectral image super-resolution enhancer network, so that the height and width of the LRMS image are both enlarged by r times, and a high spatial resolution and low-quality multispectral image, namely, a HLMS image, is obtained. The size of the HLMS image is H×W and the number of channels is C.

[0031] Further, step S4 includes:

[0032] S4.1, processing the HLMS image and the PAN image;

[0033] S4.2, construct a frequency detail injection module and train it, input the features of the HLMS image and PAN image processed in step S4.1 into the trained frequency detail injection module, and output the HMS image;

[0034] S4.3. Perform weighted addition of the HMS image and the HLMS image obtained in step S3 by an unsupervised learning method to obtain a high spatial resolution multispectral image, namely, a HRMS image.

[0035] Furthermore, step S4.1 specifically includes:

[0036] Two convolutional layers are used to convolve the input HLMS image and PAN image respectively to extract features, so that the number of channels of the HLMS image and PAN image becomes 32.

[0037] Furthermore, step S4.2 specifically includes:

[0038] Construct a frequency detail injection module and train it to inject the feature F of the HLMS image extracted in step S4.1 into the HLMS and the feature F of PAN image PAN Input to the frequency detail injection module: First, after Fourier transform, we get F HLMS and F PAN The amplitude and phase of these two features; then use F PAN The amplitude and phase of the HLMSThe amplitude and phase of the HLMS image are extracted by the convolution network and the difference is put into a 1×1 convolution network for feature extraction. The extracted features are mapped to high dimensions through the ReLu activation function. The difference of the amplitude and phase after the convolution network feature extraction and the ReLu activation function is added to the amplitude and phase of the HLMS image after the 1×1 convolution network feature extraction and the ReLu activation function. The obtained amplitude and phase are input into the inverse Fourier transform, converted from the frequency domain to the spatial domain, and then passed through a 3×3 convolution network to obtain the output of the frequency detail injection module, that is, the HMS image; the frequency detail injection module performs feature extraction on the HLMS image F HLMS and the feature F of PAN image PAN The processing process is expressed by formula (4) to formula (8) as follows:

[0039] A HLMS , P HLMS =F(HLMS) (4)

[0040] A PAN , P PAN =F(PAN) (5)

[0041] A HMS =f c1 (cat(f f (f c1 (A HLMS )),f f (f c1 (|A PAN -A HLMS |)))) (6)

[0042] P HMS =f c1 (cat(f f (f c1 (P HLMS )),f f (f c1 (|P PAN -P HLMS |)))) (7)

[0043] HMS=f c3 (F -1 (A HMS , P HMS )) (8)

[0044] In formula (4) to formula (8), F represents Fourier transform; F -1 represents inverse Fourier transform; A and P represent the amplitude and phase after Fourier transform, respectively. HLMS represents the amplitude of the HLMS image after Fourier transformation, P HLMSA represents the phase of the HLMS image after Fourier transformation. PAN represents the amplitude of the PAN image after Fourier transformation, P PAN represents the phase of the PAN image after Fourier transformation, A HMS represents the amplitude of the HMS image after Fourier transformation, P HMS represents the phase of the HMS image after Fourier transformation. The HMS on the left side of formula (8) represents the HMS image; f c1 represents a 1×1 convolutional network; f c3 represents a 3×3 convolutional network; f f ReLu activation function; cat represents the concatenation of channel dimensions.

[0045] Furthermore, step S4.3 specifically includes:

[0046] For LRMS images and PAN images, unsupervised learning method reasoning is performed by designing a loss function, and the loss function is designed as follows:

[0047] 1) Spectral loss function L in spatial domain spe , which is expressed by formula (9) as follows:

[0048] L spe =||HRMS-HLMS||1 (9)

[0049] 2) Structural loss function L in spatial domain spa , which is expressed by formula (10) and formula (11) as follows:

[0050] ρ σ (h, w, b) = Cov(PAN σ (h, w), HRMS σ (h, w, b)) (10)

[0051]

[0052] In formula (10) and formula (11), ρ σ (h, w, b) represents the local correlation between the PAN image and the HRMS image, σ represents the size of the local block, (h, w) represents the pixel position coordinates of the starting point of the local block, b represents the channel number, and Cov represents the size of the correlation coefficient between the PAN image and the HRMS image; PAN σ (h, w) represents the region of size σ with coordinates (h, w) as the starting point in the PAN image; HRMS σ (h, w, b) represents the region of size σ with coordinates (h, w) as the starting point in the HRMS image; ρ σ，refThe local correlation coefficient between the PAN image and the LRMS image is calculated using formula (10) as a reference value; σ (h, w, b) represents the value of the structure loss function in the spatial domain between local blocks with a starting pixel (h, w) and a size of σ on channel b of the PAN image and the HRMS image;

[0053] The final L is obtained by averaging the local losses of all channels and each pixel of the PAN image and the HRMS image. spa ;

[0054] 3) Frequency domain loss function L fre , which is expressed by formula (12) as follows:

[0055] L fre =||A HRMS -A PAN ||1+||P HRMS -P PAN ||1 (12)

[0056] In formula (12), A HRMS represents the amplitude of the HRMS image after Fourier transformation, P HRMS represents the phase of the HRMS image after Fourier transformation;

[0057] 4) The complete loss function is expressed as follows through formula (13):

[0058] L=L fre +λ1L spa +λ2L spe (13)

[0059] In formula (13), λ1 and λ2 represent hyperparameters, both of which are constants.

[0060] Beneficial effects of the present invention:

[0061] The method of the present invention improves the performance of panchromatic sharpening of remote sensing images on full-resolution images. The obtained multispectral images with high spatial resolution significantly improve the spatial quality of the generated images while ensuring excellent spectral performance, and effectively alleviate the problems of blurred spatial details and object deformation in the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 The present invention is a flowchart of a dual-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement according to an embodiment of the present invention.

[0063] Figure 2 Schematic diagram of a neural network used in an embodiment of the present invention.

[0064] Figure 3 It is a schematic diagram of a frequency detail injection module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to facilitate those skilled in the art to better understand the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The following is only exemplary and does not limit the protection scope of the present invention.

[0066] The present invention discloses a dual-domain unsupervised remote sensing image full color sharpening method based on super-resolution enhancement, comprising the steps of:

[0067] S1. Acquire a remote sensing satellite image, wherein the remote sensing satellite image includes a panchromatic image and a low spatial resolution multispectral image taken by a satellite, which are respectively recorded as a PAN image and a LRMS image, and preprocess the PAN image and the LRMS image, wherein the LLMS image is obtained after the LRMS image is preprocessed, and the spatial resolution of the LLMS image is lower than the spatial resolution of the LRMS image;

[0068] S2. Construct a multispectral image super-resolution enhancer network and train the multispectral image super-resolution enhancer network in a low spatial resolution domain;

[0069] S3, loading the multispectral image super-resolution enhancer network trained in step S2, and super-resolving the LRMS image through the multispectral image super-resolution enhancer network, so as to obtain a high spatial resolution and low quality multispectral image, i.e., a HLMS image;

[0070] S4. Build a frequency detail injection module, and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, namely, a HRMS image.

[0071] In order to better understand the above technical solution, the exemplary embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. These are only exemplary embodiments of the present invention. However, it should be understood that the present invention can also be implemented in various forms and is not limited to the embodiments described herein. These embodiments are intended to enable those skilled in the art to understand the present invention more clearly and thoroughly.

[0072] The present embodiment describes a dual-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement, such as Figure 1 As shown, the steps include:

[0073] S1. Acquire a remote sensing satellite image, wherein the remote sensing satellite image includes a panchromatic image and a low spatial resolution multispectral image taken by a satellite, which are respectively recorded as a PAN image and a LRMS image, and preprocess the PAN image and the LRMS image, wherein the LLMS image is obtained after the LRMS image is preprocessed, and the spatial resolution of the LLMS image is lower than the spatial resolution of the LRMS image.

[0074] In this embodiment, the PAN image and the LRMS image are preprocessed, specifically including:

[0075] The PAN image is cropped to a size of 512×512 and a channel number of 1, that is, the height and width of the cropped PAN image are both 512.

[0076] The LRMS image is cropped to a size of 128×128 and 8 channels. Then, the LRMS image is downsampled by 4 times to obtain a LLMS image, which has a size of 32×32 and 8 channels.

[0077] S2. Construct a multispectral image super-resolution enhancer network and train the multispectral image super-resolution enhancer network in a low spatial resolution domain.

[0078] In this embodiment, step S2 specifically includes:

[0079] S2.1. The input LLMS image is copied, and the number of channels of the LLMS image with 8 channels is copied to 128, so as to obtain a 32×32×128 image; then, a 128×128×8 image is obtained by rearranging pixels; wherein, the two operations of channel copying and pixel rearrangement are both completed by convolutional neural networks, and the convolutional neural networks used for channel copying and pixel rearrangement constitute a multispectral image super-resolution enhancement subnetwork; the multispectral image super-resolution enhancement subnetwork is expressed as follows by formulas (1) and (2):

[0080]

[0081] In formulas (1) and (2), M represents the multispectral image; C represents the number of channels of the image; U pc Indicates that the channels of the multispectral image M are copied, and the copy multiple is r 2 ;P sf It represents pixel rearrangement, which expands the height and width of the multispectral image M;

[0082] S2.2. Train the multispectral image super-resolution enhancement sub-network by designing the loss function L lspe Implementation, loss function L lspe It is expressed as follows using formula (3):

[0083] L lspe =||LRMS-P sf (U pc (LLMS))||1 (3)

[0084] In formula (3), LLMS is the input of the multispectral image super-resolution enhancer network, and LRMS is the supervisory signal for training the multispectral image super-resolution enhancer network.

[0085] S3, loading the multispectral image super-resolution enhancer network trained in step S2, and super-resolving the LRMS image through the multispectral image super-resolution enhancer network, so as to obtain a high spatial resolution and low quality multispectral image, i.e., a HLMS image.

[0086] In this embodiment, Figure 2 As shown, step S3 specifically includes:

[0087] By loading the multispectral image super-resolution enhancer network trained in step S2, the LRMS image is super-resolved using the trained multispectral image super-resolution enhancer network, so that the height and width of the LRMS image are expanded by 4 times, and a high spatial resolution and low-quality multispectral image, namely, the HLMS image, is obtained. The size of the HLMS image is 512×512 and the number of channels is 8.

[0088] S4. Build a frequency detail injection module, and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, namely, a HRMS image.

[0089] In this embodiment, Figure 2 and Figure 3 As shown, step S4 includes:

[0090] S4.1. Processing the HLMS image and PAN image: Use two convolutional layers to convolve the input HLMS image and PAN image respectively to extract features, so that the number of channels of the HLMS image and PAN image becomes 32.

[0091] S4.2. Build a frequency detail injection module and train it. Input the features of the HLMS image and PAN image processed in step S4.1 into the trained frequency detail injection module and output the HMS image.

[0092] Specifically, step S4.2 includes:

[0093] Construct a frequency detail injection module and train it to inject the feature F of the HLMS image extracted in step S4.1 into the HLMS and the feature F of PAN image PANInput to the frequency detail injection module: First, after Fourier transform, we get F HLMS and F PAN The amplitude and phase of these two features; then use F PAN The amplitude and phase of the HLMS The amplitude and phase of the HLMS image are extracted by the convolution network and the difference is put into a 1×1 convolution network for feature extraction. The extracted features are mapped to high dimensions through the ReLu activation function. The difference of the amplitude and phase after the convolution network feature extraction and the ReLu activation function is added to the amplitude and phase of the HLMS image after the 1×1 convolution network feature extraction and the ReLu activation function. The obtained amplitude and phase are input into the inverse Fourier transform, converted from the frequency domain to the spatial domain, and then passed through a 3×3 convolution network to obtain the output of the frequency detail injection module, that is, the HMS image; the frequency detail injection module performs feature extraction on the HLMS image F HLMS and the feature F of PAN image PAN The processing process is expressed by formula (4) to formula (8) as follows:

[0094] A HLMS , P HLMS =F(HLMS) (4)

[0095] A PAN , P PAN =F(PAN) (5)

[0096] A HMS =f c1 (cat(f f (f c1 (A HLMS )),f f (f c1 (|A PAN -A HLMS |)))) (6)

[0097] P HMS =f c1 (cat(f f (f c1 (P HLMS )),f f (f c1 (|P PAN -P HLMS |)))) (7)

[0098] HMS=f c3 (F -1 (A HMS , P HMS )) (8)

[0099] In formula (4) to formula (8), F represents Fourier transform; F -1 represents inverse Fourier transform; A and P represent the amplitude and phase after Fourier transform, respectively. HLMS represents the amplitude of the HLMS image after Fourier transformation, P HLMS A represents the phase of the HLMS image after Fourier transformation. PAN represents the amplitude of the PAN image after Fourier transformation, P PAN represents the phase of the PAN image after Fourier transformation, A HMS represents the amplitude of the HMS image after Fourier transformation, P HMS represents the phase of the HMS image after Fourier transformation. The HMS on the left side of formula (8) represents the HMS image; f c1 represents a 1×1 convolutional network; f c3 represents a 3×3 convolutional network; f f ReLu activation function; cat represents the concatenation of channel dimensions.

[0100] S4.3. Perform weighted addition of the HMS image and the HLMS image obtained in step S3 by an unsupervised learning method to obtain a high spatial resolution multispectral image, namely, a HRMS image.

[0101] Specifically, step S4.3 includes:

[0102] For LRMS images and PAN images, unsupervised learning method reasoning is performed by designing a loss function, and the loss function is designed as follows:

[0103] 1) Spectral loss function L in spatial domain spe , which is expressed by formula (9) as follows:

[0104] L spe =||HRMS-HLMS||1 (9)

[0105] 2) Structural loss function L in spatial domain spa , which is expressed by formula (10) and formula (11) as follows:

[0106] ρ σ (h, w, b) = Cov(PAN σ (h, w), HRMS σ (h, w, b)) (10)

[0107]

[0108] In formula (10) and formula (11), ρ σ(h, w, b) represents the local correlation between the PAN image and the HRMS image, σ represents the size of the local block, (h, w) represents the pixel position coordinates of the starting point of the local block, b represents the channel number, and Cov represents the size of the correlation coefficient between the PAN image and the HRMS image; PAN σ (h, w) represents the region of size σ with coordinates (h, w) as the starting point in the PAN image; HRMS σ (h, w, b) represents the region of size σ with coordinates (h, w) as the starting point in the HRMS image; ρ σ，ref The local correlation coefficient between the PAN image and the LRMS image is calculated using formula (10) as a reference value; σ (h, w, b) represents the value of the structure loss function in the spatial domain between local blocks with a starting pixel (h, w) and a size of σ on channel b of the PAN image and the HRMS image;

[0109] The final L is obtained by averaging the local losses of all channels and each pixel of the PAN image and the HRMS image. spa ;

[0110] 3) Frequency domain loss function L fre , which is expressed by formula (12) as follows:

[0111] L fre =||A HRMS -A PAN ||1+||P HRMS -P PAN ||1 (12)

[0112] In formula (12), A HRMS represents the amplitude of the HRMS image after Fourier transformation, P HRMS represents the phase of the HRMS image after Fourier transformation;

[0113] 4) The complete loss function is expressed as follows through formula (13):

[0114] L=L fre +λ1L spa +λ2L spe (13)

[0115] In formula (13), λ1 and λ2 represent hyperparameters, both of which are constants.

[0116] The method of the present invention is compared with eight remote sensing image panchromatic sharpening methods, including PNN method (Pansharpening by Convolutional Neural Network, panchromatic image fusion based on convolutional neural network), MSDCNN method (Multiscale and Multidepth Convolutional Neural Network, multi-scale and multi-depth convolutional neural network), BDPN method (Bidirectional Pyramid Network, bidirectional pyramid network), DRPNN method (Deep Residual Pan-sharpening Neural Network, deep residual panchromatic fusion neural network), DiCNN method (Detail injectionbased Convolutional Neural Networks, convolutional neural network based on detail injection), FusionNet method (Fusion Network, fusion network), Ban_aware-TCI method (PAN-guided band-aware multi-spectral feature enhancementmen, band-aware multi-spectral feature enhancement based on PAN guidance), and PYDDN method (Pyramid Dual Domain Injection Network, pyramid dual domain injection network). The three methods are respectively used: R-Q2n (Reprojection Universal Image Quality Index), R-SAM (Reprojection Spectral Angle Mapper), R-ERGAS (Reprojection Relative Dimensionless Global Error in Synthesis), (Spectral Distortion Index, spectral distortion index), D ρThe Correlation Distortion Index (Correlation Distortion Index) was used as an evaluation index and tested on full-resolution image data taken by the WorldView3 satellite, WorldView2 satellite, and QuickBird satellite. The objective index results are shown in Table 1, Table 2, and Table 3, respectively. The best-ranked index in the table is marked with a double underline, the second-ranked index is marked with a dotted line, and the third-ranked index is marked with a wavy line. It can be seen that the method of the present invention has good performance in spectral index R-Q2 n , R-SAM, R-ERGAS, The index results on the spatial evaluation index D ρ The performance is the best and far superior to the other eight methods.

[0117] Table 1 Performance comparison of the proposed method and eight remote sensing image panchromatic sharpening methods on the Worldview3 dataset

[0118]

[0119] Table 2 Performance comparison of the proposed method and eight remote sensing image pan-sharpening methods on the Worldview2 dataset

[0120]

[0121] Table 3 Performance comparison of the proposed method and eight remote sensing image pan-sharpening methods on the QuickBird dataset

[0122]

[0123] In summary, the method of the present invention improves the performance of panchromatic sharpening of remote sensing images on full-resolution images. The obtained multispectral images with high spatial resolution significantly improve the spatial quality of the generated images while ensuring excellent spectral performance, and effectively alleviate the problems of blurred spatial details and object deformation in the generated images.

[0124] The above only describes the basic principles and preferred embodiments of the present invention. Those skilled in the art may make many changes and improvements based on the above description, and these changes and improvements should fall within the protection scope of the present invention.

Claims

1. A dual-domain unsupervised remote sensing image panchromatic sharpening method based on super-resolution enhancement, characterized in that: Includes steps: S1. Acquire a remote sensing satellite image, wherein the remote sensing satellite image includes a panchromatic image and a low spatial resolution multispectral image taken by a satellite, which are respectively recorded as a PAN image and a LRMS image, and preprocess the PAN image and the LRMS image, wherein the LLMS image is obtained after the LRMS image is preprocessed, and the spatial resolution of the LLMS image is lower than the spatial resolution of the LRMS image; S2. Construct a multispectral image super-resolution enhancer network and train the multispectral image super-resolution enhancer network in a low spatial resolution domain; S3, loading the multispectral image super-resolution enhancer network trained in step S2, and super-resolving the LRMS image through the multispectral image super-resolution enhancer network, so as to obtain a high spatial resolution and low quality multispectral image, i.e., a HLMS image; S4. Build a frequency detail injection module, and input the HLMS image and PAN image into the frequency detail injection module for unsupervised learning to obtain a high spatial resolution multispectral image, namely, a HRMS image.

2. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 1, characterized in that: In step S1, the PAN image and the LRMS image are preprocessed, specifically including: The PAN image is cropped to a size of H×W and a channel number of 1, where H and W represent the height and width of the image, respectively; The LRMS image is cropped to a size of The number of channels is C, where r represents the magnification of the image. Then, the LRMS image is downsampled r times to obtain the LLMS image, and the height of the LLMS image is Width The number of channels is C.

3. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 2, characterized in that: Step S2 specifically includes: S2.

1. Copy the input LLMS image and copy the number of channels of the LLMS image with the number of channels C to C×r 2 , get a The image is then rearranged to obtain an The image is obtained by using the convolutional neural network. The convolutional neural network for channel duplication and pixel rearrangement constitutes a multispectral image super-resolution enhancer network. The multispectral image super-resolution enhancer network is expressed as follows by formulas (1) and (2): In formulas (1) and (2), M represents the multispectral image; C represents the number of channels of the image; U pc Indicates that the channels of the multispectral image M are copied, and the copy multiple is r 2 ;P sf It represents pixel rearrangement, which expands the height and width of the multispectral image M; S2.

2. Train the multispectral image super-resolution enhancement sub-network by designing the loss function L lspe Implementation, loss function L lspe It is expressed as follows using formula (3): L lspe =‖LRMS-P sf (U pc (LLMS))‖1 (3) In formula (3), LLMS is the input of the multispectral image super-resolution enhancer network, and LRMS is the supervisory signal for training the multispectral image super-resolution enhancer network.

4. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 2 or 3, characterized in that: Step S3 specifically includes: By loading the multispectral image super-resolution enhancer network trained in step S2, the LRMS image is super-resolved using the trained multispectral image super-resolution enhancer network, so that the height and width of the LRMS image are both expanded by R times, and a high spatial resolution and low-quality multispectral image, namely, an HLMS image, is obtained. The size of the HLMS image is H×W and the number of channels is C.

5. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 4, characterized in that: Step S4 includes: S4.1, processing the HLMS image and the PAN image; S4.2, construct a frequency detail injection module and train it, input the features of the HLMS image and PAN image processed in step S4.1 into the trained frequency detail injection module, and output the HMS image; S4.

3. Perform weighted addition of the HMS image and the HLMS image obtained in step S3 by an unsupervised learning method to obtain a high spatial resolution multispectral image, namely, a HRMS image.

6. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 5, characterized in that: Step S4.1 specifically includes: Two convolutional layers are used to convolve the input HLMS image and PAN image respectively to extract features, so that the number of channels of the HLMS image and PAN image becomes 32.

7. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 6, characterized in that: Step S4.2 specifically includes: Construct a frequency detail injection module and train it to inject the feature F of the HLMS image extracted in step S4.1 into the HLMS and the feature F of PAN image PAN Input to the frequency detail injection module: First, after Fourier transform, we get F HLMS and F PAN The amplitude and phase of these two features; then use F PAN The amplitude and phase of the HLMS The amplitude and phase of the HLMS image are extracted by the convolution network and the difference is put into a 1×1 convolution network for feature extraction. The extracted features are mapped to high dimensions through the ReLu activation function. The difference of the amplitude and phase after the convolution network feature extraction and the ReLu activation function is added to the amplitude and phase of the HLMS image after the 1×1 convolution network feature extraction and the ReLu activation function. The obtained amplitude and phase are input into the inverse Fourier transform, converted from the frequency domain to the spatial domain, and then passed through a 3×3 convolution network to obtain the output of the frequency detail injection module, that is, the HMS image; the frequency detail injection module performs feature extraction on the HLMS image F HLMS and the feature F of PAN image PAN The processing process is expressed by formula (4) to formula (8) as follows: A HLMS ,P HLMS =F(HLMS) (4) A PAN ,P PAN =F(PAN) (5) A HMS =f c1 (cat(f f (f c1 (A HLMS )),f f (f c1 (|A PAM -A HLMS |)))) (6) P HMS =f c1 (cat(f f (f c1 (P HLMS )),f f (fc1(|P PAN -P HLMS |)))) (7) HMS=f c3 (F -1 (A HMS ,P HMS )) (8) In formula (4) to formula (8), F represents Fourier transform; F -1 represents inverse Fourier transform; A and P represent the amplitude and phase after Fourier transform, respectively. HLMS represents the amplitude of the HLMS image after Fourier transformation, P HLMS A represents the phase of the HLMS image after Fourier transformation. PAN represents the amplitude of the PAN image after Fourier transformation, P PAN represents the phase of the PAN image after Fourier transformation, A HMS represents the amplitude of the HMS image after Fourier transformation, P HMS represents the phase of the HMS image after Fourier transformation. The HMS on the left side of formula (8) represents the HMS image; f c1 represents a 1×1 convolutional network; f c3 represents a 3×3 convolutional network; f f ReLu activation function; cat represents the concatenation of channel dimensions.

8. The dual-domain unsupervised remote sensing image pan-sharpening method based on super-resolution enhancement according to claim 7, characterized in that: Step S4.3 specifically includes: For LRMS images and PAN images, unsupervised learning method reasoning is performed by designing a loss function, and the loss function is designed as follows: 1) Spectral loss function L in spatial domain spe , which is expressed by formula (9) as follows: L spe =‖HRMS-HLMS‖1 (9) 2) Structural loss function L in spatial domain spa , which is expressed by formula (10) and formula (11) as follows: ρ σ (h,w,b)=Cov(PAN σ (h,w),HRMS σ (h,w,b)) (10) In formula (10) and formula (11), ρ σ (h, w, b) represents the local correlation between the PAN image and the HRMS image, σ represents the size of the local block, (h, w) represents the pixel position coordinates of the starting point of the local block, b represents the channel number, and Cov represents the size of the correlation coefficient between the PAN image and the HRMS image; PAN σ (h,w) represents the region of size σ starting from the coordinates (h,w) in the PAN image; HRMS σ (h,w,b) represents the region of size σ with coordinates (h,w) as the starting point in the HRMS image; ρ σ,ref The local correlation coefficient between the PAN image and the LRMS image is calculated using formula (10) as a reference value; Represents the value of the structure loss function in the spatial domain between local blocks with a starting pixel (h, w) and a size of σ on channel b of the PAN image and the HRMS image; The final L is obtained by averaging the local losses of all channels and each pixel of the PAN image and the HRMS image. spa ; 3) Frequency domain loss function L fre , which is expressed by formula (12) as follows: L fre =‖A HRMS -A PAN ‖1+‖P HRMS -P PAN ‖1 (12) In formula (12), A HRMS represents the amplitude of the HRMS image after Fourier transformation, P HRMS represents the phase of the HRMS image after Fourier transformation; 4) The complete loss function is expressed as follows through formula (13): L=L fre +λ1L spa +λ2L spe (13) In formula (13), λ1 and λ2 represent hyperparameters, both of which are constants.

Images