An unsupervised pan-sharpening method based on double cycle consistency

Abstract

The present invention discloses an unsupervised pan-sharpening method based on double cycle consistency. This method addresses the problems that existing supervised learning pan-sharpening methods rely on true values, resulting in poor model generalization, and that unsupervised learning pan-sharpening methods simply stack convolutional layers, resulting in ineffective interaction and fusion of spatial information and spectral information. Secondly, based on a generative adversarial network and an unsupervised pan-sharpening algorithm with double cycle consistency, a cyclic interaction network architecture from the source image domain to the target image domain is designed. Among them, the parallel cross-resolution fusion generator network realizes the full fusion of spatial and spectral features while ensuring that high-resolution features are not lost. The spatial information extraction pre-training network module based on cross-domain mining realizes the accurate representation of the spatial information of multispectral images. Experimental results show that this method is superior to existing unsupervised pan-sharpening methods in both quantitative and qualitative evaluation, and improves the pan-sharpening performance on full-resolution images.

Description

Technical Field

[0001] The invention belongs to the field of multispectral image processing, and in particular relates to an unsupervised panchromatic sharpening method based on double cycle consistency. Background Art

[0002] Pan-sharpening aims to produce images with high spatial and spectral resolution to meet the accuracy and precision requirements of applications such as disaster detection and environmental monitoring. However, in practice, due to the physical limitations of satellite imaging sensors, the spatial and spectral resolutions of remote sensing images often conflict with each other, making high-resolution multispectral images difficult to obtain. In the past few years, researchers have primarily explored supervised pan-sharpening methods using deep learning. Supervised methods preprocess the original multispectral and panchromatic images to generate training data at reduced resolution. The original multispectral images are then used as ground truth for supervised training, and the trained model is directly applied to the original images for pan-sharpening. However, high-resolution multispectral image data is difficult to obtain, requiring additional processing to generate simulated data for network training. Furthermore, supervised methods suffer from poor generalization performance on original full-resolution images, making them difficult to directly apply. Consequently, unsupervised pan-sharpening methods have gradually gained attention. However, existing unsupervised methods fuse spatial and spectral information in a limited manner, resulting in inaccurate extraction of spatial information from multispectral images and a failure to reflect the relationship between multispectral and panchromatic images, resulting in poor pan-sharpening results. Therefore, how to fully integrate spatial-spectral information and explore the connection between multispectral images and panchromatic images is crucial to the performance of panchromatic sharpening.

[0003] Existing pan-sharpening methods can be divided into two categories: traditional methods and deep learning-based methods. Traditional methods utilize detail injection models to inject spatial detail information from the panchromatic image into the multispectral image, thereby generating a high-resolution multispectral image. While these methods offer strong interpretability, linear models lack the ability to estimate spatial information and often suffer from spatial or spectral distortion. Deep learning-based pan-sharpening methods are primarily divided into supervised and unsupervised methods. Supervised methods typically treat pan-sharpening as an image super-resolution or image fusion problem, typically utilizing CNNs or GANs to perform pan-sharpening. Image super-resolution-based methods directly treat the panchromatic image as an additional band to the multispectral image, improving the super-resolution network architecture to achieve pan-sharpening results. Image fusion-based methods often employ a two-branch network to extract features from the panchromatic and multispectral images separately, then simply fuse the features of the two through the network. However, supervised methods require ground truth during network training, requiring artificially generated training data. Preprocessing the original-resolution image to reduce its resolution, which is then used as the ground truth. Because high-resolution information is lost during preprocessing, supervised networks are unable to learn this information, limiting the network's learning ability and resulting in poor generalization performance at the original resolution. Unsupervised methods do not require ground truth, reducing preprocessing steps and directly fusing images at the original resolution without any information loss at the input. The main network of most existing unsupervised methods is a serial structure. To reduce computational complexity, the feature map is first reduced to a low resolution, and then the high resolution is restored from the low resolution. This process may result in information loss. In general, deep learning-based methods can achieve better pan-sharpening results by leveraging the nonlinear mapping capabilities of neural networks.

[0004] However, existing methods still have the following problems: (1) Poor generalization: supervised deep learning methods are trained on reduced-resolution images, making it difficult to obtain good full-resolution pan-sharpening results and lacking practicality. (2) Incomplete feature extraction: existing unsupervised deep learning methods use a serial network architecture, ignoring the differences and complementarities between different resolutions, resulting in information loss. (3) Inaccurate feature mining: linear models are mostly used to fit the spatial information of multispectral images, which cannot accurately extract key features in the spatial domain. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned existing work, provide an unsupervised panchromatic sharpening method based on double cycle consistency, design a cyclic interactive network architecture from the source image domain to the target image domain, improve the network's feature learning ability for the input panchromatic image and multispectral image, and realize the accurate representation of the spatial information of the multispectral image.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] An unsupervised pan-sharpening method based on double cycle consistency, comprising the following steps:

[0008] 1) The panchromatic image PAN and the low-resolution multispectral image LRMS are input into the generator network to generate the pan-sharpened high-resolution multispectral image HRMS. The generator network consists of three stages: the first is the multi-resolution initial feature extraction module, followed by the multi-resolution feature fusion module repeated three times, and finally the pan-sharpened result is obtained through the high-resolution feature reconstruction module. Assuming that the pan-sharpened image obtained first is the true value, the discriminator sets its label as true.

[0009] 2) In the full-color image domain loop, HRMS is input into the spatial information extraction module of cross-domain mining to extract its spatial information, thereby obtaining a fake full-color image fake PAN. LRMS and fake PAN are outputted again through the generator to obtain a fake full-color sharpening result HRMS1. The similarity and consistency between the full-color sharpening images before and after the loop are calculated. HRMS and HRMS1 are input into the discriminator for discrimination, and the discriminator sets the label of HRMS1 to false.

[0010] 3) In the multispectral image domain loop, HRMS is blurred and downsampled to obtain its spectral information as a fake low-resolution multispectral image fake LRMS. The fake LRMS and PAN are then passed through the generator again to obtain a fake pan-sharpened result HRMS2. The discriminator sets the label of HRMS2 to false.

[0011] 4) Calculate the total loss function of the generator network, including QNR loss, cycle consistency loss, adversarial loss, spatial loss, and spectral loss, and perform back gradient propagation based on the total loss function to optimize the generator network parameters;

[0012] 5) Use the discriminator network to discriminate the high-resolution multispectral images before and after the cycle, calculate the loss function, perform back gradient propagation based on the loss function, and optimize the discriminator network parameters.

[0013] A further improvement of the present invention is that in step 1), the multi-resolution initial feature extraction module uses three branches to obtain initial features of different resolutions, wherein the three branches are a high-resolution panchromatic image branch, a low-resolution multispectral image branch, and an intermediate-resolution stitching branch.

[0014] A further improvement of the present invention is that the panchromatic image branch always maintains a high resolution four times that of the low-resolution multispectral image. First, a 3×3 convolution is used to obtain the initial features of the panchromatic image; the multispectral image branch always maintains low-resolution features. In order to provide more spectral information, a 1×1 convolution is used to obtain the initial features of the multispectral image; then a convolution with a step size of 2 is applied to the initial features of the panchromatic image, and a transposed convolution with a step size of 2 is applied to the initial features of the multispectral image. Finally, the channel dimension is spliced ​​to obtain the initial features of the spliced ​​branch; the input low-resolution multispectral image is recorded as The number of pixels is w×h, and the number of spectral bands is C; the full-color image is denoted as Where r = 4 represents the spatial resolution ratio of P and m. The process of acquiring initial features at different resolutions is expressed as follows:

[0015]

[0016] Among them, h1, h2, and h3 represent the initial features obtained by three branches with different resolutions. The resolutions of the panchromatic image branch and the stitching branch are four times and twice that of the low-resolution multispectral image branch, respectively; f c1 (·) and f c3 (·) denotes the convolution operation with kernel size of 1×1 and 3×3, respectively, and f dc (·) and f uc (·) respectively represent the convolution operation with a stride of 2 and the transposed convolution operation with a stride of 2, and concat(·) represents the splicing operation in the channel dimension.

[0017] A further improvement of the present invention is that, in step 1), in the multi-resolution feature fusion module, the low-resolution features of the multispectral image branch contain rich spectral information, the panchromatic image branch contains more accurate high-resolution spatial features, and the splicing branch is the fusion of spatial features and spectral features at an intermediate level of resolution; for three parallel features with different spatial resolutions, residual blocks are introduced to enhance feature extraction and obtain more expressive features at a deeper level; multi-resolution feature fusion is performed at three different resolutions; the features of the three branches of the same depth are fully multi-directionally flowed and interacted at multiple scales, so that the spectral features and spatial features are fully fused at different resolution levels; when fusion is performed on a certain branch, the features of the other two branches are appropriately transformed according to the target resolution size, using one or two 3×3 convolutions or transposed convolutions with a step size of 2, and then pixel-by-pixel addition for fusion; the multi-resolution feature fusion module is expressed as follows:

[0018]

[0019] Among them, i1, i2, and i3 represent the input features of the three branches with different resolutions of the module, and o1, o2, and o3 represent the output features of the three branches with different resolutions of the multi-resolution feature fusion module.

[0020] A further improvement of the present invention is that in step 1), the high-resolution feature reconstruction module uses the output features of three different resolutions to reconstruct features on the high-resolution branch in order to obtain the final high-resolution multispectral image; the three branches from high to low resolution are respectively re-adjusted to high-resolution features using 3×3 convolution, one transposed convolution with a step size of 2, and two transposed convolutions with a step size of 2; this process is expressed as follows:

[0021]

[0022] Where: i h 、i m 、i l Represent the input features of the three branches with different resolutions of the high-resolution feature reconstruction module, o h 、o m 、o l Represent the three adjusted high-resolution features respectively; then the adjusted high-resolution features are spliced ​​in the channel dimension, and the spliced ​​features are subjected to two 3×3 convolutions and one 1×1 convolution to obtain a high-resolution difference image M res , which is expressed as follows:

[0023] M res =f c1 f c3 (concat(o h ,o m ,o l ))

[0024] Finally, it is injected into the upsampled multispectral image M up In order to get the full color sharpening result M:

[0025] M=M up +M res .

[0026] A further improvement of the present invention is that, in step 2), in the spatial information extraction module of cross-domain mining, in order to return the high-resolution multispectral image after pan-sharpening to the panchromatic image domain, the linear regression coefficient between the multispectral image and the corresponding panchromatic image is not affected by the change of spatial resolution, and the coefficient at the low-resolution scale can be applied at the high-resolution scale; when the convolutional network is used to extract the spatial information of the low-resolution multispectral image, the parameters of the network can also realize the spatial information extraction of the high-resolution multispectral image; thus, a spatial information extraction network for cross-domain mining is designed, which consists of four layers of 3×3 convolutional layers, and except for the last layer, the remaining convolutional layers are all connected with a LeakyReLU activation function; the parameters of the spatial information extraction module of cross-domain mining are obtained by pre-training; during pre-training, the low-resolution multispectral image in the data set is used as the input of the network, and the downsampled panchromatic image is used as the true value; through training, the network learns the complex relationship between the multispectral image domain and the panchromatic image domain; after the pre-training is completed, it is migrated to our unsupervised network architecture to extract the spatial information of the pan-sharpened image M, which is expressed as the following formula:

[0027] I=f pre (M)

[0028] Among them, f pre (·) represents the spatial information extraction network function of cross-domain mining after pre-training, and I represents the spatial information of the extracted full-color sharpened image.

[0029] A further improvement of the present invention is that, in step 4), a spatial and spectral loss function is established between the pan-sharpened image and the input image, the spatial loss constrains the spatial similarity between the spatial information of HRMS and the input image PAN, and the spectral loss constrains the spectral similarity between the spectral information of HRMS and the input image LRMS; the pan-sharpened image requires more high-resolution spatial information of the pan-sharpened image to improve the spatial resolution, so the difference in high-frequency information between I and P is minimized to provide more high-frequency details; in order to further constrain the similarity between I and P in brightness, contrast and structure, a multi-level structural similarity index MS-SSIM is introduced; for the generator network of parallel multi-resolution fusion, MS-SSIM constrains the spatial similarity at different resolutions; therefore, the final spatial loss is expressed as:

[0030] L s =||hf(P)-hf(I)||1+(1-MSM(P,I))

[0031] Where hf(·) represents the extraction of high-frequency information of the image, and MSM(·) represents the MS-SSIM index;

[0032] The spectral information is preserved by constraining the difference in low-frequency information between the downsampled pan-sharpened image and the low-resolution multispectral image; similarly, MS-SSIM is used to ensure similarity at different resolutions. Therefore, the spectral loss is expressed as:

[0033]

[0034] Where lf(·) represents the extraction of low-frequency information of the image;

[0035] In addition, the QNR loss L is also used in the generator network Q , cycle consistency loss L C and adversarial loss L a , respectively expressed as:

[0036] L Q =1-(1-D λ )(1-D S )

[0037]

[0038]

[0039] Among them, D λ represents the spectral distortion metric, I represents the spatial information of the full-color sharpened image obtained by the pre-trained spatial information extraction network, G(·) represents the generator network, represents the downsampled pan-sharpened image, D(·) represents the discriminator network, and a and b represent the labels corresponding to the fake images;

[0040] Finally, the total loss function of the generator is expressed as:

[0041] L G =λ1L Q +λ2L c +λ3L s +λ4L b +λ5L a

[0042] Among them, λ1, λ2, λ3, λ4, and λ5 are the weight coefficients of each loss term, which are used to balance the magnitude of each loss function.

[0043] A further improvement of the present invention is that, in step 5), a discriminator network is used to discriminate the high-resolution multispectral images before and after the cycle; the high-resolution multispectral image obtained for the first time is used as the true value of pan-sharpening, and the two outputs obtained through the cycle are used as the false pan-sharpening results; the discriminator discriminates the false pan-sharpening images obtained after the two cycles respectively, and the discriminator adversarial loss function corresponding to each discrimination is expressed as:

[0044]

[0045]

[0046] Where c and d represent the labels corresponding to the pan-sharpened image and the fake pan-sharpened image after the loop, respectively; a, b, c, and d are soft labels; for the generator, labels a and b are random numbers in the range [0.7, 1.2]; for the discriminator, label c is a random number in the range [0.7, 1.2], and label d is a random number in the range [0, 0.3];

[0047] Since the discriminator performs two discriminations, the parameters of the discriminator network are shared, and the back propagation of the two loss functions accumulates the gradient of the network to update the parameters. Therefore, the loss function of the discriminator is expressed as:

[0048] L D =L d1 +L d2 .

[0049] The present invention has at least the following beneficial technical effects:

[0050] The present invention provides an unsupervised pan-sharpening algorithm based on double cycle consistency. In specific operation, the panchromatic image (PAN) and the low-resolution multispectral image (LRMS) are first input into the generator network to generate a pan-sharpened high-resolution multispectral image (HRMS). Assuming that the pan-sharpened image obtained for the first time is the true value, the discriminator sets its label as true. In the pan-sharpened image domain loop, the HRMS is input into the spatial information extraction module of cross-domain mining to extract its spatial information and obtain a false pan-sharpened image (fake PAN); the LRMS and fake PAN are outputted by the generator again to obtain a false pan-sharpened result HRMS1, and the similarity and consistency between the pan-sharpened images before and after the loop are calculated; the HRMS and HRMS1 are input into the discriminator for discrimination, and the discriminator sets the label of HRMS1 as false. In the multispectral image domain loop, the spectral information of the HRMS is obtained by blurring and downsampling the HRMS as a false low-resolution multispectral image fake LRMS. The fake LRMS and PAN are then passed through the generator again to produce a fake pan-sharpened result, HRMS2. The discriminator labels HRMS2 as fake. Finally, the total loss function of the generator network is calculated, including the QNR loss, cycle consistency loss, adversarial loss, spatial loss, and spectral loss. The discriminator network is then used to discriminate the high-resolution multispectral images before and after the cycle, and the loss function is calculated. Back gradient propagation is performed based on the loss function to optimize the discriminator network parameters.

[0051] Furthermore, the unsupervised panchromatic sharpening algorithm based on dual cycle consistency in the present invention addresses the problem of missing true values ​​in panchromatic sharpening network training. It proposes a cyclic interaction network from the low-resolution multispectral image domain and the panchromatic image domain to the high-resolution multispectral image domain, and designs a spatial information extraction pre-training network model based on cross-domain mining and a dual cycle loss to enhance the network's feature learning ability for the input panchromatic and multispectral images.

[0052] Furthermore, the parallel multi-resolution fusion generator network in the present invention designs a cross-resolution multi-directional information connection and interaction network to address the problem of high-resolution information loss during full-color sharpening. Sub-networks of different resolutions can achieve full fusion of spatial and spectral features while ensuring that high-resolution features are not lost, thereby ensuring the comprehensiveness and integrity of the information required for full-color sharpening.

[0053] Furthermore, in order to overcome the problem of inaccurate features caused by the linear extraction method of spatial information, the cross-domain mining spatial information extraction network in the present invention takes the downsampled panchromatic image as a reference, and explores the complex relationship between multispectral images and panchromatic images by designing a cross-domain mining spatial information extraction network, thereby achieving accurate representation of the spatial information of multispectral images.

[0054] In summary, the present invention provides an unsupervised pan-sharpening method based on dual cycle consistency. This method addresses the problems of existing supervised learning pan-sharpening methods, which rely on true values ​​and result in poor model generalization. Furthermore, unsupervised learning pan-sharpening methods simply stack convolutional layers, resulting in ineffective interaction and fusion of spatial and spectral information. Furthermore, the extraction of spatial information from multispectral images is imprecise, leading to poor pan-sharpening performance. An unsupervised pan-sharpening algorithm based on a generative adversarial network and dual cycle consistency is proposed. A recurrent interactive network architecture is designed, which transitions from the source image domain to the target image domain. A parallel cross-resolution fusion generator network achieves full fusion of spatial and spectral features while ensuring that high-resolution features are not lost. A pre-trained network module for spatial information extraction based on cross-domain mining enables accurate representation of spatial information in multispectral images. Furthermore, a cycle consistency loss is introduced to further improve pan-sharpening performance. Experimental results demonstrate that this method outperforms existing unsupervised pan-sharpening methods in both quantitative and qualitative evaluations, improving pan-sharpening performance on full-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 :Flowchart of the unsupervised pan-sharpening algorithm based on double cycle consistency of the present invention;

[0056] Figure 2 : Flowchart of the generator network in the present invention;

[0057] Figure 3: Flowchart of the multi-resolution initial feature extraction module in the present invention;

[0058] Figure 4 : Flowchart of the high-resolution feature reconstruction module in the present invention. DETAILED DESCRIPTION

[0059] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, unless there is a conflict, the embodiments of the present disclosure and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0060] like Figure 1 As shown, the present invention proposes an unsupervised full color sharpening method based on double cycle consistency, which includes the following steps:

[0061] 1) A panchromatic image (PAN) and a low-resolution multispectral image (LRMS) are fed into the generator network to generate a pan-sharpened high-resolution multispectral image (HRMS). The generator network consists of three stages: a multi-resolution initial feature extraction module, followed by a multi-resolution feature fusion module repeated three times, and finally a high-resolution feature reconstruction module to obtain the pan-sharpened result. Assuming that the first pan-sharpened image is the true value, the discriminator sets its label as true;

[0062] 2) In the full-color image domain loop, HRMS is input into the spatial information extraction module of cross-domain mining to extract its spatial information, resulting in a fake full-color image (fake PAN). LRMS and fake PAN are outputted again through the generator to obtain a fake full-color sharpening result HRMS1. The similarity and consistency between the full-color sharpening images before and after the loop are calculated. HRMS and HRMS1 are input into the discriminator for discrimination, and the discriminator sets the label of HRMS1 to false.

[0063] 3) In the multispectral image domain loop, the spectral information of the HRMS is obtained by blurring and downsampling, which is used as a fake low-resolution multispectral image fake LRMS. The fake LRMS and PAN are then passed through the generator again to obtain a fake pan-sharpened result HRMS2. The discriminator sets the label of HRMS2 to false.

[0064] 4) Calculate the total loss function of the generator network, including QNR loss, cycle consistency loss, adversarial loss, spatial loss, and spectral loss, and perform back gradient propagation based on the total loss function to optimize the generator network parameters;

[0065] 5) Use the discriminator network to discriminate the high-resolution multispectral images before and after the cycle, calculate the loss function, perform back gradient propagation based on the loss function, and optimize the discriminator network parameters.

[0066] The generator network extracts the spatial information of the panchromatic image and the spectral information of the low-resolution multispectral image, and fuses these two complementary information to generate a panchromatic sharpened image. In order to make full use of the specific information of the panchromatic image and the multispectral image, as well as the correlation between the two images, we designed a parallel multi-resolution fusion generator network structure to extract important information from different images in parallel and fuse the complementary information in a timely manner. The network structure of the generator is as follows: Figure 2 As shown in Figure 1, it includes three parallel branches with different resolutions. The main stages are: first, the acquisition of initial features with different resolutions, then the multi-resolution feature fusion repeated three times, maintaining high-resolution features throughout the entire process, and finally the full-color sharpening result is obtained through high-resolution feature reconstruction.

[0067] Multi-resolution initial feature extraction module such as Figure 3 As shown, three branches are used to obtain initial features of different resolutions. The three branches are a high-resolution panchromatic image branch, a low-resolution multispectral image branch, and an intermediate-resolution stitching branch. The panchromatic image branch always maintains a high resolution four times that of the low-resolution multispectral image. First, a 3×3 convolution is used to obtain the initial features of the panchromatic image. The multispectral image branch always maintains low-resolution features. In order to provide more spectral information, we use a 1×1 convolution to obtain the initial features of the multispectral image. Then, a convolution with a stride of 2 is used for the initial features of the panchromatic image, and a transposed convolution with a stride of 2 is used for the initial features of the multispectral image. Finally, the channel dimension is stitched to obtain the initial features of the stitching branch. The input low-resolution multispectral image is denoted as The number of pixels is w×h, and the number of spectral bands is C. The full-color image is denoted as Where r = 4 represents the spatial resolution ratio of P and m. The process of acquiring initial features of different resolutions can be expressed as follows:

[0068]

[0069] Among them, h1, h2, and h3 represent the initial features obtained by three branches with different resolutions. The resolutions of the panchromatic image branch and the stitching branch are four times and twice that of the low-resolution multispectral image branch, respectively. c1 (·) and fc3 (·) denotes the convolution operation with kernel size of 1×1 and 3×3, respectively, and f dc (·) and f uc (·) denotes a convolution operation with a stride of 2 and a transposed convolution operation with a stride of 2, respectively, and concat(·) denotes a concatenation operation in the channel dimension. The resulting initial features exploit the correlation between the panchromatic and multispectral images. Furthermore, the panchromatic and multispectral image branches explicitly utilize specific knowledge from each input image, helping the network to better reconstruct spatial and spectral features.

[0070] The low-resolution features of the multispectral image branch primarily contain rich spectral information, while the panchromatic image branch contains more precise high-resolution spatial features. The stitching branch fuses spatial and spectral features at intermediate resolutions. Residual blocks are introduced for each of the three parallel features at different spatial resolutions to enhance feature extraction and obtain more expressive features at deeper levels. Multi-resolution feature fusion is performed at all three resolutions. The features of the three branches at the same depth are fully integrated with multi-directional information flow and interaction at multiple scales, ensuring the full fusion of spectral and spatial features at different resolution levels. When fusion is performed on a particular branch, the features of the other two branches are appropriately transformed based on the target resolution. One or two 3×3 convolutions or transposed convolutions with a stride of 2 are performed, followed by pixel-by-pixel summation for fusion. For example, at the high-resolution level, a single transposed convolution with a stride of 2 is performed on the stitching branch, and two transposed convolutions with a stride of 2 are performed on the multispectral image branch. These are then pixel-by-pixel summed with the high-resolution features to produce the high-resolution output features. This module is expressed as follows:

[0071]

[0072] Here, i1, i2, and i3 represent the input features of the three branches of the module at different resolutions, and o1, o2, and o3 represent the output features of the three branches of the module at different resolutions. This multi-resolution feature fusion module enables each resolution branch to receive information from other parallel branches of different resolutions, leveraging the complementary differences between resolutions to generate richer feature representations and effectively fusing complementary information. At the same time, the parallel branches ensure the presence of high-resolution features without downsampling loss, thereby outputting reliable high-resolution feature representations.

[0073] High-resolution feature reconstruction modules such as Figure 4As shown in Figure 1, to obtain the final high-resolution multispectral image, we not only use the high-resolution branch features, but also use the output features of three different resolutions to reconstruct features on the high-resolution branch. The three branches from high to low resolution are resized to high-resolution features using 3×3 convolution, one transposed convolution with a stride of 2, and two transposed convolutions with a stride of 2. This process is expressed as follows:

[0074]

[0075] Where: i h 、i m 、i l Represent the input features of the three branches with different resolutions of the high-resolution feature reconstruction module, o h 、o m 、o l Represent the three adjusted high-resolution features respectively. Then the adjusted high-resolution features are spliced ​​in the channel dimension, and the spliced ​​features are subjected to two 3×3 convolutions and one 1×1 convolution to obtain a high-resolution difference image M. res , which is expressed as follows:

[0076]

[0077] Finally, it is injected into the upsampled multispectral image M up In order to get the full color sharpening result M:

[0078] M=M up +M res

[0079] In order to establish the spatial and spectral loss functions of the relationship between the full-color sharpened image and the input image, the spatial loss constrains the spatial similarity between the spatial information of HRMS and the input image PAN, and the spectral loss constrains the spectral similarity between the spectral information of HRMS and the input image LRMS. The full-color sharpened image requires more high-resolution spatial information of the full-color image to improve the spatial resolution, so it is necessary to minimize the difference in high-frequency information between I and P to provide more high-frequency details. However, this only provides pixel-level similarity. In order to further constrain the similarity between I and P in brightness, contrast and structure, the multi-level structural similarity index MS-SSIM is introduced. For the generator network of parallel multi-resolution fusion, MS-SSIM constrains the spatial similarity at different resolutions. Therefore, the final spatial loss can be expressed as:

[0080] L s =||hf(P)-hf(I)||1+(1-MSM(P,I))

[0081] Here, hf(·) represents the extraction of high-frequency information of the image, and MSM(·) represents the MS-SSIM index.

[0082] The spectral information is preserved by constraining the difference in low-frequency information between the downsampled pan-sharpened image and the low-resolution multispectral image. Similarly, MS-SSIM can be used to ensure similarity at different resolutions. Therefore, the spectral loss can be expressed as:

[0083]

[0084] Where lf(·) represents the extraction of low-frequency information of the image.

[0085] In addition, the present invention also uses QNR loss L in the generator network Q , cycle consistency loss L C and adversarial loss L a , respectively expressed as:

[0086] L Q =1-(1-D λ )(1-D S )

[0087]

[0088]

[0089] Among them, D λ represents the spectral distortion metric, I represents the spatial information of the full-color sharpened image obtained by the pre-trained spatial information extraction network, G(·) represents the generator network, represents the downsampled pan-sharpened image, D(·) represents the discriminator network, and a and b represent the labels corresponding to the fake images.

[0090] Finally, the total loss function of the generator can be expressed as:

[0091] L G =λ1L Q +λ2L c +λ3L s +λ4L b +λ5L a

[0092] Among them, λ1, λ2, λ3, λ4, and λ5 are the weight coefficients of each loss term, which are used to balance the magnitude of each loss function and avoid the obstruction of the training process of the entire model due to the excessive value of some loss functions.

[0093] The present invention uses a discriminator network to discriminate high-resolution multispectral images before and after the loop. The high-resolution multispectral image obtained first is used as the true value of pan-sharpening, and the two outputs obtained through the loop are used as the false pan-sharpening results. The discriminator needs to discriminate the false pan-sharpened images obtained after the two loops respectively. The discriminator adversarial loss function corresponding to each discrimination can be expressed as:

[0094]

[0095]

[0096] Where c and d represent the labels corresponding to the pan-sharpened image and the fake pan-sharpened image after recycling, respectively. a, b, c, and d are soft labels. For the generator, labels a and b are random numbers in the range [0.7, 1.2]. For the discriminator, label c is a random number in the range [0.7, 1.2], and label d is a random number in the range [0, 0.3] to prevent overfitting of the discriminator.

[0097] Since the discriminator performs two discriminations, the parameters of the discriminator network are shared, and the back propagation of the two loss functions accumulates the gradient of the network to update the parameters. Therefore, the loss function of the discriminator can be expressed as:

[0098] L D =L d1 +L d2

[0099] The experimental settings and experimental results are given below.

[0100] This paper trains and tests on the QuikBird (QB) dataset, whose panchromatic image (PAN) has a spatial resolution of 0.6 μm and the corresponding multispectral image (MS) has a spatial resolution of 2.4 μm, resulting in a spatial resolution ratio of 4. The dataset contains five pairs of original images, each consisting of a PAN image and a corresponding MS image. One pair is used for testing, and the others are used for training. 80% of the entire dataset was randomly selected as the training set, and the remaining 20% ​​was used as the validation set. Due to the large size of the original images and the limitations of hardware video memory, we followed conventional experimental practices and randomly cropped the images to form image patches for experimental training and testing. The size of the MS image training sample patches was 64×64, and the size of the PAN image training sample patches was 256×256. Furthermore, to better compare the visual results, we tested on larger patches: 100×100 for MS images and 400×400 for PAN image testing. The optimizer for the spatial information extraction network for cross-domain mining uses the Adam optimizer, with a learning rate of 0.0001 and a batch size of 8. Both the generator and the discriminator in the unsupervised pan-sharpening network use the Adam optimizer, with an initial learning rate of 0.0002 and a momentum of 0.9. The batch size is set to 8 and the epoch is set to 20. In addition, the weights of each term in the generator loss function are: λ1 = 1.0, λ2 = λ5 = 1 × 10 -3 ,λ3=λ4=5×10 -4 Network training is divided into two parts: first, pre-training the spatial information extraction network for cross-domain mining, and then training our unsupervised pan-sharpening network. This experiment uses an Nvidia 2080 Ti GPU for training and testing. The model is implemented in the PyTorch 1.7 framework using Python 3.6 under Ubuntu 18.04.

[0101] Experimental results and analysis

[0102] In the experimental results, the comparison algorithms are traditional methods including GS, PRACS, AWLP, and SFIM. Methods based on supervised learning include DiCNN, FusionNet, and FU-PSGAN. Methods based on unsupervised learning include PanGAN and UCGAN. The corresponding algorithm of the present invention is an unsupervised full-color sharpening algorithm (DCGAN) based on double cycle consistency. For different types of methods, their training processes are different. The unsupervised method is trained on full-resolution images, using the original full-color images and multispectral images as the input of the model, and does not require a reference image. The supervised method is trained on reduced-resolution images according to the Wald protocol, blurring the original full-color and multispectral images with a Gaussian filter, and downsampling them 4 times to obtain the reduced-resolution image required for the experiment, and then the original multispectral image is used as the true value. The traditional method does not require training, and the full-color sharpened image can be obtained by directly performing corresponding processing on the input image.

[0103] To facilitate comparison with other supervised learning-based methods, we conducted two types of tests: full-resolution testing and reduced-resolution testing using the Wald protocol. For full-resolution testing, the original panchromatic and multispectral images were directly tested, and no-reference evaluation metrics were calculated based on the pan-sharpened results. For reduced-resolution testing, the reduced-resolution image was used as input, and the pan-sharpened result was compared with the original multispectral image to calculate the reference evaluation metrics.

[0104] The first set of experiments: Comparison of reference metrics achieved by different pan-sharpening methods on the QB dataset. To evaluate the performance of each method in the resolution reduction experiment, we used three widely used reference evaluation metrics: spectral angle mapping (SAM), spatial correlation coefficient (SCC), and relative dimensionless global error (ERGAS). These three metrics assess the similarity between the pan-sharpened image and the reference image. Table 1 shows the quantitative comparison results of different pan-sharpening methods on the QB dataset. Among all the unsupervised learning-based methods, our method achieves the best results in terms of reference metrics.

[0105] The second set of experiments: Comparison of no-reference indicators achieved by different pan-sharpening methods on the QB dataset. In order to evaluate the quality of the fused image in the full-resolution experiment, we use three widely used no-reference evaluation indicators to quantitatively evaluate the experimental performance, namely D λ 、D S and QNR, respectively, are evaluated from the perspectives of spectral quality, spatial quality, and global quality. The quantitative results of different pan-sharpening methods on the QB dataset are shown in Table 4. As can be seen from the table, our proposed method performs better overall. λThe index is slightly lower than that of UCGAN. This is because UCGAN has a conversion branch in the MS domain, which is more conducive to preserving spectral information. However, our dual cycle consistency structure can take into account both spatial and spectral information, and can achieve satisfactory results overall. S In terms of indicators, DiCNN achieved the best result value, which may be because this method has fewer parameters, is less likely to overfit, and improves the generalization performance on full-resolution images. λ The poor performance in these indicators indicates that supervised learning methods are not applicable enough to full-resolution images.

[0106] Table 1. No-reference metric results of different pan-sharpening methods on the QB dataset

[0107]

[0108]

[0109] Table 2. Reference index results of different pan-sharpening methods on the QB dataset

[0110]

[0111] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made based on the present invention. Therefore, such modifications and improvements, which do not depart from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. An unsupervised pan-sharpening method based on double cycle consistency, characterized in that The following steps are involved: 1) The panchromatic image PAN and the low-resolution multispectral image LRMS are input into the generator network to generate the pan-sharpened high-resolution multispectral image HRMS. The generator network consists of three stages: the first is the multi-resolution initial feature extraction module, followed by the multi-resolution feature fusion module repeated three times, and finally the pan-sharpened result is obtained through the high-resolution feature reconstruction module. Assuming that the pan-sharpened image obtained first is the true value, the discriminator sets its label as true. 2) In the full-color image domain loop, HRMS is input into the spatial information extraction module of cross-domain mining to extract its spatial information, thereby obtaining a fake full-color image fake PAN. LRMS and fake PAN are outputted again through the generator to obtain a fake full-color sharpening result HRMS1. The similarity and consistency between the full-color sharpening images before and after the loop are calculated. HRMS and HRMS1 are input into the discriminator for discrimination, and the discriminator sets the label of HRMS1 to false. 3) In the multispectral image domain loop, HRMS is blurred and downsampled to obtain its spectral information as a fake low-resolution multispectral image fake LRMS. The fake LRMS and PAN are then passed through the generator again to obtain a fake pan-sharpened result HRMS2. The discriminator sets the label of HRMS2 to false. 4) Calculate the total loss function of the generator network, including QNR loss, cycle consistency loss, adversarial loss, spatial loss, and spectral loss, and perform back gradient propagation based on the total loss function to optimize the generator network parameters; 5) Use the discriminator network to discriminate the high-resolution multispectral images before and after the cycle, calculate the loss function, perform back gradient propagation based on the loss function, and optimize the discriminator network parameters.

2. The unsupervised pan-sharpening method based on double cycle consistency according to claim 1, characterized in that: In step 1), the multi-resolution initial feature extraction module uses three branches to obtain initial features of different resolutions, namely, a high-resolution panchromatic image branch, a low-resolution multispectral image branch, and an intermediate-resolution stitching branch.

3. The unsupervised pan-sharpening method based on double cycle consistency according to claim 2, characterized in that: The panchromatic image branch always maintains a high resolution four times that of the low-resolution multispectral image. First, a 3×3 convolution is used to obtain the initial features of the panchromatic image. The multispectral image branch always maintains low-resolution features. In order to provide more spectral information, a 1×1 convolution is used to obtain the initial features of the multispectral image. Then, a convolution with a step size of 2 is used for the initial features of the panchromatic image, and a transposed convolution with a step size of 2 is used for the initial features of the multispectral image. Finally, the channel dimension is spliced ​​to obtain the initial features of the spliced ​​branch. The input low-resolution multispectral image is denoted as The number of pixels is w×h, and the number of spectral bands is C; the full-color image is denoted as Where r = 4 represents the spatial resolution ratio of P and m. The process of acquiring initial features at different resolutions is expressed as follows: Among them, h1, h2, and h3 represent the initial features obtained by three branches with different resolutions. The resolutions of the panchromatic image branch and the stitching branch are four times and twice that of the low-resolution multispectral image branch, respectively; f c1 (·) and f c3 (·) denotes the convolution operation with kernel size of 1×1 and 3×3, respectively, and f dc (·) and f uc (·) respectively represent the convolution operation with a stride of 2 and the transposed convolution operation with a stride of 2, and concat(·) represents the splicing operation in the channel dimension.

4. The unsupervised pan-sharpening method based on double cycle consistency according to claim 3, characterized in that: In step 1), in the multi-resolution feature fusion module, the low-resolution features of the multispectral image branch contain rich spectral information, the panchromatic image branch contains more accurate high-resolution spatial features, and the splicing branch is the fusion of spatial features and spectral features at the intermediate level of resolution; for the three parallel features of different spatial resolutions, residual blocks are introduced to enhance feature extraction and obtain more expressive features at a deeper level; multi-resolution feature fusion is performed at three different resolutions; the features of the three branches of the same depth are fully multi-directionally flowed and interacted at multiple scales, so that the spectral features and spatial features are fully integrated at different resolution levels; when fusion is performed on a certain branch, the features of the other two branches are appropriately transformed according to the target resolution size, using one or two 3×3 convolutions or transposed convolutions with a step size of 2, and then pixel-by-pixel addition for fusion; the multi-resolution feature fusion module is expressed as follows: Among them, i1, i2, and i3 represent the input features of the three branches with different resolutions of the module, and o1, o2, and o3 represent the output features of the three branches with different resolutions of the multi-resolution feature fusion module.

5. The unsupervised pan-sharpening method based on double cycle consistency according to claim 3, characterized in that: In step 1), the high-resolution feature reconstruction module uses the output features of three different resolutions to reconstruct features on the high-resolution branch in order to obtain the final high-resolution multispectral image. The three branches from high to low resolution are re-adjusted to high-resolution features by using 3×3 convolution, one transposed convolution with a stride of 2, and two transposed convolutions with a stride of 2. The process is expressed as follows: Where: i h 、i m 、i l Represent the input features of the three branches with different resolutions of the high-resolution feature reconstruction module, o h 、o m 、o l Represent the three adjusted high-resolution features respectively; then the adjusted high-resolution features are spliced ​​in the channel dimension, and the spliced ​​features are subjected to two 3×3 convolutions and one 1×1 convolution to obtain a high-resolution difference image M res , which is expressed as follows: M res =f c1 f c3 (concat(o h ,o m ,o l )) Finally, it is injected into the upsampled multispectral image M up In order to get the full color sharpening result M: M=M up +M res 。 6. The unsupervised pan-sharpening method based on double cycle consistency according to claim 5, characterized in that: In step 2), in the spatial information extraction module of cross-domain mining, in order to return the high-resolution multispectral image after pan-sharpening to the panchromatic image domain, the linear regression coefficient between the multispectral image and the corresponding panchromatic image is not affected by the change of spatial resolution, and the coefficient at the low-resolution scale can be applied at the high-resolution scale; when the convolutional network is used to extract the spatial information of the low-resolution multispectral image, the parameters of the network can also realize the spatial information extraction of the high-resolution multispectral image; thus, a spatial information extraction network for cross-domain mining is designed, which consists of four layers of 3×3 convolutional layers, and except for the last layer, the other convolutional layers are followed by a LeakyReLU activation function; the parameters of the spatial information extraction module for cross-domain mining are obtained through pre-training; during pre-training, the low-resolution multispectral image in the data set is used as the input of the network, and the downsampled panchromatic image is used as the true value; through training, the network learns the complex relationship between the multispectral image domain and the panchromatic image domain; After pre-training is completed, it is transferred to our unsupervised network architecture to extract the spatial information of the full-color sharpened image M, which is expressed as follows: I=f pre (M) Among them, f pre (·) represents the spatial information extraction network function of cross-domain mining after pre-training, and I represents the spatial information of the extracted full-color sharpened image.

7. The unsupervised pan-sharpening method based on double cycle consistency according to claim 6, characterized in that: In step 4), the spatial and spectral loss functions of the relationship between the pan-sharpened image and the input image are established. The spatial loss constrains the spatial similarity between the spatial information of the HRMS and the input image PAN, and the spectral loss constrains the spectral similarity between the spectral information of the HRMS and the input image LRMS. The pan-sharpened image requires more high-resolution spatial information of the pan-sharpened image to improve the spatial resolution, so the difference in high-frequency information between I and P is minimized to provide more high-frequency details. In order to further constrain the similarity between I and P in brightness, contrast and structure, a multi-level structural similarity index MS-SSIM is introduced. For the generator network of parallel multi-resolution fusion, MS-SSIM constrains the spatial similarity at different resolutions. Therefore, the final spatial loss is expressed as: L s =||hf(P)-hf(I)||1+(1-MSM(P,I)) Where hf(·) represents the extraction of high-frequency information of the image, and MSM(·) represents the MS-SSIM index; The spectral information is preserved by constraining the difference in low-frequency information between the downsampled pan-sharpened image and the low-resolution multispectral image; similarly, MS-SSIM is used to ensure similarity at different resolutions. Therefore, the spectral loss is expressed as: Where lf(·) represents the extraction of low-frequency information of the image; In addition, the QNR loss L is also used in the generator network Q , cycle consistency loss L C and adversarial loss L a , respectively expressed as: L Q =1-(1-D λ )(1-D S ) Among them, D λ represents the spectral distortion metric, I represents the spatial information of the full-color sharpened image obtained by the pre-trained spatial information extraction network, G(·) represents the generator network, represents the downsampled pan-sharpened image, D(·) represents the discriminator network, and a and b represent the labels corresponding to the fake images; Finally, the total loss function of the generator is expressed as: L G =λ1L Q +λ2L c +λ3L s +λ4L b +λ5L a Among them, λ1, λ2, λ3, λ4, and λ5 are the weight coefficients of each loss term, which are used to balance the magnitude of each loss function.

8. The unsupervised pan-sharpening method based on double cycle consistency according to claim 7, characterized in that: In step 5), the discriminator network is used to discriminate the high-resolution multispectral images before and after the cycle; the high-resolution multispectral image obtained for the first time is used as the true value of the pan-sharpening, and the two outputs obtained through the cycle are used as the false pan-sharpening results; the discriminator discriminates the false pan-sharpening images obtained after the two cycles respectively, and the discriminator adversarial loss function corresponding to each discrimination is expressed as: Where c and d represent the labels corresponding to the pan-sharpened image and the fake pan-sharpened image after the loop, respectively; a, b, c, and d are soft labels; for the generator, labels a and b are random numbers in the range [0.7, 1.2]; for the discriminator, label c is a random number in the range [0.7, 1.2], and label d is a random number in the range [0, 0.3]; Since the discriminator performs two discriminations, the parameters of the discriminator network are shared, and the back propagation of the two loss functions accumulates the gradient of the network to update the parameters. Therefore, the loss function of the discriminator is expressed as: L D =L d1 +L d2 。

Images