A pan-color sharpening method based on a deep recurrent network of a degradation model

Abstract

This invention discloses a panchromatic sharpening method based on a deep recurrent network of a degradation model. The main steps include: analyzing and constructing a degradation model for high-resolution multispectral images; obtaining maximum a posteriori probability expressions for panchromatic and low-resolution multispectral images based on global nonlinear smoothing constraints according to the degradation model; constructing residual group convolutional neural network modules to replace various components in the expressions for panchromatic and low-resolution multispectral images respectively; alternately cascading the constructed panchromatic and low-resolution multispectral image expression networks to form a deep recurrent fusion network; uniformly training the deep recurrent fusion network using a training dataset, and outputting the final fusion result. The method described in this invention achieves good subjective and objective results and makes the network interpretable to a certain extent. Therefore, this invention is an effective panchromatic sharpening method based on a deep recurrent network of a degradation model.

Description

Technical Field

[0001] This invention relates to image fusion technology, specifically to a panchromatic sharpening method based on a deep recurrent network of a degradation model, belonging to the field of image processing. Background Technology

[0002] Image fusion is a research hotspot in the field of information fusion, and panchromatic sharpening is an image information fusion technique that can effectively fuse multispectral and panchromatic images. Panchromatic sharpening utilizes correlation and complementarity to reconstruct a new multispectral image from multispectral and panchromatic image data acquired by different sensors of the same scene. This new multispectral image incorporates rich spectral and spatial structural information. Panchromatic sharpening technology has significant application value for multispectral images in remote sensing mapping, military target detection, and agricultural meteorological analysis.

[0003] Pancolor sharpening methods mainly include traditional methods and learning-based methods. Among the traditional methods, model-based methods have been extensively studied. They solve the optimal solution of the inverse problem by constructing a degradation model of the high-resolution multispectral image. Learning-based methods are currently the most mainstream, but they rely on a large amount of training data. Most current pancolor sharpening methods rely on deep learning technology, obtaining the fused image through end-to-end black-box training. They do not combine the advantages of traditional model-based methods and lack interpretability. Summary of the Invention

[0004] The purpose of this invention is to propose a novel panchromatic sharpening method that combines traditional model-based and learning-based methods to achieve a panchromatic sharpening method that balances fusion performance and interpretability. First, a degradation model for high-resolution multispectral images is constructed, thereby obtaining maximum a posteriori probability expressions for panchromatic and low-resolution multispectral images based on global nonlinear smoothing constraints. Then, by replacing various components in these expressions, representation networks for panchromatic and low-resolution multispectral images are constructed. Finally, these networks are alternately concatenated to form a deep recursive fusion network.

[0005] The pancolor sharpening method based on a degradation model deep recurrent network proposed in this invention mainly includes the following steps:

[0006] (1) Analyze and construct a degradation model for high-resolution multispectral images;

[0007] (2) Based on the model in step (1), construct the maximum a posteriori probability expressions for the panchromatic image and the low-resolution multispectral image based on the global nonlinear smoothing constraint respectively.

[0008] (3) For the expressions of panchromatic images and low-resolution multispectral images, a residual group convolutional neural network module is built to replace each link in the expression respectively;

[0009] (4) The panchromatic image and low-resolution multispectral image representation networks constructed by alternating tandem connections constitute a deep recursive fusion network;

[0010] (5) Use the training image dataset to train the network constructed in step (4);

[0011] (6) Using low-resolution multispectral images and panchromatic images as network inputs, the final fusion result is output. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of image degradation of high-resolution multispectral images in this invention.

[0013] Figure 2 This is a schematic diagram illustrating the principle of the full-color sharpening method based on a deep recurrent network of a degradation model, as described in this invention.

[0014] Figure 3 This is a comparison chart of the fusion results of the present invention and five other methods on the test image "Town" (downsampling factor of 4, Gaussian blur kernel size of 5×5, standard deviation of 1.5): where (a) is the reference image, (b) is the panchromatic image, (c) is the multispectral image magnified by bicubic interpolation by 4 times, and (d)(e)(f)(g)(h)(i) are the reconstruction results of method 1, method 2, method 3, method 4, method 5 and the present invention, respectively. Detailed Implementation

[0015] The present invention will be further described below with reference to the accompanying drawings:

[0016] Figure 1 The degradation of high-resolution multispectral images is demonstrated. The panchromatic image is obtained by reducing the spectrum of the high-resolution multispectral image, while the low-resolution multispectral image is obtained by downsampling and blurring the high-resolution multispectral image.

[0017] Specifically, Figure 1 The high-resolution image degradation model constructed in this paper is used to analyze the expression of high-resolution images based on panchromatic images and low-resolution multispectral images.

[0018] The representations of high-resolution images based on panchromatic images and low-resolution multispectral images are as follows:

[0019]

[0020]

[0021] H represents a high-resolution multispectral image. For the high-resolution multispectral image observer, P is the panchromatic image, L is the low-resolution multispectral image, S is the spectral transformation matrix, D is the downsampling matrix, K is the blur and de-enhancing matrix, and λ is the regularization parameter. P and D L These are constraints for panchromatic images and constraints for low-resolution multispectral images, respectively.

[0022] The iterative solution formulas for high-resolution multispectral images are thus obtained as follows:

[0023]

[0024]

[0025] get:

[0026]

[0027]

[0028] The final high-resolution multispectral image obtained by solving the inverse problem of the panchromatic image and the low-resolution multispectral image is as follows:

[0029] H k+1 =Φ αk ·P+Φ βk ·D P (H k )

[0030] H k+1 =Ψ αk ·L+Ψ βk ·D L (H k )

[0031] Ψ αk and Ψ βk The low-resolution multispectral image L and the constraint term D are respectively. L (H k The parameter matrix Φ αk and Φ βk The panchromatic image P and the constraint term D are respectively. P (H k The parameter matrix of ).

[0032] Figure 2 In this paper, the pancolor sharpening method based on a deep recurrent network of a degradation model can be divided into the following steps:

[0033] (1) Based on the degradation model of the constructed high-resolution multispectral image and the expression of the high-resolution multispectral image based on the inverse problem of panchromatic image and low-resolution multispectral image, a residual group convolutional neural network module is built to replace each link in the expression respectively.

[0034] (2) The panchromatic image and low-resolution multispectral image representation networks constructed by alternating tandem connections constitute a deep recursive fusion network;

[0035] (3) Use the training image dataset to train the network constructed in step (2);

[0036] (4) Using low-resolution multispectral images and panchromatic images as network inputs, the final fusion result is output.

[0037] Specifically, in step (1), a residual group convolutional neural network module is used to replace the link in the high-resolution multispectral image expression based on solving the inverse problem of panchromatic image and low-resolution multispectral image. The residual group module includes two 3×3 convolutional layers and skip connections.

[0038] In step (2), such as Figure 2 The diagram shows an alternating series connection of panchromatic and low-resolution multispectral image representation networks, chained K times to form a deep recursive fusion network. Because this network is constructed based on a high-resolution multispectral image degradation model, it possesses a degree of interpretability.

[0039] To verify the effectiveness of this invention, extensive comparative experiments were conducted on the commonly used remote sensing satellite Pléiades (including 2600 training images and 10 test images). In the experiments, this invention was compared with five typical remote sensing image fusion methods, including traditional component substitution fusion methods and fusion algorithms based on convolutional neural networks. These five panchromatic sharpening methods used for comparison are:

[0040] Method 1: The method proposed by Chavez et al., see reference "PS Chavez and AY Kwarteng, "Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis," Photogrammetric Engineering and Remote Sensing. 55(3), 339-348 (1989)."

[0041] Method 2: The method proposed by Garzelli et al., see reference "A. Garzelli, F. Nencini, and L. Capobianco, "Optimal MMSE pan-sharpening of very high resolution multispectral images," IEEE Transactions on Geoscience and Remote Sensing. 46(1), 228-236 (2008)."

[0042] Method 3: The method proposed by Masi et al., see reference "G. Masi, D. Cozzolino, and L. Verdoliva, "Pansharpening by convolutional neural networks," Remote Sensing. 8(7), 594(2016)."

[0043] Method 4: The method proposed by Yang et al., see reference "J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding, and J. Paisley, "PanNet: A deep network architecture for pan-sharpening," in Proc. IEEE Int. Conf. Comput. Vision. 1753-1761 (2017)."

[0044] Method 5: The method proposed by Wang et al., see reference "P. Wang and E. Sertel, "Channel-spatial Attention-based Pan-sharpening of Very High-resolution Satellite Images," Knowledge-based System. 11, 229 (2021)."

[0045] The comparative experiment is as follows:

[0046] Experiment 1 involved fusing and reconstructing multispectral and panchromatic images generated from 10 test images from the Pléiades remote sensing satellite using methods 1, 2, 3, 4, 5, and the method of this invention. In this experiment, a Gaussian blur kernel of size 5×5 with a standard deviation of 1.5 was used. Table 1 presents the CC (Correlation Coefficient), ERGAS (Relative Dimensionless Global Error in Synthesis), UIQI (Universal Image Quality Indexes), SAM (Spectral Angle Mapper), RASE (Relative Average Spectral Error), and RMSE (Root Mean Squared Error) parameters for the reconstruction results of each method. The optimal values ​​for CC and UIQI are 1, while the optimal values ​​for all other indicators are 0. Additionally, results for the "Town" image are provided for visual comparison. The original "Town" image, panchromatic image, multispectral image with bicubic interpolation and quadruple magnification, and the reconstruction results of each method are shown below. Figure 3 (a) Figure 3 (b) Figure 3 (c) Figure 3 (d), Figure 3 (e) Figure 3 (f) Figure 3 (g) Figure 3 (h) and Figure 3 As shown in (i).

[0047] Table 1

[0048] Evaluation indicators Method 1 Method 2 Method 3 Method 4 Method 5 This invention CC 0.8645 0.9025 0.9432 0.9866 0.9889 0.9909 ERGAS 11.5215 5.9848 4.5052 1.9060 1.7134 1.5261 UIQI 0.8462 0.8896 0.9341 0.9862 0.9885 0.9909 SAM 2.9321 4.1184 3.5598 2.0587 2.3604 2.0894 RASE 54.1324 23.5571 18.1121 8.6301 7.4265 6.6595 RMSE 331.9775 144.0836 110.3092 50.9920 45.2942 40.6068

[0049] from Figure 3The experimental results show that traditional fusion methods 1 and 2 produce noticeable blurring and spectral distortion, resulting in poor image visual quality. For fusion methods 3, 4, and 5 based on convolutional neural networks, all three methods effectively avoid spectral distortion, although methods 3, 4, and 5 exhibit image blurring. The results from this invention show the best preservation of spectral information, some detail recovery, better edge retention, and superior visual quality. Furthermore, based on the evaluation metrics given in Table 1, this invention achieves the highest values ​​across all metrics compared to other conventional pancolor sharpening methods, demonstrating significant improvement. Therefore, a comprehensive comparison of the subjective visual effects and objective evaluation metrics of the fusion results from each method shows that this invention can obtain high-quality fused images and possesses a certain degree of interpretability. In conclusion, this invention is an effective pancolor sharpening method based on a deep recurrent network of a degradation model.

Claims

1. A panchromatic sharpening method based on a deep recurrent network of a degradation model, characterized in that... Includes the following steps: Step 1: Analyze and construct a degradation model for high-resolution multispectral images. Panchromatic images are obtained from high-resolution multispectral images by reducing the spectrum, while low-resolution multispectral images are obtained from high-resolution multispectral images by downsampling and blurring degradation. Step 2: Based on the model in Step 1, construct maximum a posteriori probability expressions for panchromatic images and low-resolution multispectral images based on global nonlinear smoothing constraints. The expression aims to minimize the sum of the error term and the regularization term. Step 3: For the expressions of panchromatic images and low-resolution multispectral images, build a residual group convolutional neural network module to replace each part in the expression respectively; Step 4: The panchromatic image and low-resolution multispectral image representation networks constructed by alternating cascade connections form a deep recursive fusion network; Step 5: Train the network constructed in Step 4 using the training image dataset; Step 6: Using low-resolution multispectral images and panchromatic images as network inputs, output the final fusion result.

2. The panchromatic sharpening method based on a deep recurrent network of a degradation model according to claim 1, characterized in that... The degradation modeling analysis of the high-resolution multispectral image described in step one mainly includes the maximum a posteriori probability expression of the panchromatic image based on global nonlinear smoothing constraints and the maximum a posteriori probability expression of the low-resolution multispectral image based on global nonlinear smoothing constraints. The panchromatic image is obtained by reducing the spectral channels of the high-resolution multispectral image, while the low-resolution multispectral image is obtained by downsampling and blurring the high-resolution multispectral image. Based on the degradation modeling results, the maximum a posteriori probability expressions of the panchromatic image and the low-resolution multispectral image based on global nonlinear smoothing constraints are constructed respectively.

3. The panchromatic sharpening method based on a deep recurrent network of a degradation model according to claim 1, characterized in that... The alternating cascaded deep recursive panchromatic image and low-resolution multispectral image representation fusion network described in step four mainly includes a panchromatic image and low-resolution multispectral image representation network based on residual groups. After each link in the expression of the panchromatic image and the low-resolution multispectral image is replaced by a residual group convolutional neural network module, they are alternately cascaded to form a deep recursive fusion network.

Images