Pan-sharpening method for obtaining super-resolution fused images

Abstract

The present invention discloses a panchromatic sharpening method capable of obtaining a super-resolution fused image. The method mainly comprises the following steps: building a panchromatic image super-resolution reconstruction network model based on a panchromatic iterative projection block group; pre-training the model to output a super-resolution double-reconstructed panchromatic image; building a primary fusion network module for input low-resolution multispectral images and panchromatic images; building a multi-image fusion network module for input low-resolution multispectral images, primary fused multispectral images, and super-resolution reconstructed panchromatic images; the primary fusion network module and the multi-image fusion network module together constitute a multi-level fusion network, and the fusion network is uniformly trained using a training data set to output the final fusion result. The fusion method described in the present invention can achieve good subjective and objective effects. Therefore, the present invention is an effective panchromatic sharpening method capable of obtaining a super-resolution fused image.

Description

Technical Field

[0001] The present invention relates to image fusion technology, in particular to a full-color sharpening method capable of obtaining a super-resolution fused image, and belongs to the field of image processing. Background Art

[0002] Image fusion is a research hotspot in the field of information fusion. Pan-sharpening is an image fusion technique that reconstructs information from multispectral and panchromatic images. It combines multispectral and panchromatic image data acquired by different sensors for the same scene, leveraging correlation and complementarity to reconstruct a new, information-rich and robust multispectral image. Pan-sharpening technology is widely used in remote sensing mapping, military target detection, agricultural meteorological analysis, and other fields. Pan-sharpening improves the spatial resolution of multispectral images while preserving spectral information, providing rich image information for subsequent analysis.

[0003] There are three main types of panchromatic sharpening methods: component replacement-based methods, multi-resolution analysis-based methods, and learning-based methods. Component replacement-based methods are simple and fast, performing component replacement on upsampled multispectral images; multi-resolution analysis-based methods extract details from panchromatic images and inject them into upsampled multispectral images; and learning-based methods train convolutional neural network models to achieve panchromatic sharpening. Component replacement-based methods suffer from severe spectral distortion, multi-resolution analysis-based methods rely on the performance of decomposition methods and are prone to spatial blurring, while learning-based methods rely on large amounts of training data. Current panchromatic sharpening methods are limited to ensuring that the resolution of the fused image reaches that of the panchromatic image, and cannot directly obtain a fused image with a higher resolution than the panchromatic image through panchromatic sharpening. Summary of the Invention

[0004] The goal of this paper is to construct a novel pan-sharpening framework that, combined with super-resolution reconstruction techniques, further improves the resolution of the fused image. The input panchromatic image is first passed through a pre-trained panchromatic image super-resolution reconstruction network, and then pan-sharpened using a multi-image fusion network. This results in a pan-sharpening method that can produce a super-resolution fused image.

[0005] The present invention proposes a method for obtaining a super-resolution fusion image by performing full color sharpening, which mainly includes the following steps:

[0006] (1) Build a network model for full-color image super-resolution reconstruction using attention-iterative projection blocks as the main building block;

[0007] (2) Using the network in step (1), train a super-resolution reconstruction network model with a reconstruction factor of 2;

[0008] (3) For the input full-color image, super-resolution reconstruction is performed through the trained network, and the output is a full-color image with twice the resolution;

[0009] (4) Using the residual block group as the main building block, build the primary fusion network module;

[0010] (5) Using the multi-image multi-scale fusion block group as the main building block, build a multi-image fusion network module;

[0011] (6) Construct a multi-level fusion network model using the primary fusion network module and the multi-graph fusion network module;

[0012] (7) Using the training image dataset, train the network constructed in step (6);

[0013] (8) The multispectral image, the panchromatic image and the super-resolution reconstructed panchromatic image are used as network inputs, and the final fusion result is output. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a principle block diagram of the full-color sharpening method for obtaining a super-resolution fusion image according to the present invention.

[0015] Figure 2 This is a diagram of the network structure of full-color image super-resolution reconstruction of the present invention.

[0016] Figure 3 This is a structural diagram of the full-color iterative projection block group in the full-color image super-resolution reconstruction network of the present invention.

[0017] Figure 4 This is the primary fusion network structure diagram of the present invention.

[0018] Figure 5 It is a network structure diagram of the multi-graph fusion network of the present invention.

[0019] Figure 6 This is a comparison chart of the fusion results of the test image "Factory" between the present invention and five methods (downsampling factor is 8, Gaussian blur kernel size is 5×5, standard deviation is 1.5): among them, (a) is the reference image, (b) is the full-color image, (c) is the multispectral image magnified eight times by bicubic interpolation, and (d)(e)(f)(g)(h)(i) are the reconstruction results of method 1, method 2, method 3, method 4, method 5 and the present invention respectively.

[0020] Figure 7This is a comparison diagram of the fusion results of the test image "Dock" between the present invention and five methods (downsampling factor is 8, Gaussian blur kernel size is 5×5, standard deviation is 1.5): among them, (a) is the reference image, (b) is the full-color image, (c) is the multispectral image magnified eight times by bicubic interpolation, and (d)(e)(f)(g)(h)(i) are the reconstruction results of method 1, method 2, method 3, method 4, method 5 and the present invention respectively. DETAILED DESCRIPTION

[0021] The present invention will be further described below in conjunction with the accompanying drawings:

[0022] Figure 1 In the above, the full color sharpening method of super-resolution fusion image can be obtained, which can be divided into the following steps:

[0023] (1) Build a network model for full-color image super-resolution reconstruction using full-color iterative projection block groups as the main building blocks;

[0024] (2) Using the network in step (1), train a super-resolution reconstruction network model with a reconstruction factor of 2;

[0025] (3) For the input full-color image, super-resolution reconstruction is performed through the trained network, and the output is a full-color image with twice the resolution;

[0026] (4) Using the residual block group as the main building block, build the primary fusion network module;

[0027] (5) Using the multi-image multi-scale fusion block group as the main building block, build a multi-image fusion network module;

[0028] (6) Construct a multi-level fusion network model using the primary fusion network module and the multi-graph fusion network module;

[0029] (7) Using the training image dataset, train the network constructed in step (6);

[0030] (8) The multispectral image, the panchromatic image and the super-resolution reconstructed panchromatic image are used as network inputs, and the final fusion result is output.

[0031] Specifically, in step (1), the network model for full-color image super-resolution reconstruction is constructed with a full-color iterative projection block group as the main building unit, including: a full-color upsampling block, a full-color downsampling block and a jump connection.

[0032] Figure 2 It shows that after the full-color image is input into the 3×3 and 1×1 convolutional layers for preliminary processing, it passes through the full-color upsampling block and the full-color downsampling block in sequence, and undergoes several up- and down-sampling iterative projections to obtain the final super-resolution reconstruction of a 2x full-color image.

[0033] Figure 3 The structural diagram of the full-color upsampling block and full-color downsampling block is shown. The full-color upsampling block consists of an attention mechanism unit and an upsampling iterative back-projection unit. The attention mechanism unit includes parallel max pooling and average pooling stages. The output is concatenated, passed through two 1×1 convolutional layers, and then multiplied with the original input. The upsampling iterative back-projection unit first upsamples the input feature map through a 6×6 convolutional layer, then downsamples it through a 3×3 convolutional layer. The difference between the downsampled result and the input feature map is again upsampled through a 6×6 convolutional layer. The difference is then added to the output of the first upsampling via a skip connection to output the final upsampled result. The full-color downsampling block consists of an attention mechanism unit and a downsampling iterative back-projection unit. The attention mechanism unit includes parallel max pooling and average pooling stages. The output is concatenated, passed through two 1×1 convolutional layers, and then multiplied with the original input. The downsampling iterative back-projection unit first downsamples the input feature map through a 3×3 convolutional layer, then upsamples it through a 6×6 convolutional layer. The difference between the upsampled result and the input feature map is downsampled again through a 3×3 convolutional layer. The difference is then added to the output of the first downsampling layer via a skip connection, and the final downsampled result is output. The activation function is used in the excitation layer.

[0034] The loss function used by the network model for full-color image super-resolution reconstruction is expressed as:

[0035] L SR =L mse =||YX|| F

[0036] L mse is the mean square error loss function, where Y is the high-resolution panchromatic reference image and X is the predicted super-resolution reconstructed panchromatic image.

[0037] In the step (4), the constructed primary fusion network module is mainly composed of a conventional residual block group.

[0038] like Figure 4 The multispectral image and panchromatic image, upsampled by a factor of 4, are concatenated and fed into the primary fusion network module. After a 3×3 convolutional layer, the image passes through a conventional residual block group, consisting of two 3×3 convolutional layers and skip connections. Four residual blocks are cascaded to form a residual block group. After another 3×3 convolutional layer and a long skip connection, the final result is summed and output. The excitation layer uses the ReLU activation function.

[0039] In the step (5), the constructed multi-graph fusion network module is mainly composed of a multi-graph multi-scale fusion block group.

[0040] like Figure 5The multispectral image upsampled 8x, the primary fused image upsampled 2x, and the super-resolution reconstructed panchromatic image reconstructed 2x are concatenated and then fed into the multi-image fusion network module. The network module begins with a multi-scale feature extractor, which uses three different convolution kernels (3×3, 5×5, and 7×7) to extract feature information from images of different input resolutions. The outputs are concatenated and passed through a 3×3 convolution layer. The feature maps are then fed into a residual attention block group, which consists of a cascade of four residual attention blocks. Each residual attention block includes two 3×3 convolution layers, a parallel max pooling and average pooling step, two 1×1 convolution layers, and several skip connections. The outputs of the residual attention block group are then passed through a 3×3 convolution layer and summed with a long skip connection to produce the final output. The excitation layer uses the Rectangular Unit (ReLU) activation function.

[0041] In step (6), the multi-level fusion network is composed of the primary fusion network module in step (4) and the multi-graph fusion network module in step (5), and is trained uniformly. The loss function used by the network model of the multi-level fusion network is expressed as:

[0042] L fusion =L mse =||Y fusi o n -X ms || F

[0043] where Y fusion is the fused image, X ms The multispectral image is used as the reference.

[0044] To verify the effectiveness of the present invention, a large number of comparative experiments were conducted on the commonly used remote sensing satellites Pléiades (including 2,600 training images and 10 test images) and Jilin-1 (including 2,800 training images and 10 test images). In the experiments, the present invention was compared with five typical remote sensing image fusion methods, including traditional component replacement fusion methods and fusion algorithms based on convolutional neural networks. The five pan-sharpening methods used for comparison are:

[0045] Method 1: The method proposed by Chavez et al., reference “P.S. Chavez and A.Y. Kwarteng, “Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis,” Photogrammetric Engineering and Remote Sensing. 55(3), 339-348(1989).”

[0046] Method 2: The method proposed by Garzelli et al., reference “A. Garzelli, F. Nencini, and L. Capobianco, “Optimal MMSE pan-sharpening of very high resolution multispectral images,” IEEE Transactions on Geoscience and Remote Sensing. 46(1), 228-236 (2008).”

[0047] Method 3: The method proposed by Masi et al., reference “G.Masi, D.Cozzolino, and L.Verdoliva, “Pansharpening by convolutional neural networks,” Remote Sensing. 8(7), 594(2016).”

[0048] Method 4: Yang et al., “J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding, and J. Paisley, “PanNet: A deep network architecture for pan-sharpening,” in Proc. IEEE Int. Conf. Comput. Vision. 1753–1761 (2017).”

[0049] Method 5: Wang et al., reference “P. Wang and E. Sertel, “Channel-spatial Attention-based Pan-sharpening of Very High-resolution Satellite Images,” Knowledge-based System. 11, 229 (2021).”

[0050] The contents of the comparative experiment are as follows:

[0051] Experiment 1, method 1, method 2, method 3, method 4, method 5 and the method of the present invention are used to fuse and reconstruct the multispectral images and panchromatic images simulated from 10 test images of the remote sensing satellite Pléiades. In this experiment, the blur kernel is a Gaussian blur kernel with a size of 5×5 and a standard deviation of 1.5. Table 1 shows the CC (Correlation Coefficient), ERGAS (Relative Dimensionless Global Error in Synthesis), UIQI (Universal Image Quality Indexes), SAM (SpectralAngle Mapper), RASE (RelativeAverage Spectral Error) and RMSE (RootMean Squared Error) parameters of the reconstruction results of each method. Among them, the optimal value of the parameter CC and UIQI is 1, and the optimal values ​​of other indicators are 0. In addition, for visual comparison, the results of the "Factory" image are given. The "Factory" original image, panchromatic image, multispectral image bicubic interpolation fourfold magnification and the reconstruction results of each method are shown as follows. Figure 6 (a) Figure 6 (b) Figure 6 (c) Figure 6 (d) Figure 6 (e) Figure 6 (f) Figure 6 (g) Figure 6 (h) and Figure 6 (i) shown.

[0052] Table 1

[0053]

[0054]

[0055] Experiment 2, method 1, method 2, method 3, method 4, method 5 and the method of the present invention are used to fuse and reconstruct the multispectral images and panchromatic images simulated from 10 test images of the remote sensing satellite Jilin-1. In this experiment, the blur kernel is a Gaussian blur kernel with a size of 5×5 and a standard deviation of 1.5. Table 1 shows the CC (Correlation Coefficient), ERGAS (Relative Dimensionless Global Error in Synthesis), UIQI (Universal Image Quality Indexes), SAM (Spectral Angle Mapper), RASE (Relative Average Spectral Error) and RMSE (Root Mean Squared Error) parameters of the reconstruction results of each method. Among them, the optimal value of the parameter CC and UIQI is 1, and the optimal values ​​of other indicators are 0. In addition, for visual comparison, the results of the "Dock" image are given. The "Dock" original image, panchromatic image, multispectral image bicubic interpolation fourfold magnification and the reconstruction results of each method are shown as follows. Figure 7 (a) Figure 7 (b) Figure 7 (c) Figure 7 (d) Figure 7 (e) Figure 7 (f) Figure 7 (g) Figure 7 (h) and Figure 7 (i) shown.

[0056] Table 2

[0057] Evaluation indicators Method 1 Method 2 Method 3 Method 4 Method 5 The present invention CC 0.8902 0.9370 0.9579 0.9325 0.9417 0.9711 ERGAS 3.4151 1.9921 1.5418 1.9696 1.8529 1.2875 UIQI 0.6975 0.9363 0.9561 0.9298 0.9395 0.9699 SAM 2.5916 2.0507 1.8577 1.9173 1.7579 1.6132 RASE 13.8596 8.1264 6.2302 7.9775 7.4915 5.2039 RMSE 86.9826 51.1374 39.1907 50.0265 46.9957 32.7596

[0058] from Figure 6 and Figure 7The experimental results shown show that the results of traditional fusion methods 1 and 2 contain relatively obvious blurring and spectral distortion, resulting in poor image visual effects. For fusion methods 3, 4, and 5 based on convolutional neural networks, all three comparison methods are able to effectively avoid spectral distortion. Method 3 fully restores details, while methods 4 and 5 exhibit relatively obvious image blurring. The results of the present invention best preserve spectral information, restore details to a certain extent, better preserve edges, and achieve better visual effects. In addition, based on the various evaluation indicators given in Tables 1 and 2, compared with other conventional panchromatic sharpening methods, the present invention can obtain higher-resolution fused images due to the super-resolution reconstruction of the panchromatic image, achieving the highest values ​​in all indicators and significant improvements. Therefore, a comprehensive comparison of the subjective visual effects and objective evaluation indicators of the fusion results of each method shows that the present invention can obtain higher-resolution fused images with the best image quality. In summary, the present invention is an effective panchromatic sharpening method for obtaining super-resolution fused images.

Claims

1. A full-color sharpening method for obtaining a super-resolution fusion image, characterized by The following steps are involved: Step 1: Build a network model for full-color image super-resolution reconstruction using full-color iterative projection block groups as the main building blocks; Step 2: Using the network in step (1), train a super-resolution reconstruction network model with a reconstruction factor of 2; Step 3: For the input full-color image, super-resolution reconstruction is performed through the trained network, and the output is a full-color image with twice the resolution; Step 4: Using the residual block group as the main building block, build the primary fusion network module; Step 5: Using the multi-image multi-scale fusion block group as the main building block, build a multi-image fusion network module; Step 6: Construct a multi-level fusion network model using the primary fusion network module and the multi-graph fusion network module; Step 7: Use the training image dataset to train the network constructed in step (6); Step 8: Using the multispectral image, the panchromatic image, and the super-resolution reconstructed panchromatic image as network input, the final fusion result is output; the panchromatic iterative projection block group in the network model for panchromatic image super-resolution reconstruction described in step 1 mainly includes a panchromatic upsampling block and a panchromatic downsampling block; the panchromatic upsampling block is composed of an attention mechanism unit and an upsampling iterative back-projection unit; The attention mechanism unit consists of parallel max pooling and average pooling. The output is concatenated and passed through two 1×1 convolutional layers before being multiplied by the original input. The upsampling iterative back-projection unit first upsamples the input feature map through a 6×6 convolutional layer, then downsamples it through a 3×3 convolutional layer. The difference between the downsampled result and the input feature map is then upsampled through a 6×6 convolutional layer. Finally, it is added to the output of the first upsampling through the skip connection to output the final upsampled result. The full-color downsampling block consists of an attention mechanism unit and a downsampling iterative back-projection unit. The attention mechanism unit consists of parallel max pooling and average pooling. The output is concatenated and passed through two 1×1 convolutional layers before being multiplied by the original input. The downsampling iterative back-projection unit first downsamples the input feature map through a 3×3 convolution layer, then upsamples it through a 6×6 convolution layer, and the difference between the upsampled result and the input feature map is downsampled again through a 3×3 convolution layer, and finally added to the output result of the first downsampling transmitted through the jump connection to output the final downsampled result; the full-color iterative projection block group can better extract the detail features of the full-color image to obtain satisfactory 2x super-resolution reconstruction performance; the multi-image multi-scale fusion block group in the multi-image fusion network module described in step 5 mainly includes a multi-scale feature extractor and a residual attention block group; the feature extractor uses three different types of convolution kernels of 3×3, 5×5 and 7×7 to extract feature information of multiple images with different input resolutions; The output results are concatenated and passed through a 3×3 convolution layer. The feature map is then fed into a residual attention block group, which consists of four cascaded residual attention blocks. Each residual attention block includes two 3×3 convolution layers, a parallel maximum pooling and average pooling link, two 1×1 convolution layers, and several skip connections. The output of the residual attention block group is then passed through a 3×3 convolution layer and added with a long skip connection to output the final result. The multi-image and multi-scale fusion block group can effectively extract feature information of multiple input images with different resolutions and achieve satisfactory final fusion performance.

Images