A method for panchromatic sharpening based on U-shaped pyramid residual structure

Abstract

The application provides a panchromatic sharpening method based on a U-shaped pyramid residual structure. The method comprises the following steps: acquiring a multispectral image and a panchromatic image, and cutting the two images according to a set size to obtain a training set and a test set; training a panchromatic sharpening network model based on the U-shaped pyramid residual structure by using the training set to obtain an optimal panchromatic sharpening network model; and performing panchromatic sharpening on image data in the test set by using the optimal panchromatic sharpening network model obtained by training. The structural features comprise the following steps: progressively resampling the multispectral and panchromatic images to obtain two pyramid image sequences; merging the same scale images and extracting features; performing feature fusion on the same scale images from top to bottom; performing image reconstruction on the same scale images from bottom to top; and finally maintaining spectral information through a skip connection to obtain a high-resolution multispectral image.

Description

Technical Field

[0001] This invention relates to the field of remote sensing image fusion technology, and in particular to a panchromatic sharpening method based on a U-shaped pyramid residual structure. Background Technology

[0002] Current satellite sensors are limited by various technologies, making it impossible to directly provide high-resolution multispectral images, which greatly restricts the application of remote sensing images. Compared to optimizing physical equipment, panchromatic sharpening technology is much less expensive. Traditional panchromatic sharpening methods are divided into three types: component replacement-based methods, multi-resolution analysis-based methods, and optimization model-based methods. The component replacement-based method mainly uses transformation tools to decompose the multispectral image into spatial and spectral components, then replaces the spatial components with the panchromatic image and performs an inverse transformation to obtain the fused image. This method is simple to implement and fast, but it causes spectral distortion. The multi-resolution analysis-based method decomposes the original image into different scales and resolution sequences, then fuses images of the same scale separately according to a fusion strategy, and finally generates the final fused image through an inverse transformation. The multi-resolution analysis method has better fusion quality than the component replacement method, but it increases computational cost. The optimization model method transforms the panchromatic sharpening method into a process of building an efficient fusion model to find the optimal solution. The cost of the optimization model method is high time complexity; in practice, the model solution is not ideal, resulting in low efficiency.

[0003] With the impressive breakthroughs achieved by deep learning technology in several popular research areas of computer vision, deep learning-based panchromatic sharpening methods have also been proposed. Based on the knowledge of image super-resolution, PNN was the first to propose applying convolutional neural networks to panchromatic sharpening. The network structure consists of three convolutional layers, making it simple to implement (Reference 1: Masi, Giuseppe, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. 2016. "Pansharpening by Convolutional Neural Networks" Remote Sensing 8, no. 7: 594). Compared with traditional methods, PNN significantly improves both spatial information enhancement and spectral information preservation; however, shallow mapping operations cannot extract more image features. ResNet utilizes a residual network design with two branches. One branch trains the network in the high-frequency domain of the image, while the other branch uses skip connections to achieve spectral supplementation (Reference 2: J. Yang, X. Fu, Y. Hu, Y. Huang, X. Ding and J. Paisley, "PanNet: A Deep Network Architecture for Pan-Sharpening," 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 1753-1761). The successful implementation of residual networks has led to the application of more residual connections. ResTFnet combines residual networks with a two-stream convolutional network structure (Reference 3: Xiangyu Liu, Qingjie Liu, Yunhong Wang, Remote sensing image fusion based on two-stream fusion network, Information Fusion, Volume 55, 2020, Pages 1-15, ISSN 1566-2535). The above methods upsample the multispectral image to the size of a panchromatic image before inputting it into the network for training. This not only results in the loss of multispectral information but also incurs a large computational cost.The DUnet method progressively injects multi-scale original image feature information into the fusion process. First, it uses a gradient calculator to extract the spatial information of the panchromatic image as the input of the network. Then, it designs a multi-scale feature reuse network structure to reconstruct the image (Reference 4: D. Lei, Y. Huang, L. Zhang and W. Li, "Multibranch Feature Extraction and Feature Multiplexing Network for Pansharpening", in IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-13, 2022, Art no. 5402613, doi: 10.1109 / TGRS.2021.3074624). Summary of the Invention

[0004] In order to address some of the technical problems such as low image quality after upsampling using interpolation methods, loss of high-frequency components, gradual supplementation of spectral information in multispectral images ignoring the correlation between multispectral images and panchromatic images of the same resolution, and the inability to fully extract effective features from simple channel merging input of the original image into the network, this invention proposes a panchromatic sharpening method based on a U-shaped pyramid residual structure.

[0005] This invention provides a panchromatic sharpening method based on a U-shaped pyramid residual structure, the steps of which include:

[0006] Step 1: Acquire multispectral and panchromatic images, and crop the two images to the set size to obtain the training and test sets;

[0007] Step 2: Train the pancolor sharpening network model based on the U-shaped pyramid residual structure using the training set to obtain the optimal pancolor sharpening network model;

[0008] Step 3: Use the trained optimal panchromatic sharpening network model to perform panchromatic sharpening on the image data in the test set;

[0009] The construction process of the full-color sharpening network model based on the U-shaped pyramid residual structure includes:

[0010] A multi-scale feature extraction module is constructed, specifically including: a resampling unit, a merging unit, and a feature extraction unit. The resampling unit progressively enlarges the input multispectral image to obtain a multi-layer multispectral image pyramid; it also progressively shrinks the input panchromatic image to obtain a multi-layer panchromatic image pyramid. The merging unit performs pixel-level and channel-level merging on multispectral and panchromatic images of the same scale. The feature extraction unit extracts features from multiple images of different scales output by the merging unit.

[0011] The U-shaped pyramid residual structure is constructed, specifically including a feature fusion module and an image reconstruction module. The feature fusion module is used to perform top-down feature fusion on feature maps at different resolutions output by the feature extraction unit. The image reconstruction module is used to perform bottom-up image reconstruction on feature maps at different resolutions output by the feature fusion module. Finally, skip connections are used to sum the maximum-scale multispectral image and the reconstructed image.

[0012] Furthermore, the resampling unit is specifically used for:

[0013] The input multispectral image is sequentially transposed and convolved s times to achieve progressively t-fold image enlargement, generating an s+1-layer multispectral image pyramid; s≥2, t≥2;

[0014] The input panchromatic image is subjected to s convolution operations in sequence to achieve a progressively smaller image by t times, generating an s+1 layer panchromatic image pyramid.

[0015] Furthermore, the multi-scale feature extraction module also includes a high-pass filtering unit;

[0016] Correspondingly, the merging unit is specifically used to perform channel stitching on multispectral images and panchromatic images of the same scale according to the following formula;

[0017]

[0018] Where i represents the i-th pyramid layer, f represents the stitched image, MS represents the multispectral image, PAN represents the panchromatic image, and h(PAN) represents the high-pass domain panchromatic image output after the panchromatic image is processed by a high-pass filter unit. This indicates a pixel addition operation. This indicates a channel splicing operation.

[0019] Furthermore, the feature fusion module is specifically used to: starting from the highest resolution layer in the pyramid, perform feature fusion on the images output by the feature extraction units corresponding to any two adjacent layers in the pyramid using a given fusion process, until the lowest resolution layer in the pyramid is fused.

[0020] Specifically, the given fusion process includes: pooling downsampling the image output by the last feature extraction unit corresponding to the higher resolution layer in the pyramid to reduce the image size by a factor of s, then performing pixel fusion with the image output by the first feature extraction unit corresponding to the lower resolution layer in the pyramid, and finally performing feature extraction again on the feature map of the pixel fusion using the feature extraction unit.

[0021] Furthermore, the image reconstruction module is specifically used to: starting from the lowest resolution layer in the pyramid, perform feature fusion on the images output by the feature extraction units corresponding to any two layers in the pyramid using a given reconstruction process, until the highest resolution layer in the pyramid is fused.

[0022] Specifically, the given reconstruction process includes: performing convolutional upsampling on the image output by the last feature extraction unit corresponding to the lower resolution layer in the pyramid to enlarge the image by a factor of s, then performing pixel fusion with the image output by the first feature extraction unit corresponding to the higher resolution layer in the pyramid, and finally performing feature extraction again on the feature map of the pixel fusion using the feature extraction unit.

[0023] Furthermore, in step 2, the following formula is used as the loss function during training:

[0024]

[0025]

[0026]

[0027] loss = λ(loss cc +loss ssim )+εloss MAE ,ε=1-λ

[0028] Wherein, HRMS represents the final high-resolution multispectral image obtained after summing using jump connections, GT represents the reference multispectral image, Cov(·) represents the image covariance, D(·) represents the image variance, U(·) represents the image mean, δ(·) represents the image standard deviation, C1, C2, C3, and λ represent constants, and n represents the number of samples.

[0029] Furthermore, the feature extraction unit employs a dense residual convolutional network to extract spatial and spectral features.

[0030] The beneficial effects of this invention are:

[0031] By employing a U-shaped pyramid structure to divide the original image into proportionally proportioned image sequences, extracting fusion features from the top down and reconstructing the image from the bottom up, and designing a horizontal cascade to promote feature reuse, this invention can generate more targeted spectral and spatial information while injecting corresponding scale spectral and spatial detail features, resulting in high-resolution image quality. By dual merging of pixels and channels in the panchromatic image, compared with simple channel merging, it not only enhances spatial details but also preserves spatial structure. Attached Figure Description

[0032] Figure 1 A flowchart illustrating a pancolor sharpening method based on a U-shaped pyramid residual structure provided in an embodiment of the present invention;

[0033] Figure 2 This is one of the structural diagrams of the full-color sharpening network model based on the U-shaped pyramid residual structure provided in the embodiments of the present invention;

[0034] Figure 3 This is the second schematic diagram of the full-color sharpening network model based on the U-shaped pyramid residual structure provided in the embodiments of the present invention;

[0035] Figure 4 Provided for embodiments of the present invention: (a) is a down-resolution multispectral image, (b) is a down-resolution panchromatic image, and (c) is a simulated test fusion image of an example.

[0036] Figure 5 The following are provided for embodiments of the present invention: (a) is a full-resolution multispectral image, (b) is a full-resolution panchromatic image, and (c) is a real test fusion image of an example. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment of the invention provides a pan-color sharpening method based on a U-shaped pyramid residual structure, including the following steps:

[0040] S101: Acquire multispectral and panchromatic images, and crop the two images to a set size to obtain training and test sets;

[0041] S102: Train the full-color sharpening network model based on the U-shaped pyramid residual structure using the training set to obtain the optimal full-color sharpening network model;

[0042] S103: Use the trained optimal panchromatic sharpening network model to perform panchromatic sharpening on the image data in the test set;

[0043] Specifically, the training set images are input into the network model to obtain the optimal weights after the model is fully trained, and the test set images are input into the trained network model to generate high-resolution multispectral images, achieving the purpose of full-color sharpening.

[0044] Among them, such as Figure 2 As shown, the construction process of the full-color sharpening network model based on the U-shaped pyramid residual structure includes:

[0045] A multi-scale feature extraction module is constructed, specifically including: a resampling unit, a merging unit, and a feature extraction unit. The resampling unit progressively enlarges the input multispectral image to obtain a multi-layer multispectral image pyramid; it also progressively shrinks the input panchromatic image to obtain a multi-layer panchromatic image pyramid. The merging unit performs pixel-level and channel-level merging on multispectral and panchromatic images of the same scale. The feature extraction unit extracts features from multiple images of different scales output by the merging unit.

[0046] The U-shaped pyramid residual structure is constructed, specifically including a feature fusion module and an image reconstruction module. The feature fusion module is used to perform top-down feature fusion on feature maps at different resolutions output by the feature extraction unit. The image reconstruction module is used to perform bottom-up image reconstruction on feature maps at different resolutions output by the feature fusion module. Finally, skip connections are used to sum the maximum-scale multispectral image and the reconstructed image.

[0047] The panchromatic sharpening method based on the U-shaped pyramid residual structure provided in this invention constructs an image pyramid and performs convolution operations on the image. It autonomously learns the optimal upsampling weights so that the image is divided into multiple resolution layers and merged accordingly, thereby extracting multi-scale detail features. The pyramid-based network structure can better learn the correlation between the bands of multispectral and panchromatic images.

[0048] Example 2

[0049] Based on the above embodiment 1, as follows Figure 3 As shown, this embodiment of the invention provides another network architecture for a full-color sharpening network model based on a U-shaped pyramid residual structure;

[0050] Specifically, the resampling unit is used to: perform s transpose convolution operations on the input multispectral image to achieve a progressively larger image by t times, generating an s+1 layer multispectral image pyramid; and perform s convolution operations on the input panchromatic image to achieve a progressively smaller image by t times, generating an s+1 layer panchromatic image pyramid; s≥2, t≥2.

[0051] The multi-scale feature extraction module also includes a high-pass filtering unit; correspondingly, the merging unit is specifically used to perform channel stitching on multispectral images and panchromatic images of the same scale according to the following formula;

[0052]

[0053] Where i represents the i-th pyramid layer, f represents the stitched image, MS represents the multispectral image, PAN represents the panchromatic image, and h(PAN) represents the high-pass domain panchromatic image output after the panchromatic image is processed by a high-pass filter unit. This indicates a pixel addition operation. This indicates a channel splicing operation.

[0054] The feature fusion module is specifically used to: starting from the highest resolution layer in the pyramid, perform feature fusion on the images output by the feature extraction units corresponding to any two adjacent layers in the pyramid using a given fusion process, until the lowest resolution layer in the pyramid is fused; wherein, the given fusion process specifically includes: performing pooling downsampling on the image output by the last feature extraction unit corresponding to the higher resolution layer in the pyramid to reduce the image by a factor of s, and then performing pixel fusion with the image output by the first feature extraction unit corresponding to the lower resolution layer in the pyramid, and finally performing feature extraction again on the feature map of the pixel fusion using the feature extraction unit.

[0055] The image reconstruction module is specifically used to: starting from the lowest resolution layer in the pyramid, perform feature fusion on the images output by the feature extraction units corresponding to any two layers in the pyramid using a given reconstruction process, until the highest resolution layer in the pyramid is fused; wherein, the given reconstruction process specifically includes: performing convolutional upsampling on the image output by the last feature extraction unit corresponding to the lower resolution layer in the pyramid to enlarge the image by a factor of s, and then performing pixel fusion with the image output by the first feature extraction unit corresponding to the higher resolution layer in the pyramid, and finally performing feature extraction again on the feature map of the pixel fusion using the feature extraction unit.

[0056] Specifically, Figure 3 The network architecture shown is based on a three-layer image pyramid with s=2. Here, t=2 is used as an example.

[0057] Example 3

[0058] Based on the above embodiments, this invention provides a pan-color sharpening method based on a U-shaped pyramid residual structure, specifically including the following steps:

[0059] S301: Prepare the dataset.

[0060] Specifically, the example data comes from WorldView 2 satellite sensors. First, the data was cropped to a 64x64 pixel size to generate a dataset. The dataset was divided into a test set and a training set. The test set was further divided into a full-resolution test set (real test set) and a down-resolution test set (simulated test set). The full-resolution test set consisted of 100 64x64 pixels, and the down-resolution test set comprised 20% of the dataset. The input images for training the network were set according to the Wald protocol. After cropping, the WorldView 2 satellite training data volume was 7988.

[0061] S302: Extract detailed features from images at multiple scales.

[0062] Specifically, the multi-scale feature extraction module progressively enlarges the input multispectral image by a factor of 2 through transpose convolution, and progressively reduces the input panchromatic image by a factor of 2 through convolution, thus forming two image pyramids. The multispectral images at different scales are then superimposed with the pixels of the panchromatic image layer after high-pass filtering, and then concatenated with the panchromatic image channels. The expression for this step can be found in formula (1) above.

[0063] S303: Use feature extraction units (e.g., dense residual convolutional networks) to extract detailed features of the image (including spatial and spectral features), with each layer yielding 64 channels of spectral spatial features.

[0064] S304: Construction of the U-shaped pyramid model structure, including the top-down feature fusion process, the bottom-up image reconstruction process, and the skip connection process.

[0065] Specifically, based on Figure 3 The network structure shown, from top to bottom, includes the following steps for feature fusion: Starting with the highest resolution layer in the pyramid, the image output by the feature extraction unit corresponding to the highest resolution layer is pooled and downsampled to reduce the image size by a factor of 2. This image is then fused pixel-wise with the image output by the first feature extraction unit corresponding to the second highest resolution layer in the pyramid. Finally, the feature map of the pixel fusion is used for feature extraction again by the feature extraction unit. Then, for the second highest resolution layer in the pyramid, the image output by the last feature extraction unit corresponding to it is pooled and downsampled to reduce the image size by a factor of 2. This image is then fused pixel-wise with the image output by the first feature extraction unit corresponding to the lowest resolution layer in the pyramid. Finally, the feature map of the pixel fusion is used for feature extraction again by the feature extraction unit. This completes the feature fusion process.

[0066] based on Figure 3 The network structure shown, in its bottom-up image reconstruction process, specifically includes: starting from the lowest resolution layer in the pyramid, the image output by the last feature extraction unit corresponding to the lowest resolution layer in the pyramid is convolved and upsampled to enlarge the image by a factor of 2, and then pixel-wise fused with the image output by the first feature extraction unit corresponding to the second highest resolution layer in the pyramid. Finally, the feature map of the pixel fusion is subjected to feature extraction again by the feature extraction unit; then, for the second highest resolution layer in the pyramid, the image output by the last feature extraction unit corresponding to it is convolved and upsampled to enlarge the image by a factor of 2, and then pixel-wise fused with the image output by the first feature extraction unit corresponding to the highest resolution layer in the pyramid. Finally, the feature map of the pixel fusion is subjected to feature extraction again by the feature extraction unit.

[0067] Specifically, in the bottom-up image reconstruction process, feature reuse (in Figure 3 In this process, lateral connections accelerate the convergence speed of the network and improve the efficiency of model training.

[0068] To preserve spectral information, the model structure finally uses skip connections to fuse the maximum-scale multispectral image and the reconstructed image. In this embodiment, the skip connection expression is: In the formula, HRMS represents the final high-resolution multispectral image, F represents the reconstructed image obtained from bottom to top, and MS↑4 represents the maximum scale multispectral image, which in this embodiment is a multispectral image enlarged by four times, with the aim of achieving the same scale as the reconstructed image.

[0069] S305: After the network model is built, set the loss function and perform backpropagation. The loss function design uses SSIM and CC combined with MAE multi-task guided training of the network, continuously optimizing the parameters to produce the optimal pan-color sharpening network model.

[0070] Specifically, the loss function expression is as follows:

[0071]

[0072]

[0073]

[0074] loss = λ(loss cc +loss ssim )+εloss MAE ,ε=1-λ

[0075] Where GT represents the reference multispectral image, Cov(·) represents the image covariance, D(·) represents the image variance, U(·) represents the image mean, δ(·) represents the image standard deviation, C1, C2, C3, and λ represent constants, and n represents the number of samples.

[0076] S306: Training the Network. Using PyCharm, design a program based on the above steps, with the following parameters: batchSize set to 16; select the mainstream Adam optimizer; learning rate set to 0.0001. Input the training dataset from step S301 into the network for training. The experimental hardware environment was: CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz; GPU: Tesla P40.

[0077] S307: Test Network. The test set images are input into the trained network model to obtain the final fused image, i.e., a high-resolution multispectral image. For example... Figure 4 and Figure 5 As shown.

[0078] Table 1. Objective evaluation of fusion results of various methods for down-resolution images.

[0079] method ERGAS SAM UIQI <![CDATA[Q2 n ]]> SCC PNN 3.1527 5.3866 0.8922 0.8886 0.9406 PanNet 3.0976 5.1741 0.8992 0.8951 0.9439 SRPPNN 2.6957 4.4999 0.9135 0.9098 0.9596 DUnet 2.6585 4.4465 0.9112 0.9079 0.9612 This invention 2.6012 4.3364 0.9189 0.9158 0.9633

[0080] Table 2 Objective evaluation of fusion results of various methods for full-resolution images

[0081] Method QNR <![CDATA[D S ]]> <![CDATA[D λ ]]> PNN 0.8913 0.0686 0.0442 PanNet 0.8952 0.0777 0.0339 SRPPNN 0.8910 0.0729 0.0422 DUnet 0.8963 0.0614 0.0498 This invention 0.9050 0.0643 0.0352

[0082] PNN, PanNet, SRPPNN, and DUnet pancolor sharpening algorithms were selected as comparison algorithms for comparative experiments. Table 1 shows the results of different pancolor sharpening methods applied to the down-resolution images of the examples. Five quality assessment metrics were used to evaluate the reference images: Global Relative Spectral Loss (ERGAS), Spectral Angle Mapping (SAM), Universal Image Quality Index (UIQI), and Q2. n The index and spatial correlation coefficient (SCC), where UIQI and Q2 n The closer the SCC value is to 1, and the closer the ERGAS and SAM values ​​are to 0, the closer the image is to the original image. Table 2 shows the results of the full-resolution images of the examples obtained using different panchromatic sharpening methods, using three quantitative evaluation metrics without reference images: QNR, D... S D λ D S and D λThe closer the corresponding value is to 0, and the closer the QNR value is to 1, the better the quality of the fused image. Three example experiments demonstrate that the panchromatic sharpening method proposed in this invention is superior to other methods in both spatial detail and spectral structure, proving the effectiveness of the method.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A panchromatic sharpening method based on a U-shaped pyramid residual structure, characterized in that the steps include... include: Step 1: Acquire multispectral and panchromatic images, and crop the two images to the set size to obtain the training and test sets; Step 2: Train the pancolor sharpening network model based on the U-shaped pyramid residual structure using the training set to obtain the optimal pancolor sharpening network model; Step 3: Use the trained optimal panchromatic sharpening network model to perform panchromatic sharpening on the image data in the test set; The construction process of the full-color sharpening network model based on the U-shaped pyramid residual structure includes: A multi-scale feature extraction module is constructed, specifically including: a resampling unit, a merging unit, and a feature extraction unit. The resampling unit progressively enlarges the input multispectral image to obtain a multi-layer multispectral image pyramid; it also progressively shrinks the input panchromatic image to obtain a multi-layer panchromatic image pyramid. The merging unit performs pixel-level and channel-level merging on multispectral and panchromatic images of the same scale. The feature extraction unit extracts features from multiple images of different scales output by the merging unit. The U-shaped pyramid residual structure is constructed, specifically including a feature fusion module and an image reconstruction module. The feature fusion module is used to perform top-down feature fusion on feature maps at different resolutions output by the feature extraction unit. The image reconstruction module is used to perform bottom-up image reconstruction on feature maps at different resolutions output by the feature fusion module. Finally, skip connections are used to sum the maximum-scale multispectral image and the reconstructed image. The resampling unit is specifically used for: The input multispectral image is sequentially transposed and convolved s times to achieve progressively t-fold image enlargement, generating an s+1-layer multispectral image pyramid; s≥2, t≥2; The input panchromatic image is subjected to s convolution operations in sequence to achieve a progressively smaller image by t times, generating an s+1 layer panchromatic image pyramid; The feature fusion module is specifically used to: starting from the highest resolution layer in the pyramid, perform feature fusion on the images output by the feature extraction units corresponding to any two adjacent layers in the pyramid using a given fusion process, until the lowest resolution layer in the pyramid is fused. The given fusion process specifically includes: pooling downsampling the image output by the last feature extraction unit corresponding to the higher resolution layer in the pyramid to reduce the image by a factor of s, and then performing pixel fusion with the image output by the first feature extraction unit corresponding to the lower resolution layer in the pyramid; finally, performing feature extraction again on the feature map of the pixel fusion using the feature extraction unit. The image reconstruction module is specifically used to: starting from the lowest resolution layer in the pyramid, perform feature fusion on the images output by the feature extraction units corresponding to any two layers in the pyramid using a given reconstruction process, until the highest resolution layer in the pyramid is fused. Specifically, the given reconstruction process includes: performing convolutional upsampling on the image output by the last feature extraction unit corresponding to the lower resolution layer in the pyramid to enlarge the image by a factor of s, then performing pixel fusion with the image output by the first feature extraction unit corresponding to the higher resolution layer in the pyramid, and finally performing feature extraction again on the feature map of the pixel fusion using the feature extraction unit.

2. The panchromatic sharpening method based on a U-shaped pyramid residual structure according to claim 1, characterized in that, The multi-scale feature extraction module also includes a high-pass filtering unit; Correspondingly, the merging unit is specifically used to perform channel stitching on multispectral images and panchromatic images of the same scale according to the following formula; Where i represents the i-th pyramid layer, f represents the stitched image, MS represents the multispectral image, PAN represents the panchromatic image, and h(PAN) represents the high-pass domain panchromatic image output after the panchromatic image is processed by a high-pass filter unit. This indicates a pixel addition operation. This indicates a channel splicing operation.

3. The panchromatic sharpening method based on a U-shaped pyramid residual structure according to claim 1, characterized in that, In step 2, the following formula is used as the loss function during training: loss=λ(loss cc +loss ssim )+loss MAE ,ε=1-λ Wherein, HRMS represents the final high-resolution multispectral image obtained after summing using jump connections, GT represents the reference multispectral image, Cov(·) represents the image covariance, D(·) represents the image variance, U(·) represents the image mean, δ(·) represents the image standard deviation, C1, C2, C3, and λ represent constants, and n represents the number of samples.

4. A panchromatic sharpening method based on a U-shaped pyramid residual structure according to any one of claims 1-3, characterized in that, The feature extraction unit uses a dense residual convolutional network to extract spatial and spectral features.

Images