Deep learning based pan-sharpening framework acceleration method

Abstract

The application discloses a deep learning-based panchromatic sharpening framework acceleration method, which comprises the following steps: splicing a multispectral image and a panchromatic image, and then cutting the spliced image into multiple sub-images as a training set; constructing an unsupervised panchromatic sharpening model based on a generative adversarial network according to the training set, wherein the unsupervised panchromatic sharpening model comprises a generator network and two discriminator networks; adding an attention mechanism between the first layer and the second layer of the generator network to accelerate the extraction of features of the multispectral image and the panchromatic image; calculating the average gradient of the sub-images after data preprocessing, performing gradient classification to divide the training set into simple and medium category sets; and performing secondary classification on the medium category set to divide it into a medium and a difficult category set; and training different sub-image sets of different categories by using panchromatic sharpening model branches of different magnitudes. Through the above scheme, the accuracy of the reconstructed image can be improved.

Description

Technical Field

[0001] This invention relates to the fields of remote sensing image processing and deep learning, specifically to a deep learning-based method for accelerating panchromatic sharpening frameworks. In particular, it is a method based on channel attention to improve the quality of image reconstruction and introduces a classification method to speed up image reconstruction, thereby improving network efficiency while ensuring image restoration accuracy. Background Technology

[0002] Remote sensing images are films or photographs that record the electromagnetic wave magnitudes of various ground features. They are mainly divided into aerial photographs and satellite photographs, and are primarily used for land water resource surveys, land resource surveys, vegetation resource surveys, geological surveys, urban remote sensing surveys, marine resource surveys, surveying and mapping, archaeological surveys, environmental monitoring, and planning management.

[0003] In the process of super-resolution reconstruction of remote sensing images, the common practice is to train the model using a pair of high- and low-resolution multispectral images. Dong et al. proposed a convolutional neural network model for super-resolution reconstruction of natural images (Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38: 295-307, 2016.). This model learns from sample libraries of low-resolution and high-resolution images to obtain the mapping relationship between low-resolution and high-resolution images, and then performs image super-resolution reconstruction. J et al. proposed a convolutional neural network model for super-resolution reconstruction of remote sensing images (Yang J, Fu X, Hu Y. A Deep Network Architecture for Pan-Sharpening[C]. IEEE International Conference on Computer Vision (ICCV), 2017.). This method generates a low-resolution multispectral image using bicubic interpolation and fuses it with a panchromatic image. The model learns from sample libraries of the fused low-resolution image and high-resolution remote sensing images to obtain the mapping relationship between them, and then performs image super-resolution reconstruction. Ledig et al. proposed a generative adversarial network model for super-resolution reconstruction of natural images (C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Photo-realistic single image super-resolution using a generative adversarial network. arXiv preprint arXiv:1609.04802, 2016.). This method has two models: a generator model generates fake target images, and a discriminator determines whether the generated images are real or fake. By continuously enhancing the adversarial relationship between the generator model and the discriminator model, a high-resolution natural image is finally obtained.Ma J et al. proposed an unsupervised pan-sharpening model (Ma J, Yu W, Chen C, et al. Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion[J]. Information Fusion, 2020, 62: 110-120.). This method is a pan-sharpening network framework that can fuse multispectral and panchromatic images into a high-resolution multispectral image. Its network structure includes a generator, a spectral discriminator, and a spatial discriminator. The spectral discriminator preserves the spectral information of the LRMS image, while the spatial discriminator preserves the spatial information of the PAN. Lu Y et al. proposed introducing a channel attention mechanism into the field of computer vision (Lu Y, Zhou Y, Jiang Z, et al. Channel Attention and Multi-level Features Fusion for Single Image Super-Resolution[J]. IEEE, 2018.). The channel attention mechanism is used to enhance the accuracy of the output image. Kong et al. proposed a classification super-resolution method (Kong X, Zhao H, Qiao Y, et al. Classesr: A general framework to accelerate super-resolution networks by data characteristic[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 12016-12025.). This method uses a classification model to classify other input sub-images after gradient calculation into different categories according to the difficulty of restoration. Different full-color sharpening models are used to train different categories of sub-images, which can improve network efficiency while ensuring image restoration accuracy. In addition, an invention patent application with application number 201810235045.5 discloses a remote sensing image target detection method based on deep learning, including: constructing a relevant dataset using remote sensing images: after classifying and labeling the remote sensing images, the image dataset and the category labels generated by the labeling work; building a full-color sharpening model based on generative adversarial networks; building a target detection model based on deep convolutional neural networks, and training the model end-to-end through methods such as backpropagation and stochastic gradient descent; and performing end-to-end testing on the constructed model.The implementation steps of the deep network-based detection algorithm of this invention are as follows: First, input the remote sensing image after pan-sharpening; input the entire image into a convolutional neural network for feature extraction; generate suggestion windows using RPN, generating 300 suggestion windows for each image; map the suggestion windows onto the last convolutional feature map of the convolutional neural network; generate a fixed-size feature map for each region of interest through a pooling layer; and jointly train the classification probability and location regression using probe classification probability and probe bounding box regression. As can be seen from the above, the pan-sharpened remote sensing image is used as input during the pan-sharpening process, resulting in a low image reconstruction speed. Furthermore, the detection network algorithm used in the reconstruction process directly performs end-to-end detection on the image using a pre-trained model, which not only reduces network efficiency but also fails to improve detection accuracy. Summary of the Invention

[0004] A brief overview of embodiments of the invention is provided below to provide a basic understanding of certain aspects of the invention. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0005] The purpose of this invention is to overcome the shortcomings of existing methods and provide a deep learning-based method for accelerating panchromatic sharpening framework training. This invention accelerates the training efficiency of the panchromatic sharpening framework through classification training. The method uses multispectral and panchromatic images as inputs to construct an unsupervised panchromatic sharpening model based on a generative adversarial network (GAN). A channel attention mechanism is introduced into the generator network to improve the accuracy of panchromatic sharpening. During reconstruction, the method utilizes gradient partitioning and classification to accelerate image reconstruction, ensuring image restoration accuracy while improving network efficiency. This method has a fast convergence speed, accelerating image reconstruction while maintaining accuracy.

[0006] To achieve the above-mentioned objectives, the technical solution adopted is as follows:

[0007] A deep learning-based method for accelerating full-color sharpening frameworks includes the following steps:

[0008] Step 1: Stitch the multispectral image and the panchromatic image together, and then crop the stitched image into multiple sub-images as the training set;

[0009] Step 2: Construct an unsupervised panchromatic sharpening model based on generative adversarial networks (GANs) using the training set. The unsupervised panchromatic sharpening model includes a generator network and a two-discriminator network. An attention mechanism is added between the first and second layers of the generator network to accelerate the extraction of features from the multispectral and panchromatic images. The two-discriminator network is implemented using existing technologies, which will not be elaborated upon in this invention.

[0010] Step 3: Calculate the average gradient of the sub-images of the training set obtained in Step 1, in order to classify multiple sub-images of the training set. Specifically, perform gradient classification to divide the training set into a set of simple categories and a set of medium categories.

[0011] Step 4: Perform secondary classification on the classified sub-images. Specifically, use a simple classification model to perform secondary classification on the intermediate category sub-images after gradient classification, dividing them into a set of intermediate category sub-images and a set of difficult category sub-images.

[0012] Step 5: Train the panchromatic sharpening model branches of different magnitudes for different categories of sub-image sets. Sub-images of the quadratic hard category set are trained using the unsupervised panchromatic sharpening model constructed in Step 2. Sub-images of the quadratic medium category set are trained using a model with half the number of channels of the unsupervised panchromatic sharpening model constructed in Step 2. Sub-images of the simple category set are reconstructed using a linear method.

[0013] Step 6: Iteratively correct the spectral and spatial information of the image output from Step 5 to obtain the final reconstructed image. The image output from Step 5 is the training set image processed by the panchromatic sharpening model; the panchromatic sharpening task corrects the spectral and spatial information of the reconstructed image.

[0014] This invention utilizes a classification training method to accelerate the training efficiency of the panchromatic sharpening framework. It uses multispectral and panchromatic images as inputs to construct an unsupervised panchromatic sharpening model based on a generative adversarial network. A channel attention mechanism is introduced into the generator network to improve the accuracy of panchromatic sharpening. During the reconstruction process, the method uses gradient partitioning and classification to accelerate the image reconstruction speed, ensuring image restoration accuracy while improving network efficiency.

[0015] Furthermore, in step 1, the multispectral image and the panchromatic image are stitched together, specifically including: obtaining the panchromatic image as a reference image. ; Obtain multispectral images paired with panchromatic images And convert it to an image of the same size as the panchromatic image. Multispectral images Upsampled to and image Same size, among Indicates 4x upsampling; stitching reference image and upsampled multispectral images .

[0016] In step 1, the stitched image is cropped into multiple sub-images as the training set, specifically including: taking the reference image... and upsampled multispectral images The stitched image is used as the input image. The input image The training set is defined as follows:

[0017]

[0018] in This indicates channel stitching. After stitching, the sub-images are cropped using a sliding window method to create multiple images. Sub-image As a training set.

[0019] Furthermore, in step 2, an unsupervised panchromatic sharpening model based on generative adversarial networks is constructed, specifically including:

[0020] Based on the training set generated in step 1, an unsupervised pan-color sharpening model based on a generative adversarial network is constructed. The unsupervised pan-color sharpening model includes a generator network. and two discriminant networks , Generator Network It is a mapping function from the input to the output. Two discriminant networks and The results from the generator network are subjected to spectral and spatial information discrimination, respectively. Both the generator network and the two discrimination networks consist of convolutional layers. This represents the parameter set in the generator and discriminative models. ,in Indicates the first The weight matrix of the layer, Indicates the first Layer bias, It represents the total number of layers in the network model.

[0021] Reference image and upsampled multispectral images The stitched image is used as the input image. The image is used as a reference image for the discriminative network to construct an unsupervised panchromatic sharpening framework based on generative adversarial networks.

[0022] Furthermore, the loss function of the generator network is defined as:

[0023]

[0024]

[0025]

[0026]

[0027] in Indicates the first Zhang Tu, It is the result generated by the generator. It is the number of samples in the training set. It is a loss of space. It is spectral loss. It is a hyperparameter that balances spatial and spectral losses. It is average pooling. Denotes the square of the F norm of a matrix. It is balance The hyperparameters of the two terms, It is a discriminant network For generator networks The confidence level of the results produced. It is balance The hyperparameters of the two terms, It is a discriminant network For generator networks The confidence level of the results.

[0028] This invention introduces a channel attention mechanism into generative networks, adding an attention mechanism between the first and second layers of the generator's neural network structure. This attention mechanism can filter out important information from a large amount of data, resulting in better extraction of multispectral images. and reference image Its characteristics.

[0029] Preferably, in step 3, the average gradient of the images in the training set is calculated to classify multiple sub-images of the training set, specifically including:

[0030] Calculate the input image The average gradient is used to classify sub-images with an average gradient less than 10 into simple category sub-images. The rest are classified as medium category sub-images. Simple category sub-images can be restored using simple bicubic interpolation upsampling, and the results are no worse than those of deep learning restoration methods, while also improving reconstruction speed.

[0031] Preferably, in step 4, the sub-images are subjected to secondary classification, specifically including:

[0032] A pre-trained panchromatic sharpening model is used to sharpen medium-class sub-images after gradient partitioning. Perform full-color sharpening, then calculate the Peak Signal-to-Noise Ratio (PSNR) of the sharpened image and the original input image. Define a PSNR threshold of 25.8; images with a PSNR higher than the threshold are classified as medium-resolution sub-images. Images below a certain threshold are classified as difficult category sub-images. Sub-images of the medium category. Sub-images of the difficulty category The labels are used to train the classification model. To make the sub-images more evenly distributed, a classification loss is added to the loss function, where the loss function of the classification model is defined as:

[0033]

[0034] Where M is the number of categories. The image is The probability of a class.

[0035] Preferably, a divide-and-conquer training strategy is used on the divided training set, specifically including:

[0036] After gradient classification and classifier classification, the training set was divided into three sub-image sets with different recovery difficulties. The sub-images of the difficult category were trained using the optimized panchromatic sharpening model (i.e. the trained panchromatic sharpening model above). The images of the medium category were less difficult to recover, and the computational cost could be reduced by decreasing the number of channels in the optimized panchromatic sharpening model. The sub-images of the easy category were relatively smooth and were reconstructed using a simple bicubic interpolation linear method.

[0037] Preferably, the reconstructed image is iteratively corrected for spectral and spatial information to obtain the final reconstructed image, specifically including:

[0038]

[0039] As can be seen from the above description of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention utilizes classification training to accelerate the training efficiency of the panchromatic sharpening framework. The method uses multispectral and panchromatic images as input to construct an unsupervised panchromatic sharpening model based on a generative adversarial network. A channel attention mechanism is introduced into the generator network to improve the accuracy of panchromatic sharpening. During reconstruction, the method uses gradient partitioning and classification to construct training sets with different recovery difficulties. Training sets with lower recovery difficulties enhance the lightweight panchromatic sharpening model to accelerate image reconstruction, ensuring image recovery accuracy while improving network efficiency. Furthermore, this accelerated method is applicable to all panchromatic sharpening tasks. Attached Figure Description

[0041] Figure 1 This is a flowchart of the full-color sharpening framework acceleration method in this embodiment;

[0042] Figure 2 This is a flowchart of the full-color sharpening framework in this embodiment;

[0043] Figure 3 This describes the stitching process of the panchromatic image and the multispectral image in this embodiment.

[0044] Figure 4 This is a schematic diagram of the channel attention mechanism module in this embodiment;

[0045] Figure 5 This is a flowchart of the reference classification method in this embodiment;

[0046] Figure 6(a) is an example of the simple category in this embodiment, Figure 6(b) is an example of the medium category, and Figure 6(c) is an example of the difficult category.

[0047] Figure 7(a) is a reference to the result of sharpening a low-resolution panchromatic image, Figure 7(b) is a reconstructed image of a high-resolution multispectral image generated by the network, Figure 7(c) is a multispectral image corresponding to the panchromatic image, and Figure 7(d) is a multispectral image reconstructed using a classification method. Detailed Implementation

[0048] Embodiments of the present invention will now be described with reference to the accompanying drawings. Elements and features described in one drawing or embodiment of the invention may be combined with elements and features shown in one or more other drawings or embodiments. It should be noted that, for clarity, representations and descriptions of components and processes unrelated to the present invention and known to those skilled in the art have been omitted from the drawings and description.

[0049] The present invention will now be further described with reference to the accompanying drawings.

[0050] This invention provides an example of a full-color sharpening process utilizing an adversarial convolutional neural network, which is a detailed description of the method proposed in this invention.

[0051] like Figure 1 and Figure 2 As shown, the deep learning-based full-color sharpening framework acceleration method in this embodiment includes the following steps:

[0052] S101, introduce panchromatic and multispectral images of different resolutions as reference images.

[0053] The dataset used in this example consists of remote sensing images from the WorldView-2 and GF-2 high-resolution satellites. The multispectral images used in WorldView-2 have a pixel size of [missing information]. The data size is The pixel size of a panchromatic image is... The data size is The multispectral image pixel size used in GF-2 is... The data size is The pixel size of a panchromatic image is... The data size is .

[0054] Among them, the panchromatic image is Multispectral images are .

[0055] S102, Initialize model input.

[0056] For multispectral images A 4x upsampling was performed to obtain an image of the same size as the panchromatic image. .

[0057] See Figure 3 As shown, splicing Image and upsampling The image is Then, crop the images using a sliding window method to create multiple images. Sub-image The reference image used to generate images for the training set.

[0058]

[0059] S103, Construct an unsupervised full-color sharpening model based on generative adversarial networks.

[0060] Specifically, the panchromatic sharpening model consists of a generator network, a spectral discriminant network, and a spatial discriminant network. The network is composed of a generator network. and two discriminant networks , A generator network is a mapping function from input to output. , This represents the parameter set in the generator and discriminative models. ,in Indicates the first The weight matrix of the layer, Indicates the first Layer bias, This refers to the total number of layers in the network model. (This is after data preprocessing.) As input, the upsampled multispectral image and panchromatic image Using the reference image as the discriminative network, an unsupervised panchromatic sharpening framework based on a generative adversarial network is constructed. The loss function is defined as:

[0061]

[0062]

[0063]

[0064] in Indicates the first Zhang Tu, It is the result generated by the generator. It is the number of samples in the training set. It is a loss of space. It is spectral loss. It is a hyperparameter that balances spatial and spectral losses. It is average pooling. Denotes the square of the F norm of a matrix. It is balance The hyperparameters of the two terms, It is a discriminant network For generator networks The confidence level of the results produced. It is balance The hyperparameters of the two terms, It is a discriminant network For generator networks The confidence level of the results.

[0065] S104 introduces a channel attention mechanism in generative networks.

[0066] See Figure 4 As shown in Table 1, an attention mechanism was added between the first and second layers of the generator's neural network structure. This attention mechanism can filter out important information from a large amount of data, leading to better image extraction. and images The characteristics of this feature are improved in all indicators after the attention mechanism is added.

[0067] S105, Gradient partitioning of the input image.

[0068] Specifically, calculate the input image The average gradient is used to classify sub-images with an average gradient less than 10 into simple category sub-images. The rest are classified as medium category sub-images. Simple category sub-images can be restored using simple bicubic interpolation upsampling, and the results are no worse than those of deep learning restoration methods, while also improving reconstruction speed.

[0069] S106, Training the classification model, secondary classification.

[0070] For details, see Figure 5 As shown, the training set is first divided into two categories, simple and other, by calculating the average gradient of the images. Then, the medium-sized sub-images are further classified. A panchromatic sharpening model is pre-trained to process the medium-sized sub-images after gradient classification. Perform full-color sharpening, then calculate the Peak Signal-to-Noise Ratio (PSNR) of the sharpened image and the original input image. Define a PSNR threshold of 25.8; images with a PSNR higher than the threshold are classified as medium-resolution sub-images. Images below a certain threshold are classified as difficult category sub-images. Sub-images of the medium category. Sub-images of the difficulty category The labels are used to train the classification model. To make the sub-images more evenly distributed, a classification loss is added to the loss function, where the loss function of the classification model is defined as:

[0071]

[0072] Where M is the number of categories. The image is The probability of a class.

[0073] S107, Classification training of a full-color sharpening model based on a conditional generative adversarial neural network.

[0074] Specifically, as shown in Figures 6(a), 6(b), and 6(c), gradient classification divides the data into a simple category set and a medium category set. A classifier then performs secondary classification on the medium category set, resulting in a secondary medium category set and a secondary difficult category set. In other words, gradient classification and classification models can yield three sets of sub-images with different recovery difficulties: a simple category set, a secondary medium category set, and a secondary difficult category set. The sub-images in the secondary difficult category set are trained using an optimized pan-color sharpening model. The sub-images in the secondary medium category set are less difficult to recover, and computational overhead is reduced by decreasing the number of channels in the optimized pan-color sharpening model. The sub-images in the simple category set are relatively smooth and are reconstructed using a simple linear method. Since the gradient classification and classification models use different classification algorithms and conditions, two classification steps are used to achieve the three levels of classification sets: simple category set, secondary medium category set, and secondary difficult category set.

[0075] Specifically, simple category images Simple texture, medium category image The textures will be more complex, with more difficult sub-images. Images with the most complex textures are those in simpler categories. The medium-class sub-image was magnified 4 times using bicubic interpolation. Inputting the image into a lightweight panchromatic sharpening network with reduced network channels results in a difficult category subimage. The input is fed into a full-color sharpening network without lightweighting. The loss function is minimized. To estimate the mapping function Medium parameters Optimal value .

[0076] S108, iteratively corrects the spectral and spatial information of the reconstructed image to obtain the final reconstructed image. Specifically, for Perform data correction This process yields the final reconstructed image. ,as follows:

[0077] .

[0078] Table 1 shows a comparison of the results before and after introducing the attention mechanism; Table 2 shows a comparison of the reconstruction results for the three categories of easy, medium, and difficult; and Table 3 shows a comparison of the results before and after introducing the classification training method.

[0079] Table 1

[0080]

[0081] Table 2

[0082]

[0083] Table 3

[0084]

[0085] As can be seen from Figures 7(a), 7(b), 7(c), and 7(d), the image accuracy reconstructed by the method of the present invention does not decrease in visual quality after the introduction of the classification method.

[0086] As can be seen from Tables 1 and 2, the restoration effect of the simple category sub-images using the bicubic interpolation method is not significantly different from the original pancolor sharpening method. The medium category sub-images using the lightweight pancolor sharpening method are not significantly different from the original pancolor sharpening method, but the performance is significantly improved. Using classification training can improve the reconstruction speed of images while ensuring the image restoration effect.

[0087] As can be seen from Table 3, the classification training method significantly improves the image reconstruction speed.

[0088] This invention divides images into three categories using gradient classification and a classification model. Different levels of panchromatic sharpening model branches are used to train sub-image sets for each category. Sub-images in the difficult category are trained using an optimized panchromatic sharpening model, while those in the medium category are trained using a model with half the number of channels of the optimized panchromatic sharpening model. Sub-images in the easy category are reconstructed using a linear method. The reconstructed images are then iteratively corrected for spectral and spatial information to obtain the final reconstructed image. The acceleration achieved through this scheme is achieved by using a lightweight panchromatic sharpening model, which not only improves the accuracy of the reconstructed image but is also applicable to all panchromatic sharpening framework scenarios, demonstrating excellent universality and practicality.

[0089] The method of this invention is not limited to being executed in the chronological order described in the specification, but can also be executed in other chronological orders, in parallel, or independently. Therefore, the execution order of the method described in this specification does not constitute a limitation on the technical scope of this invention.

[0090] Although the invention has been disclosed above through the description of specific embodiments, it should be understood that all the embodiments and examples described above are exemplary and not restrictive. Those skilled in the art can design various modifications, improvements, or equivalents to the invention within the spirit and scope of the appended claims. These modifications, improvements, or equivalents should also be considered to be included within the protection scope of the invention.

Claims

1. A deep learning based pan-sharpening framework acceleration method, characterized in that: The method comprises the following steps: Step 1: stitching the multispectral image and the panchromatic image, and then cutting the stitched image into multiple sub-images as a training set; Step 2: constructing an unsupervised panchromatic sharpening model based on a generative adversarial network according to the training set, wherein the unsupervised panchromatic sharpening model comprises a generator network and two discriminator networks; an attention mechanism is added between the first layer and the second layer of the generator network to accelerate the extraction of features of the multispectral image and the panchromatic image; Step 3: calculating the image average gradient of the training set to classify the multiple sub-images of the training set; Step 4: performing secondary classification on the classified sub-images; Step 5: training the sub-images of different categories by using panchromatic sharpening models of different magnitudes; Step 6: performing iterative spectral information and spatial information correction on the images output by step 5 to obtain a final reconstructed image; The step 4 specifically comprises: Pre-training an unsupervised panchromatic sharpening model, which uses a PNN framework as a pre-trained panchromatic sharpening model, and through the model, the medium category sub-image after gradient division Panchromatic sharpening is performed, and the peak signal-to-noise ratio of the sharpened result and the original input image is calculated, and the threshold of PSNR is defined as 25.8, and the image with a PSNR higher than the threshold is classified as a medium category sub-image , and the image below the threshold is classified as a difficult category sub-image ; the medium category sub-image and the difficult category sub-image are used as the label of the classification model to train the classification model; meanwhile, a classification loss is added to the loss function, wherein the loss function of the classification model is defined as: ； where M is the number of classes, is the probability that the image is of class k.

2. The deep learning based panchromatic sharpening framework acceleration method of claim 1, wherein: In step 1, the multispectral image and the panchromatic image are stitched, specifically including: obtaining the panchromatic image as a reference image ; obtaining the multispectral image paired with the panchromatic image , and converting into an image with the same size as the panchromatic image , up-sampling the multispectral image to the same size as the image , wherein represents 4 times up-sampling; stitching the reference image and the up-sampled multispectral image .

3. The deep learning based panchromatic sharpening framework acceleration method of claim 2, wherein: In step 1, the spliced image is cropped into multiple sub-images as a training set, specifically including: the reference image and the up-sampled multispectral image The spliced image is taken as an input image , and the input image is a training set, and the definition of the training set is: wherein represents channel concatenation, after concatenation, the sub-images are cropped by a sliding window method, and are cropped into multiple sub-images as a training set.

4. The deep learning based panchromatic sharpening framework acceleration method of claim 1, wherein: In the step 2, the unsupervised panchromatic sharpening model based on the generative adversarial network is constructed, and specifically comprises: Based on the training set generated in step 1, an unsupervised panchromatic sharpening model based on a generative adversarial network is constructed, the unsupervised panchromatic sharpening model comprising a generator network and two discriminant networks , , the generator network is a mapping function from the input end to the output end , the two discriminant networks and respectively perform spectral information discrimination and spatial information discrimination on the results of the generator network, the generator network and the two discriminant networks are all composed of convolutional layers, denote the parameter set in the generator and discriminant model, wherein denotes the weight matrix of the i-th layer, denotes the bias of the i-th layer, is the total number of network layers.

5. The deep learning based panchromatic sharpening framework acceleration method of claim 4, wherein: The loss function of the generator network is defined as: wherein represents the zhang chart, is the result generated by the generator, is the number of samples in the training set, is the spatial loss, is the spectral loss, is the hyperparameter balancing the spatial and spectral loss, is the average pooling, represents the square of the matrix F norm, is the hyperparameter balancing the two terms in it, is the confidence of the discriminator network on the result generated by the generator network . is the hyperparameter balancing the two terms in it, is the confidence of the discriminator network on the result generated by the generator network .

6. The deep learning based panchromatic sharpening framework acceleration method of claim 5, wherein: In the step 3, the image average gradient is calculated to classify the multiple sub-images of the training set, which is the average gradient of the sub-images of the training set obtained in step 1, and the gradient classification divides the training set into a simple category set and a medium category set. It specifically comprises: Computing the average gradient of the input image Sub-images with average gradient less than 10 are classified as simple class sub-images The rest are classified as medium class sub-images Where simple class sub-images are recovered by bicubic interpolation up-sampling.

7. The deep learning based panchromatic sharpening framework acceleration method of claim 1, wherein: In the step 5, the sub-images of different categories are trained by using panchromatic sharpening models of different magnitudes, specifically: the sub-images of the secondary difficult category set are trained by the unsupervised panchromatic sharpening model constructed in step 2, the sub-images of the secondary medium category set are trained by a model obtained by reducing the channel number of the unsupervised panchromatic sharpening model constructed in step 2 by half, and the sub-images of the simple category set are reconstructed by a linear method.

8. The deep learning based panchromatic sharpening framework acceleration method of claim 1, wherein: In the step 6, the reconstructed image is iteratively corrected in spectral information and spatial information to obtain a final reconstructed image, specifically comprising: 。

Images