A method and system for panchromatic sharpening of hyperspectral images at any resolution

Abstract

The present invention discloses a method and system for panchromatic sharpening of hyperspectral images at any resolution, including reading an original hyperspectral image and synthesizing a corresponding panchromatic image, preprocessing the original hyperspectral image and the corresponding panchromatic image to obtain a multi-scale image pair, obtaining a training data set, a verification data set and a test sample; constructing an arbitrary resolution panchromatic sharpening network based on extended sub-pixel convolution; initializing network parameters, randomly selecting any scale data in the training data set for training, obtaining the minimum value when the network loss is stable, and the corresponding network is an arbitrary resolution panchromatic sharpening network of extended sub-pixel convolution trained, inputting a test sample, and obtaining a sharpened target resolution hyperspectral image. The present invention further improves the performance of arbitrary resolution panchromatic sharpening of hyperspectral images by introducing scale as prior information to adjust the importance of different spatial position information.

Description

Technical Field

[0001] The invention relates to the field of remote sensing images, and in particular to a method and system for panchromatic sharpening of hyperspectral images at any resolution. Background Art

[0002] Hyperspectral images are widely used in land cover classification, agricultural management, target detection, etc., because of their fine spectral resolution (up to 10 nm) and can reflect the inherent spectral characteristics of the observed objects. However, the actual hyperspectral imaging process is physically limited by the imaging sensor, and its spatial resolution is often poor, which limits its subsequent application scope. In order to obtain hyperspectral images with high spatial resolution, a feasible solution is to perform panchromatic sharpening of hyperspectral images, fuse single-band panchromatic images with high spatial resolution with hyperspectral images, and use the spatial details of panchromatic images to enhance the spatial quality of hyperspectral images. Traditional panchromatic sharpening methods can be divided into component replacement, multi-scale analysis, Bayesian estimation, matrix decomposition, etc. With the widespread application of deep learning, panchromatic sharpening methods based on deep convolutional neural networks have gradually become a research hotspot. The mapping relationship between input and output hyperspectral data is fitted through end-to-end learning, and this mapping relationship is estimated for a given observed image in the prediction stage to obtain the final sharpening result.

[0003] Pan-sharpening of hyperspectral images is also often used as a preprocessing step for remote sensing thematic map enhancement and change detection. However, in these applications, the desired target spatial resolution may not be exactly the spatial resolution of the panchromatic image. Most existing pan-sharpening methods only focus on improving the resolution of hyperspectral images to the spatial resolution of their corresponding panchromatic images, which is difficult to meet the needs of the above applications. In this regard, arbitrary resolution hyperspectral pan-sharpening provides a new solution to sharpen hyperspectral images to any desired spatial resolution, which can flexibly meet any possible user-customized resolution requirements in practical applications. Pan-sharpening of hyperspectral images at arbitrary resolution aims to obtain hyperspectral images at arbitrary target spatial resolution by fusion and sharpening low spatial resolution hyperspectral images and high spatial resolution panchromatic images with user-defined spatial resolution. As a new type of pan-sharpening task, arbitrary resolution pan-sharpening of hyperspectral images not only overcomes some prominent problems in general hyperspectral image sharpening, such as the significant difference in spectral coverage between hyperspectral images and panchromatic images, and the simultaneous restoration of image details on a large number of continuous bands of low spatial resolution hyperspectral images. A major challenge is how a single model can generalize the pan-sharpening process to arbitrary scales while being trained on a limited scale. Summary of the invention

[0004] In order to overcome the shortcomings and deficiencies of the prior art, the present invention provides a method and system for panchromatic sharpening of a hyperspectral image at any resolution.

[0005] This method is a method for pan-sharpening of hyperspectral images at arbitrary resolution based on extended sub-pixel convolution. First, the scale information is introduced as a priori extraction feature through the scale attention residual module, and then the extended sub-pixel convolution module is used for arbitrary scale upsampling to obtain a hyperspectral image at the target spatial resolution. After training at a limited scale, it can be generalized to pan-sharpening of hyperspectral images at arbitrary resolution by extending the spatial pattern consistency of sub-pixel convolution, realizing the generalization of a single model to arbitrary scale through limited scale training.

[0006] The present invention adopts the following technical solution:

[0007] A method for panchromatic sharpening of a hyperspectral image at any resolution, comprising:

[0008] Read the original hyperspectral image H∈R M×N×C , and synthesize the corresponding full-color image P∈R M×N , where M, N represent the height and width of the hyperspectral image, C represents the number of bands, and R represents the set of all real numbers;

[0009] The original hyperspectral image and the corresponding panchromatic image are preprocessed according to the selected training scale set S = {s 1 ,s 2 ,…,s k Generate multi-scale image pairs to obtain training data sets, verification data sets and test samples;

[0010] Construct an arbitrary resolution full color sharpening network based on extended sub-pixel convolution;

[0011] Initialize the network parameters, randomly select any scale data in the training data set for training, and obtain the minimum value when the network loss is stable. The corresponding network is the trained extended sub-pixel convolution full-color sharpening network of any resolution;

[0012] Input the test sample to obtain the sharpened target resolution hyperspectral image.

[0013] Further, read the original hyperspectral image H∈R M×N×C , and synthesize the corresponding full-color image P∈R M×N Specifically, the panchromatic image corresponding to the acquired hyperspectral image is obtained by averaging all visible light bands contained in the hyperspectral image.

[0014] Furthermore, the preprocessing step includes firstly calculating the scale set S={s 1 ,s 2 ,…,s k} and the resolution ratio r of the panchromatic image to the low-resolution hyperspectral image, calculate the intermediate scale set s′={s′ 1 ,s′ 2 ,…s′ k}, where s′ i =s i / r,i=1,2,..,k. A Gaussian filter with a specific frequency response is used to blur the original hyperspectral image H and the panchromatic image P, and s′ is performed. i Downsample by times to obtain the reference hyperspectral image for standard sharpening And full color image Where m′, n′ satisfy the expression: M=m′×s′ i , N = n′×s′ i , then H r Then downsample by r times to obtain a low-resolution hyperspectral image Where m, n satisfy the expression: M = m × r × s′ i , N = n × r × s′ i Finally, the low-resolution hyperspectral image is interpolated using the polynomial interpolation method. Pre-interpolate the scale r and get The original hyperspectral image is used as the target reference image H s Thus, we can obtain a multi-scale image pair s∈S.

[0015] Furthermore, the arbitrary resolution full color sharpening network based on extended sub-pixel convolution includes N cascaded scale attention residual modules, an arbitrary scale attention upsampling module and a light reconstruction module.

[0016] Furthermore, the scaled attention residual module is specifically:

[0017] Randomly select scale s i ∈S, input corresponding Use a single layer 3×3 convolution to obtain the initial feature map F 0 , expressed as:

[0018]

[0019] where f 3*3 (·) denotes a 3×3 convolution.

[0020] Scale Space Attention Residual Block SSARB-N, for a single SSARB, input feature map F N-1 , first two layers of 3×3 convolution are performed to obtain the intermediate feature map F′, which is specifically expressed as:

[0021] F′=f 3*3(ReLU(f 3*3 (F N-1 )))

[0022] Then it is input into the spatial attention module, and the feature map F′ is firstly subjected to maximum pooling and average pooling along the channel dimension, and then the feature map F′ is subjected to the intermediate scale s′. i The generated 2D matrices are concatenated and sent to two layers of 7×7 convolution. Finally, the Sigmoid function is used to obtain the scale space attention map M.

[0023] M=σ(f 7*7 (ReLU(f 7*7 ([MaxPool(F′);AvgPool(F′);ρ(1 / s′ i )]))))

[0024] Among them, MaxPool(·) represents the maximum pooling, AvgPool(·) represents the average pooling, ρ(1 / s′ i ) means that all generated values ​​are 1 / s′ i The output of SSARB is

[0025] F N =F N-1 +M⊙F′

[0026] Among them, ⊙ represents point-by-point multiplication. By stacking N scale-space attention residual blocks, we get the hyperspectral and panchromatic fusion feature F N .

[0027] Furthermore, the arbitrary scale attention upsampling module is specifically:

[0028] Input fusion feature F N With the intermediate scale s′ i , through a layer of 3×3 expanded sub-pixel convolution and a layer of 3×3 convolution, and then with the bicubic interpolated F N Add to get F′ N , and then through the spatial attention module, obtain the feature map F with the target resolution s The spatial attention module in this step is the same as the scale-space attention residual block.

[0029] Specifically expressed as:

[0030] F′ N =f 3*3 (ReLU(ESPC(F N )))

[0031] F s =M⊙F′ N

[0032] Where ESPC(·) represents extended sub-pixel convolution. The extended sub-pixel convolution is first performed by coordinate transformation The target resolution feature F HR Each coordinate (i, j) of is projected back to its corresponding low-resolution feature F LR , perform zero interpolation at the sub-pixel position according to the projection result Then establish s′ i The sub-pixel image at the same scale is finally convolved.

[0033] Specifically expressed as:

[0034]

[0035]

[0036] Where W SP Represents a sub-pixel convolution kernel.

[0037] Furthermore, for the spectral reconstruction, after the hyperspectral and panchromatic fusion feature extraction and arbitrary resolution upsampling, we need to reconstruct its spectral dimension back to the spectral dimension of the original image, and use two layers of 1×1 convolution to obtain the predicted target resolution hyperspectral image, which is specifically expressed as:

[0038]

[0039] A system for implementing a full color sharpening method for arbitrary resolution, comprising:

[0040] The image acquisition module is used to read the original hyperspectral image and synthesize the corresponding full-color image;

[0041] A preprocessing module is used to preprocess the original hyperspectral image and the corresponding panchromatic image to obtain a multi-scale image pair, and obtain a training data set, a verification data set, and a test sample;

[0042] Build a network module for constructing an arbitrary resolution full color sharpening network based on extended sub-pixel convolution;

[0043] The training network module is used to initialize the network parameters, randomly select any scale data in the training data set for training, and obtain the minimum value when the network loss is stable. The corresponding network is the trained extended sub-pixel convolution full-color sharpening network of any resolution;

[0044] The test module is used to input the test sample and obtain the sharpened target resolution hyperspectral image.

[0045] Beneficial effects of the present invention:

[0046] (1) The present invention divides the process of panchromatic sharpening of hyperspectral images at arbitrary resolution into three stages. In the first stage, multiple cascaded scale-space attention residual blocks are used to extract the fusion features of hyperspectral and panchromatic. In the second stage, an arbitrary scale attention upsampling module based on extended sub-pixel convolution is used to perform arbitrary resolution enhancement on the fusion features. In the last stage, spectral reconstruction is used to reconstruct the spectral dimension of the output image back to the spectral dimension of the original hyperspectral image. The model designed by the present invention can continuously predict the spatial resolution of hyperspectral images and realize panchromatic sharpening of hyperspectral images at arbitrary resolution.

[0047] (2) The extended sub-pixel convolution of the present invention can achieve upsampling of arbitrary resolutions, which is different from the traditional sub-pixel convolution and deconvolution that can only achieve upsampling of integer scales, and a single model can only process a single or finite number of integer scales. The present invention adaptively creates sub-pixel maps for different sharpening scales, and the convolutional spatial pattern generated at a finite scale remains consistent during the training and testing phases. Therefore, through finite scale training, we can generalize the single model to any scale, greatly improving the practicality of full-color sharpening of hyperspectral images.

[0048] (3) The scale-space attention residual feature extraction module and the arbitrary-scale attention upsampling module designed in the present invention both introduce scale as prior information, thereby adjusting the importance of different spatial position information of hyperspectral images at different scales, further improving the performance of hyperspectral arbitrary-resolution full-color sharpening. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a method for pan-sharpening a hyperspectral image at any resolution based on extended sub-pixel convolution according to an embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of the scale space attention residual block of the present invention;

[0051] Figure 3 This is a structural diagram of the arbitrary scale attention upsampling module of the present invention;

[0052] Figure 4(a)-Figure 4(c) This is an example diagram of the extended sub-pixel convolution one-dimensional upsampling of the present invention;

[0053] Figure 5 All convolution spatial pattern diagrams of the embodiments of the present invention within the intermediate scale of [1.0, 2.5];

[0054] Figure 6 It is a structural diagram of a neural network based on extended sub-pixel convolution of the present invention;

[0055] Figure 7(a)-Figure 7(d) Performance comparison results of different sharpening algorithms at different test scales on the PaviaC dataset. DETAILED DESCRIPTION

[0056] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0057] Example

[0058] The hyperspectral image PaviaC used in this embodiment comes from the ROSIS satellite sensor. This image originally contains 115 bands. After removing 13 noise and water absorption bands, 102 bands remain. The ratio of the spatial resolution of the panchromatic image and the hyperspectral image is r=5. The size of the panchromatic image is 1000*700, and the size of the hyperspectral image is 200*140*102.

[0059] like Figure 1 , Figure 2 and Figure 3 As shown, a method for panchromatic sharpening of hyperspectral images at arbitrary resolution based on extended sub-pixel convolution includes the following steps:

[0060] S1. Read the original hyperspectral image H∈R 1000×700×102 , synthesize the corresponding full-color image P∈R 1000×700 , where R represents the set of all real numbers;

[0061] S2. Preprocess the hyperspectral image and its corresponding panchromatic image, generate multi-scale image pairs according to the selected scale set S=5*{1.0,1.1,…,2.5}, divide the image pairs into regions to generate training data sets and validation data sets, and generate test samples according to the user-defined scale {1.0,1.5,1.7,2.5,3.2,3.6};

[0062] S3. Design an arbitrary resolution full color sharpening network based on extended sub-pixel convolution, including a scaled attention residual module, an extended sub-pixel convolution module, and a spectral reconstruction module;

[0063] S4, initialize the network parameters, randomly select any scale data of the training set for training, update the parameters through the adaptive moment estimation optimization algorithm, and obtain the optimal parameters when the network loss is stable at the minimum value;

[0064] S5, input the test sample generated in S2 into the trained extended sub-pixel convolutional network to obtain a sharpened target resolution hyperspectral image;

[0065] Specifically, in S1, the panchromatic image corresponding to the acquired hyperspectral image is obtained by averaging the first 83 bands contained in the hyperspectral image;

[0066] Specifically, S2 includes:

[0067] S2.1, Preprocessing: First, according to the selected scale set S = 5*{1.0, 1.1, ..., 2.5} and the resolution ratio of the panchromatic image to the low-resolution hyperspectral image r = 5, the intermediate scale set S′ = {1.0, 1.1, ..., 2.5} of the target resolution hyperspectral image and the panchromatic image is calculated. A Gaussian filter with a specific frequency response is used to blur the original hyperspectral image H and the panchromatic image P, and s′ is degraded. i The downsampling is times, and the reference hyperspectral image H is obtained when the standard sharpening is performed. r ∈R m′×n′×C And the full-color image P r ∈R m′×n′ , where m′, n′ satisfy the expression: M=m′×s′ i , N = n′×s′ i , then H r Then downsample by r times to obtain a low-resolution hyperspectral image Where m, n satisfy the expression: M = m × r × s′ i , N = n × r × s′ i Finally, the low-resolution hyperspectral image is interpolated using the polynomial interpolation method. Perform pre-interpolation with scale r = 5, and we get The original hyperspectral image is used as the target reference image H s Thus, we can obtain a multi-scale image pair s∈S.

[0068] S2.2 Image Region Division: From Multi-Scale Image Pairs s∈S is divided into training area size of 1000×200, validation area size of 300×400, and test area size of 200×200. Test samples are generated according to the user-defined scale {1.0, 1.5, 1.7, 2.5, 3.2, 3.6}.

[0069] Furthermore, if Figure 6 As shown, the convolutional network based on extended sub-pixel convolution in S3 mainly includes a scale attention residual module, an arbitrary scale upsampling module based on extended sub-pixel convolution, and a spectral reconstruction module:

[0070] The scale attention residual module: for the input feature map F N-1 , two layers of 3×3 convolution are used to obtain the intermediate feature map F′, which is specifically expressed as:

[0071] F′=f 3*3 (ReLU(f 3*3 (F N-1 )))

[0072] Then it is input into the spatial attention module, and the feature map F′ is subjected to maximum pooling and average pooling along the channel dimension, and the intermediate scale s′ is used. i The generated 2D matrices are concatenated and fed into two layers of 7×7 convolutions. Finally, the Sigmoid function is used to obtain the scale space attention map M:

[0073] M=σ(f 7*7 (ReLU(f 7*7 ([MaxPool(F′);AvgPool(F′);ρ(1 / s′ i )]))))

[0074] Among them, MaxPool(·) represents the maximum pooling, AvgPool(·) represents the average pooling, ρ(1 / s′ i ) means that all generated values ​​are 1 / s′ i A two-dimensional matrix, σ(·) represents the sigmoid function. The output of this module is:

[0075] F N =F N-1 +M⊙F′

[0076] Among them, ⊙ represents point-by-point multiplication. By stacking 6 scale-space attention residual blocks, we get the hyperspectral and panchromatic fusion feature F N .

[0077] The arbitrary scale attention upsampling module: input fusion feature F N With the intermediate scale s′ i , through a layer of 3×3 expanded sub-pixel convolution and a layer of ordinary 3×3 convolution, and then with the bicubic interpolated F N Add to get F′ N , and then pass the same scale space attention module as in step 3.2 to obtain the feature map F with the target resolution s , specifically expressed as:

[0078] F′ N =f 3*3 (ReLU(ESPC(F N )))

[0079] F s =M⊙F′ N

[0080] Where ESPC(·) represents extended sub-pixel convolution. The extended sub-pixel convolution is first performed by coordinate transformation The target resolution feature F HR Each coordinate (i, j) of is projected back to its corresponding low-resolution feature F LR, perform zero interpolation at the sub-pixel position according to the projection result like Figure 4(a)-Figure 4(c) Shown are three one-dimensional upsampling examples, and then s′ is established i The sub-pixel image at the same scale is finally convolved.

[0081] Specifically expressed as:

[0082]

[0083]

[0084] Where W SP Denotes the sub-pixel convolution kernel. The spatial patterns produced by convolution on sub-pixel images of different scales are as follows: Figure 5 shown. Figure 5 The spatial patterns in already contain possible spatial patterns of sub-pixel images of all scales, so it can be generalized during the testing phase.

[0085] Spectral reconstruction module: After the hyperspectral and panchromatic fusion feature extraction and arbitrary resolution upsampling, we need to reconstruct its spectral dimension back to the spectral dimension of the original image. Two layers of 1×1 convolution are used to obtain the predicted target resolution hyperspectral image, which is specifically expressed as:

[0086]

[0087] In addition, we use four commonly used sharpening image evaluation indicators to evaluate the sharpening results, namely, cross correlation coefficient (CC), spectral angle mapping (Spectral Angle Mapper), comprehensive global relative error (ERGAS), and multi-band general image quality index Q2n.

[0088] The sharpening algorithms used for comparison are: Gram-Schmidt Adaptive (GSA), Smoothing Filter-based Intensity Modulation (SFIM), Sparse Bayes (BayesSparse), PNN, HyperPNN. Since these algorithms are designed specifically for standard sharpening, we add bicubic interpolation as the last layer to achieve arbitrary resolution full color sharpening for comparison.

[0089] Figure 7(a)-Figure 7(d)The performance comparison results of different sharpening algorithms at different test scales on the PaviaC dataset are shown in the figure. It can be seen from the figure that among the traditional algorithms, the sharpening results obtained by the sparse Bayesian algorithm are relatively good, but the distortion of its spectral angle mapping is greater than that of the other two traditional algorithms as the test scale increases; while the results obtained by the algorithm proposed in this embodiment have achieved the best results in all indicators, especially in the spectral angle mapping indicator, which is obviously better than the compared advanced algorithms, indicating that the designed method can achieve good spectral fidelity in the full-color sharpening task of arbitrary resolution.

[0090] In summary, the invention provides a method for pan-sharpening of hyperspectral images at arbitrary resolution based on extended sub-pixel convolution. The designed extended sub-pixel convolution can achieve learnable upsampling of arbitrary resolution, and by extending the consistency of the spatial pattern of sub-pixel convolution, the problem and challenge of generalizing a single model to arbitrary scale under limited scale training is solved. In addition, by introducing scale as prior information to adjust the importance of different spatial position information, the performance of pan-sharpening of hyperspectral images at arbitrary resolution is further improved.

[0091] Another embodiment of the present invention provides a system for panchromatic sharpening of a hyperspectral image at any resolution, comprising:

[0092] The image acquisition module is used to read the original hyperspectral image and synthesize the corresponding full-color image;

[0093] A preprocessing module is used to preprocess the original hyperspectral image and the corresponding panchromatic image to obtain a multi-scale image pair, and obtain a training data set, a verification data set, and a test sample;

[0094] Build a network module for constructing an arbitrary resolution full color sharpening network based on extended sub-pixel convolution;

[0095] The training network module is used to initialize the network parameters, randomly select any scale data in the training data set for training, and obtain the minimum value when the network loss is stable. The corresponding network is the trained extended sub-pixel convolution full-color sharpening network of any resolution;

[0096] The test module is used to input the test sample and obtain the sharpened target resolution hyperspectral image.

[0097] When the system is executed, the arbitrary resolution panchromatic sharpening method of the hyperspectral image can be realized.

[0098] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A method for panchromatic sharpening of a hyperspectral image at any resolution, characterized in that: include: Read the original hyperspectral image H∈R M×N×C , and synthesize the corresponding full-color image P∈R M×N , where M, N represent the height and width of the hyperspectral image, C represents the number of bands, and R represents the set of all real numbers; The original hyperspectral image and the corresponding panchromatic image are preprocessed according to the selected training scale set S = {s1, s2, …, s k Generate multi-scale image pairs to obtain training data sets, verification data sets and test samples; Construct an arbitrary resolution full color sharpening network based on extended sub-pixel convolution; Initialize the network parameters, randomly select any scale data in the training data set for training, and obtain the minimum value when the network loss is stable. The corresponding network is the trained extended sub-pixel convolution full-color sharpening network of any resolution; Input the test sample to obtain the sharpened target resolution hyperspectral image; The arbitrary resolution full color sharpening network based on extended sub-pixel convolution includes N cascaded scale attention residual modules, arbitrary scale attention upsampling modules and spectral reconstruction modules; The scale attention residual module is specifically: Input feature map F N-1 , two layers of 3×3 convolution are used to obtain the intermediate feature map F′; The intermediate feature map F′ is input into the spatial attention module, and the maximum pooling and average pooling are performed along the channel dimension, and then combined with the intermediate scale s i 'The generated 2D matrix concatenation; The concatenated result is input into two layers of 7×7 convolution, and finally the Sigmoid function is used to obtain the scale space attention map; By stacking Q scale attention residual modules, we can obtain the hyperspectral and panchromatic fusion feature F N ; The arbitrary scale attention upsampling module is specifically: Input fusion feature F N With intermediate scale s i ′, through a layer of 3×3 expanded sub-pixel convolution and a layer of 3×3 convolution, and then with the bicubic interpolated F N Add to get F N ′N, and then through the spatial attention module, obtain the feature map F with the target resolution s ; The spectral reconstruction module specifically uses two layers of 1×1 convolution to obtain a high-resolution high-spectral image of the predicted target.

2. The method for pan-sharpening at any resolution according to claim 1, characterized in that: The panchromatic image is obtained by averaging all visible light bands contained in the corresponding hyperspectral image.

3. The method for pan-sharpening at any resolution according to claim 1, characterized in that: The pre-processing comprises: First, according to the selected scale set S = {s1, s2, ..., s k } and the resolution ratio r of the panchromatic image to the low-resolution hyperspectral image, calculate the intermediate scale set S′={s′1,s′2,…s′ k }, where s′ i =s i / r,i=1,2,..,k; A Gaussian filter with a specific frequency response is used to blur and degrade the original hyperspectral image H and panchromatic image P. i ′ times downsampling to obtain the reference hyperspectral image H for standard sharpening r ∈R m′×n′×C And the full-color image P r ∈R m ′×n′ , where m′, n′ satisfy the expression: M=m′×s i ′,N=nχ×s i χ; Then H r Then downsample by r times to obtain a low-resolution hyperspectral image Where m, n satisfy the expression: M = m × r × s i χ, N = n × r × s i ′, and finally the low-resolution hyperspectral image is transformed into Pre-interpolate the scale r and get The original hyperspectral image is used as the target reference image H s , thus obtaining a multi-scale image pair 4. The method for panchromatic sharpening at any resolution according to claim 3, characterized in that: The obtaining of the training data set, the verification data set and the test sample is specifically, The training data set, validation data set and test samples are obtained by regional division.

5. The method for pan-sharpening at any resolution according to claim 1, characterized in that: The extended sub-pixel convolution first projects each coordinate of the target resolution feature to the corresponding low-resolution feature through coordinate transformation; Then, according to the projection result, 0 is interpolated at the sub-pixel position to establish S i ′ scale, and finally convolve the sub-pixel image.

6. A system for implementing the arbitrary resolution full color sharpening method according to any one of claims 1 to 5, characterized in that: include: The image acquisition module is used to read the original hyperspectral image and synthesize the corresponding full-color image; A preprocessing module is used to preprocess the original hyperspectral image and the corresponding panchromatic image to obtain a multi-scale image pair, and obtain a training data set, a verification data set, and a test sample; Build a network module for constructing an arbitrary resolution full color sharpening network based on extended sub-pixel convolution; The training network module is used to initialize the network parameters, randomly select any scale data in the training data set for training, and obtain the minimum value when the network loss is stable. The corresponding network is the trained extended sub-pixel convolution full-color sharpening network of any resolution; The test module is used to input the test sample and obtain the sharpened target resolution hyperspectral image.

Images