Integrated Fusion Method for Spaceborne Panchromatic-Multispectral-Hyperspectral Remote Sensing Images

Abstract

The present invention relates to an integrated fusion method for spaceborne panchromatic - multispectral - hyperspectral remote sensing images. By leveraging the characteristics of multispectral images, it realizes the purification of spatial high-frequency detail information and discriminative spectral information contained in multi-source input images, thus converging into two pure feature components, namely spatial components and spectral components, for subsequent reconstruction. In spectral feature reconstruction, taking the aggregated features as the main branch, while injecting details using multi-scale encoded features, interactive reasoning is performed on spectral features through channel expansion and splitting to obtain up and down correlation clues, and finally an ideal image is generated to meet the multi-source remote sensing spatio-spectral high-fidelity fusion requirements under large spatial resolution differences and spectral differences.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing image processing, and particularly relates to an integrated fusion method for spaceborne panchromatic-multispectral-hyperspectral remote sensing images. Background Art

[0002] With the maturity of satellite technology, hyperspectral imaging platforms have developed from the initial ground-based and airborne platforms to spaceborne platforms, greatly promoting the application of hyperspectral images in fields such as national defense, agriculture, forestry, and environmental monitoring. However, due to the performance limitations of optical remote sensing imaging systems, there is an inevitable trade-off between the spatial resolution and spectral resolution of the imaging system. The purpose of spatial-spectral fusion is to alleviate the problem of mutual restriction between the high spatial resolution and high spectral resolution of the imaging system. By aggregating the spatial and spectral complementary advantages of multi-source remote sensing images in the same area, a fused image with both high spatial resolution and high spectral resolution can be obtained, which can effectively utilize the advantages of multi-source complementary observation images, break through the performance constraints of a single sensor, and greatly improve the usability of hyperspectral images.

[0003] Currently, a variety of integrated fusion methods for panchromatic, multispectral, and hyperspectral images have been proposed, mainly including traditional methods and deep learning-based methods. Traditional methods usually introduce the intermediate-level spatial and spectral information of multispectral images as a bridge between panchromatic and hyperspectral images to alleviate the challenges brought by spatial resolution and spectral differences. However, traditional methods only regard multispectral images as a spatial-spectral information aggregate and do not fully explore and utilize their spatial characteristics and discriminative spectral information in the integrated fusion process. In the deep learning framework, by introducing auxiliary images, the problem of improving the resolution of hyperspectral images can be solved more effectively, thereby better mining the inherent characteristics of panchromatic, multispectral, and hyperspectral images and further improving the fusion effect. However, although some progress has been made in the field of integrated fusion with deep learning-based methods, there are still many challenges. First, as the resolution differences between images, such as the differences in spatial and spectral resolutions, increase, it becomes more challenging to accurately fuse complementary information. Therefore, how to recover fine spectral and clear spatial information from two or more input images and accurately fuse them remains a key challenge. Second, existing deep learning methods can be roughly divided into two categories: one is to extract features after simply splicing the inputs, and the other is to extract features through parallel networks and then fuse them. However, the former ignores the distribution gap between the spatial and spectral dimensions, while the latter is difficult to capture the structural consistency of spatial and spectral information.

[0004] Therefore, there is an urgent need for a new integrated fusion method for multi-source images to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an integrated fusion method for satellite-borne panchromatic-multispectral-hyperspectral remote sensing images.

[0006] This integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images includes the following steps:

[0007] Step 1: Obtain panchromatic images, multispectral images and hyperspectral images, and preprocess them as original multi-source input images;

[0008] Step 2: align the original multi-source input images and perform data enhancement processing;

[0009] Step 3: According to the characteristics of panchromatic and hyperspectral images, the spatial and spectral information of the multispectral image in the fused input image are compared for pixel-by-pixel feature similarity, so as to obtain the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs;

[0010] Step 4: Reconstruct the shared parameters of the consistency component output to learn the spatial-spectral intrinsic feature correlation, perform cross-scale and level-by-level fusion, and thus produce a high-resolution hyperspectral fused image.

[0011] Preferably, in step three, according to the characteristics of the original multi-source input image, and for the spatial and spectral characteristics, 3×3 convolution and 1×1 convolution are used to extract multi-source spatial-spectral features; spatial and spectral migration branches are constructed respectively, and combined with multi-source spatial-spectral features, the high-frequency detail information of the panchromatic image and the discriminative spectral information of the hyperspectral image are migrated to the multispectral image, and the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs are obtained.

[0012] Preferably, in step 4, the spatial-spectral aggregation feature obtained in step 3 is encoded in a parameter sharing manner, and then decoded to obtain decoded features S1, S2 and S3, and the scales of S1, S2 and S3 are increased in sequence, and feature S i Upsample to feature S i+1 Consistent, then input the cross-scale attention module to obtain S i and S i+1 The cross-layer multi-scale attention weights w between i , 1≤i≤2, and then obtain the multi-scale spatial detail features T1, T2 and T3 weighted by the attention weights.

[0013] As a preferred method, in step 4, the information interaction and refinement between channel features are performed through the spectral prediction module. First, a 1×1 convolution is used to expand the aggregate feature F mer The number of channels is divided into G groups along its channel dimension, namely g k ,1≤k≤G, each set of features is divided into and wherein, each group of features g k in is used for cross-channel information exchange with the next group of features and is used for intra-group channel interaction learning. and is used for intra-group channel interaction learning.

[0014] Preferably, in step four, the multi-scale spatial detail features T1, T2, and T3 are injected into the spectral prediction module in stages in the main branch for spatial-spectral joint reconstruction to obtain a high-resolution hyperspectral fused image.

[0015] Preferably, during the supervised training in step four, the L1 norm loss function is used for optimization. The L1 norm loss function is used to calculate Loss, and Loss is the average sum of the differences between the label H ref and the fusion result H result for N input images in each training iteration.

[0016] The beneficial effects of the present invention are as follows:

[0017] 1) By leveraging the characteristics of multi-spectral images, the present invention realizes the purification of the spatial high-frequency detail information and discriminative spectral information contained in multi-source input images, thus converging into two pure feature components, namely the spatial component and the spectral component, for subsequent reconstruction; in the spectral feature reconstruction, using the aggregated features as the main branch, while injecting details with multi-scale encoded features, the spectral features are interactively inferred through channel expansion and splitting to obtain up and down related clues, and finally an ideal image is generated.

[0018] 2) The proposed panchromatic and hyperspectral image joint spatial-spectral information reconstruction framework based on multi-spectral images and the novel hyperspectral image spatial-spectral feature migration and fusion network in the present invention can overcome the technical bottleneck of the existing methods in the multi-source remote sensing spatial-spectral high-fidelity fusion technology with large spatial spectral resolution differences, meet the multi-source remote sensing spatial-spectral high-fidelity fusion requirements under large spatial resolution differences and spectral differences, and produce high-resolution hyperspectral image data with clear space and faithful spectra. Description of the Drawings

[0019] Figure 1 is the flow chart of the integrated fusion method for panchromatic / multi-spectral / hyperspectral images;

[0020] Figure 2 is the schematic diagram of the image preprocessing process;

[0021] Figure 3 is the schematic diagram of the overall structure of the integrated fusion method for panchromatic / multi-spectral / hyperspectral images;

[0022] Figure 4 It is a schematic structural diagram of the self-guiding module;

[0023] Figure 5 It is a schematic structural diagram of the space-spectral feature migration module;

[0024] Figure 6 It is a schematic structural diagram of the cross-scale attention module;

[0025] Figure 7 It is a schematic structural diagram of the spectral prediction module;

[0026] Figure 8 It is a schematic structural diagram of the channel interaction unit. Specific implementation manners

[0027] The following further describes the present invention in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several modifications can still be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0028] Embodiment 1

[0029] As an embodiment, for the integrated fusion method of spaceborne panchromatic-multispectral-hyperspectral remote sensing images, first, preprocess and perform data enhancement operations on the panchromatic, multispectral, and hyperspectral images. Secondly, after extracting features for space and spectrum, according to the feature characteristics, and using the space-spectral feature migration network to transfer the spatial information of the panchromatic image and the spectral information of the hyperspectral image to the multispectral image respectively, to obtain the preliminary spatial and spectral components. Then, with the help of the multispectral features screened by self-guidance and the preliminarily extracted features, perform feature aggregation to obtain the final spatial, spectral, and space-spectral feature components. Finally, perform shared parameter reconstruction on the output of the consistency component to learn the internal feature correlation of space-spectral, and at the same time perform cross-scale and hierarchical fusion, so as to generate high-resolution hyperspectral images.

[0030] Involve the following modules:

[0031] An acquisition module, configured to acquire the panchromatic image, the multispectral image, and the hyperspectral image, and preprocess the panchromatic image, the multispectral image, and the hyperspectral image;

[0032] A registration module, configured to accurately register the panchromatic image, the multispectral image, and the hyperspectral image, and then perform data enhancement processing on each group of images, and ensure that the images to be fused have the same ground object scene information and projection system;

[0033] A feature extraction module, which is used to compare the spatial and spectral information of a multispectral image pixel by pixel according to the characteristics of panchromatic and hyperspectral images, so as to obtain the output of spatial consistency and spectral consistency components;

[0034] An information reconstruction module, which is used to reconstruct the shared parameters of the consistency component output to learn the internal feature correlation of space-spectral, and perform cross-scale and hierarchical fusion, so as to generate a high-resolution hyperspectral fusion image.

[0035] As Figure 1 shown, this integrated fusion method for spaceborne panchromatic-multispectral-hyperspectral remote sensing images includes the following steps:

[0036] Step 1: Obtain panchromatic, multispectral and hyperspectral images, and preprocess the panchromatic, multispectral and hyperspectral images.

[0037] The multi-sensor data includes panchromatic, multispectral and hyperspectral images. As Figure 2 shown, the present application first preprocesses the multi-sensor data. The preprocessing before fusing the panchromatic, multispectral and hyperspectral images mainly includes operations such as radiometric calibration, atmospheric correction, and orthorectification. In addition, in order to avoid the influence of distorted bands of the hyperspectral image on the accuracy of the fusion image, the hyperspectral image also needs to remove bad bands.

[0038] Step 2: Accurately register the panchromatic, multispectral and hyperspectral images, and then perform data augmentation processing on each group of images, and ensure that the images to be fused have the same ground object scene information and projection system.

[0039] In step 2, according to the original multi-source input images obtained in step 1, due to the limitation of experimental hardware, a group of original multi-source input images cannot be directly input into the network for training. And when training a deep learning network, the data demand is large. To avoid overfitting during network training and improve the robustness and generalization ability of the model, in this embodiment, data augmentation processing is adopted, that is, a group of original multi-source input images are cut into small images and then flipped, rotated, cropped, etc. are performed on each group of images.

[0040] Step 3: Compare the spatial and spectral information of the multispectral image pixel by pixel according to the characteristics of the panchromatic and hyperspectral images, so as to obtain the output of spatial consistency and spectral consistency components, as Figure 3 、 Figure 4 and Figure 5 shown. Figure 3 and Figure 6 In, C represents the feature concatenation step, S represents the sigmoid activation function, ↑ represents the upsampling operation, SPU represents the spectral prediction unit, and CAM represents the cross-scale attention module.

[0041] AsFigure 3 As shown on the left, a spatial-spectral feature migration module is established, including feature extraction of different input images, spatial / spectral feature migration, and feature self-guidance enhancement of the multispectral branch. It should be noted that the spatial and spectral resolutions of the multispectral image itself are at an intermediate level compared to the panchromatic and hyperspectral images, and it can serve as a bridge to solve problems such as spatial blurring and spectral distortion caused by the huge differences in the spatial and spectral resolution ratios in hyperspectral pansharpening. Through step three, by leveraging the characteristics of the multispectral image, it is possible to purify the spatial high-frequency detail information and discriminative spectral information contained in the multi-source input images, thus converging into two pure feature components, namely the spatial component and the spectral component, for subsequent reconstruction.

[0042] Step four: Perform shared-parameter reconstruction on the consistency component output to learn the internal feature correlation of spatial-spectral, and perform cross-scale and hierarchical fusion to generate a high-resolution hyperspectral image, as Figure 3 , Figure 6 , Figure 7 and Figure 8 shown. Conv represents convolution, Relu represents rectified linear, AvgPool represents average pooling, MaxPool represents max pooling, C in the diamond box represents channel concatenation-convolution-group operation, and CIBlock represents channel interaction unit.

[0043] As Figure 3 shown on the right, a spatial-spectral joint reconstruction module is established, including the generation of cross-scale attention feature maps and spectral prediction reconstruction. It should be noted that multi-scale features are particularly important for image fusion, and simple convolution operations alone cannot extract multi-scale features. Through the progressive fusion in step four, first, the spatial-spectral features are decoded in a multi-scale and shared-parameter manner, where the scale consistency is enforced between different decoder scales through the cross-scale attention module. In spectral feature reconstruction, using the aggregated features as the main branch, while injecting details using the multi-scale encoded features, the spectral features are interactively inferred through channel expansion and splitting to obtain up-and-down related clues, and finally an ideal high-resolution hyperspectral fusion image is generated.

[0044] Step five: During the supervised training process, the L1 loss function is used for optimization to make the fusion image generated in step 4 closer to the reference image, where the reference image is the original high-resolution hyperspectral image, and the expression is:

[0045]

[0046] where Loss represents the average sum of the differences between the label H ref and the fusion result H result calculated using the L1 norm for N input images in each training iteration.

[0047] Since it is difficult to obtain real high-spatial hyperspectral images in a real imaging environment, based on the Wald protocol, the training images are simulated for spatial and spectral degradation to obtain multiple pairs of training data. Furthermore, in the process of supervised training in this application, the L1 norm loss function is used for optimization to obtain hyperspectral images with high spatial resolution.

[0048] Embodiment 2

[0049] As another embodiment, this Embodiment 2 is proposed based on Embodiment 1. A more specific integrated fusion method for spaceborne panchromatic-multispectral-hyperspectral remote sensing images. Step 3 is specifically as follows:

[0050] Step 3.1, according to the characteristics of the original multi-source input images, and aiming at spatial and spectral characteristics, use 3×3 convolution and 1×1 convolution to extract multi-source spatial-spectral characteristics, expressed as:

[0051] F P =Conv3(σ(Conv3(σ(Conv3(P)))))

[0052] F h =Conv1(σ(Conv1(σ(Conv1(H)))))

[0053]

[0054] Among them, P represents the input panchromatic image, H represents the input hyperspectral image, M represents the input multispectral image, σ(·) represents the Relu activation function, Conv3 and Conv1 respectively represent convolution kernels of 3×3 and 1×1, F P represents the spatial features extracted in the panchromatic branch, F h represents the spectral features extracted in the hyperspectral branch, and respectively represent the spatial and spectral features obtained after corresponding convolution in the multispectral branch.

[0055] For multi-source input images, the panchromatic image contains fine spatial information, the hyperspectral image has rich spectral information, and the multispectral image contains both spatial and spectral information. Therefore, the panchromatic branch uses 3×3 convolution to extract spatial information, the hyperspectral branch uses 1×1 convolution that pays more attention to channel information to extract information, and the multispectral branch uses both 3×3 and 1×1 convolution to extract information.

[0056] Step 3.2: Using the two core information, spatial and spectral, extracted from the three input images in Step 3.1, and adopting the idea of feature transfer, transfer the high-frequency detail information of the panchromatic image and the discriminative spectral information of the hyperspectral image to the multispectral image, which can greatly improve its spatial detail information and spectral discriminative features, such as Figure 5 shown

[0057] The expression of Step 3.2 is:

[0058]

[0059] M spa = max(M spa,((u,v),(x,y)) )

[0060]

[0061] M spe = max(M spe,(a,b) )

[0062]

[0063] where Cos(·) represents the cosine similarity distance, Norm(·) represents the operation of normalizing the calculation result to the range of (0, 1), max(·) represents the maximum value function, represents element-wise multiplication, represents element-wise addition, is the vector representation after unfolding along the channel dimension at the (u, v) position, F p,(x,y) is a representation similar to , M spa represents the spatial weight map, f spa represents the spatial component output of the panchromatic image with enhanced spatial features, represents the spectral feature of the multispectral image branch at the a-th column, F h,b represents the spectral feature of the hyperspectral image branch at the b-th column, M spe represents the spectral weight map, f spe represents the output after spectral transfer of the hyperspectral image.

[0064] The multispectral branch will extract spatial and spectral information, where the spatial information will perform a feature transfer operation with the high-frequency detail information of the panchromatic image, and the spectral information will perform a feature transfer operation with the discriminative spectral information of the hyperspectral image.

[0065] Step 3.3: By adopting a self-guiding unit to screen the important information of the multispectral image branch, at the same time, the important features in the previous steps are reused to avoid the disappearance of important features during the network training process, such as Figure 3 andFigure 4 as shown

[0066] The expression in Step 3.3 is:

[0067]

[0068] F spa = Conv(Cat(F p , f spa , F))

[0069] F spe = Conv(Cat(F h , f spe , F))

[0070] F mer = Conv(Cat(F spa , F, F spe ))

[0071] where Sig(·) represents the Sigmoid activation function, Cov(·) represents the convolution operation, Cat(·) represents the operation of stacking input features in the channel dimension, F represents the output of the multi-spectral image branch passing through the self-guiding unit, F spa represents the final spatial component, F spe represents the final spectral component, F mer represents the spatial-spectral aggregation feature.

[0072] The feature F output by the self-guiding module, the shallow features F p and F h obtained in Step 3.1, and the features f spa and f spa obtained by feature migration in Step 3.2 are used for feature aggregation to obtain the final spatial component, spectral component, and spatial-spectral aggregation feature.

[0073] It should be noted that the parts that are the same as or similar to those in Embodiment 1 in this embodiment can be referred to each other and will not be elaborated in this application.

[0074] Embodiment 3

[0075] As another embodiment, this Embodiment 3 is proposed based on Embodiments 1 and 2. A more specific integrated fusion method for spaceborne panchromatic-multi-spectral-hyperspectral remote sensing images, and Step 4 is specifically as follows:

[0076] Step 4.1: First, encode the spatial-spectral features obtained in Step 3.3 in a parameter-sharing manner, and then decode them to obtain the decoded features S1, S2, and S3. Upsample the small-scale feature S i to the same size as the adjacent large-scale S i+1After making them consistent, input the cross-scale attention module to obtain the cross-layer multi-scale attention weights w i (1 ≤ i ≤ 2), and then obtain the feature maps T1, T2, and T3 weighted by the attention weights, as Figure 3 shown on the right side and Figure 6 as shown.

[0077] The expression for step 4.1 is:[[]]

[0078] w i = CAM(S i , S i+1 )

[0079]

[0080] where CAM(·) represents the cross-scale attention module, represents element-wise multiplication.

[0081] The cross-layer multi-scale attention module integrates the scale information of different receptive fields by learning the relationship between two different input sizes of the same scene image, and makes full use of the multi-scale feature representation to improve the spatial expression ability. The cross-scale attention module can learn the spatial correlation between two different input scales and effectively strengthen the dense feature aggregation ability between different scales.

[0082] In step 4.2, after step 4.1, the multi-scale information is injected into the main branch reconstruction, and the information interaction and refinement between channel features are carried out through the Spectral Prediction Unit (SPU). First, a 1×1 convolution is used to expand the number of channels of the aggregated feature F mer and divide it into G groups along its channel dimension. In addition, each group of features is divided into three parts where the first part is used for cross-channel information exchange with the next group of features , and the other two parts of the features are used for intra-group channel interaction learning, as Figure 3 shown on the right side,[[]] Figure 7 and Figure 8 as shown.

[0083] The expression for step 4.2 is:[[]]

[0084]

[0085] where L represents the output feature of the SPU, CCS(·) represents the operation of dividing the convolution into two groups of features after channel connection, and CIBlock(·) represents the channel interaction unit, and its structure is as Figure 8 shown.

[0086] After step 4.1, multi-scale information is injected into the main branch reconstruction. Similar to the spatial multi-scale characteristics, different channels also contain different information. Therefore, it is necessary to mine valuable clues contained in different channels. Through step 4.2, information interaction and refinement between channel features are carried out, as Figure 7 shown, enhancing the ability of the ideal hyperspectral image to reconstruct spectral information from coarse-grained to fine-grained.

[0087] In step 4.3, the multi-scale spatial detail features T1, T2, and T3 obtained in step 4.1 are injected into the spectral prediction module (SPU) in stages in the main branch, realizing spatial-spectral joint reconstruction, as Figure 3 shown on the right.

[0088] The expression of step 4.3 is:

[0089]

[0090] where, F out represents the finally output fused image, F mer represents the spatial-spectral aggregation feature, and SPU represents the spectral prediction module.

[0091] It should be noted that the parts that are the same or similar to those in Embodiment 1 and Embodiment 2 in this embodiment can be referred to each other and will not be elaborated in this application.

[0092] Embodiment 4

[0093] As another embodiment, this Embodiment 4 is proposed on the basis of Embodiments 1 to 3. Using this integrated fusion method for spaceborne panchromatic-multispectral-hyperspectral remote sensing images, integrated fusion of multi-source images is carried out:

[0094] In this embodiment, the data used are the existing domestic Ziyuan 02D (ZY-02D) satellite hyperspectral data and the internationally public Chikusei and Pavia Center hyperspectral images. Using computer software ENVI5.3, MATLAB2019a, and Pycharm, and the Pytorch framework based on NVIDIA Quadro P5000 GPU to implement the automatic operation process, the integrated fusion steps of multi-source images are as Figure 3 shown.

[0095] Step 1: Preprocess the dataset. First, for the ZY-1 02D images acquired under real conditions, image registration is required. Based on the multi-source image registration workflow tool in ENVI 5.3 software, manual-assisted point selection and correction are added, and finally three sets of georegistered images are obtained. In addition, bad bands are removed from the hyperspectral images to avoid the influence of distorted bands in the hyperspectral images on the accuracy of the final fused images. Finally, due to the lack of real images and the inability to obtain corresponding panchromatic-multispectral-hyperspectral images, the hyperspectral images of the Chikusei and Pavia regions are respectively processed through the Wald protocol to obtain low-resolution hyperspectral, medium-resolution multispectral, high-resolution panchromatic images, and high-resolution hyperspectral images for reference.

[0096] Step 2: Flip, rotate, crop and other processing are performed on the multi-source image data to achieve the dataset expansion method, so as to avoid problems such as overfitting caused by insufficient data during the deep learning training process.

[0097] Step 3: The spatial and spectral resolutions of the multispectral images themselves are at an intermediate level compared with the panchromatic and hyperspectral images, and they can be used as a bridge to solve problems such as spatial blurring and spectral distortion caused by the huge differences in the spatial and spectral resolution ratios in hyperspectral pansharpening. According to the characteristics of the original multi-source input images and aiming at the spatial and spectral characteristics, convolutional layers with different kernel sizes are used to extract the fine spatial information of the panchromatic images, the rich spectral information of the hyperspectral images, and the spatial and spectral information of the multispectral images respectively. Then, for the space and spectrum, spatial and spectral feature migration units are used to migrate the high-frequency detail information of the panchromatic images and the discriminative spectral information of the hyperspectral images to the multispectral images. Finally, the self-guided unit is used to screen out the important information of the multispectral image branch, and feature aggregation is performed with the shallow features and migration features to obtain the output of the spatial-spectral consistency component, reducing the influence of the huge spatial and spectral resolution gaps between the panchromatic and hyperspectral images on the fused images, such as blurring and distortion.

[0098] Step 4: The consistency component output obtained in Step 3 is encoded with shared parameters, and the decoded features of different scales are obtained. Then, it is input into the cross-scale attention module to learn the relationship between two different input sizes of images in the same scene, integrate the scale information of different receptive fields, make full use of the multi-scale feature representation, and strengthen the dense feature aggregation ability between different scales. Then, the obtained multi-scale information is injected into the main branch reconstruction in stages, and information interaction and refinement are performed between the channel features through the spectral prediction module to achieve spatial-spectral joint reconstruction.

[0099] Step 5: According to the supervised deep network training strategy, use the L1 norm to establish a loss model that fuses the image and the label image, and adopt the ADAM optimizer for optimization. During the training process, continuously approximate the fused result to the label image to obtain a high-spatial-resolution hyperspectral image.

[0100] It should be noted that the same or similar parts in this embodiment and Embodiments 1 to 3 can be referred to each other and will not be elaborated in this application.

[0101] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other.

Claims

1. An integrated fusion method for spaceborne panchromatic-multispectral-hyperspectral remote sensing images, characterized in that, Including the following steps: Step 1: Obtain panchromatic images, multispectral images, and hyperspectral images, and perform preprocessing to obtain the original multi-source input images; Step 2: Register the original multi-source input images and perform data augmentation processing; Step 3: According to the characteristics of the panchromatic and hyperspectral images, perform per-pixel feature similarity comparison on the spatial and spectral information of the multispectral image in the fused input image, so as to obtain the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs; In step 3, according to the characteristics of the original multi-source input images, and aiming at the spatial and spectral characteristics, use 3×3 convolution and 1×1 convolution to perform multi-source spatial-spectral feature extraction; respectively construct spatial and spectral migration branches, and combine the multi-source spatial-spectral features to transfer the high-frequency detail information of the panchromatic image and the discriminative spectral information of the hyperspectral image to the multispectral image, so as to obtain the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs; Step 4: reconstruct the shared parameters of the consistency component output to learn the spatial-spectral intrinsic feature correlation, perform cross-scale and level-by-level fusion, and thus produce a high-resolution hyperspectral fusion image; in step 4, the spatial-spectral aggregation features obtained in step 3 are encoded in a parameter-sharing manner, and then decoded to obtain the decoded features S1, S2 and S3. The scales of S1, S2 and S3 are increased in sequence, and the feature S i Upsample to feature S i+1 Consistent, then input the cross-scale attention module to obtain S i and S i+1 The cross-layer multi-scale attention weights w between i , 1≤i≤2, and then obtain the multi-scale spatial detail features T1, T2 and T3 weighted by the attention weights; in step 4, the information interaction and refinement between channel features are performed through the spectral prediction module, and first 1×1 convolution is used to expand the aggregate feature F mer The number of channels is divided into G groups along its channel dimension, namely gk, 1≤k≤G, and each group of features is divided into and Among them, each group of features g k In Used with the next set of features Exchange information across channels. and It is used for intra-group channel interactive learning; in step 4, the multi-scale spatial detail features T1, T2 and T3 are injected into the spectral prediction module in stages in the main branch to perform spatial-spectral joint reconstruction to obtain a high-resolution hyperspectral fusion image; in step 4, during the supervised training, the L1 norm loss function is used for optimization. The L1 norm loss function is used to calculate the Loss, which is the label H of N input images in each training iteration. ref And the fusion result H result The average and difference between .

Images