Satellite-borne panchromatic-multispectral-hyperspectral remote sensing image integrated fusion method

Abstract

The invention relates to a satellite-borne panchromatic-multispectral-hyperspectral remote sensing image integrated fusion method, which realizes purification of spatial high-frequency detail information and discriminative spectral information contained in a multi-source input image by means of the characteristics of a multispectral image, so that two pure characteristic components, namely a spatial component and a spectral component, are converged; the method is used for subsequent reconstruction; in spectral feature reconstruction, aggregation features are used as main branches, multi-scale coding features are used for detail injection, meanwhile, interactive reasoning is carried out on spectral features through channel expansion and splitting, upper and lower related clues are obtained, and finally an ideal image is generated. And the multi-source remote sensing space-spectrum high-fidelity fusion requirement under large spatial resolution difference and spectrum difference is met.

Description

Technical Field

[0001] The invention belongs to the field of remote sensing image processing, and in particular relates to an integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images. Background Art

[0002] With the maturity of satellite technology, hyperspectral imaging platforms have developed from the initial ground-based and airborne platforms to satellite-based platforms, which has greatly promoted the application of hyperspectral images in the fields of national defense, agriculture, forestry, environmental monitoring, etc. However, due to the performance limitations of optical remote sensing imaging systems, there is an inevitable trade-off between the spatial resolution and spectral resolution of imaging systems. The purpose of spatial-spectral fusion is to alleviate the problem of mutual constraints between high spatial resolution and high spectral resolution of imaging systems. By combining the complementary advantages of spatial and spectral images of multi-source remote sensing images in the same area, the images with both high spatial resolution and high spectral resolution are obtained by fusion, which can effectively give play to the advantages of multi-source complementary observation images, break through the performance constraints of a single sensor, and greatly improve the availability of hyperspectral images.

[0003] At present, a variety of integrated fusion methods for panchromatic, multispectral, and hyperspectral images have been proposed, mainly including traditional methods and deep learning-based methods. Traditional methods usually introduce the intermediate spatial and spectral information of multispectral images as a bridge between panchromatic and hyperspectral images to alleviate the challenges brought by spatial resolution and spectral differences. However, traditional methods only regard multispectral images as a collection of spatial-spectral information, and do not fully explore and utilize their spatial characteristics and discriminative spectral information in the integrated fusion process. Under the framework of deep learning, by introducing auxiliary images, the problem of improving the resolution of hyperspectral images can be more effectively solved, so as to better explore the intrinsic characteristics of panchromatic, multispectral, and hyperspectral images and further improve the fusion effect. However, although some progress has been made in the field of integrated fusion, there are still many challenges. First, with the increase of inter-image resolution differences, such as spatial and spectral resolution differences, it becomes more challenging to accurately fuse complementary information. Therefore, how to recover fine spectral and clear spatial information from two or more input images and accurately fuse them remains a key challenge. Secondly, existing deep learning methods can be roughly divided into two categories: one is to extract features by simply concatenating the input, and the other is to extract features separately through parallel networks and then fuse them. However, the former ignores the distribution gap between spatial and spectral dimensions, while the latter has difficulty capturing the structural consistency of spatial and spectral information.

[0004] Therefore, a new multi-source image integration fusion method is urgently needed to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an integrated fusion method for satellite-borne panchromatic-multispectral-hyperspectral remote sensing images.

[0006] This integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images includes the following steps:

[0007] Step 1: Obtain panchromatic images, multispectral images and hyperspectral images, and preprocess them as original multi-source input images;

[0008] Step 2: align the original multi-source input images and perform data enhancement processing;

[0009] Step 3: According to the characteristics of panchromatic and hyperspectral images, the spatial and spectral information of the multispectral image in the fused input image are compared for pixel-by-pixel feature similarity, so as to obtain the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs;

[0010] Step 4: Reconstruct the shared parameters of the consistency component output to learn the spatial-spectral intrinsic feature correlation, perform cross-scale and level-by-level fusion, and thus produce a high-resolution hyperspectral fused image.

[0011] Preferably, in step three, according to the characteristics of the original multi-source input image, and for the spatial and spectral characteristics, 3×3 convolution and 1×1 convolution are used to extract multi-source spatial-spectral features; spatial and spectral migration branches are constructed respectively, and combined with multi-source spatial-spectral features, the high-frequency detail information of the panchromatic image and the discriminative spectral information of the hyperspectral image are migrated to the multispectral image, and the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs are obtained.

[0012] Preferably, in step 4, the spatial-spectral aggregation feature obtained in step 3 is encoded in a parameter sharing manner, and then decoded to obtain decoded features S1, S2 and S3, and the scales of S1, S2 and S3 are increased in sequence, and feature S i Upsample to feature S i+1 Consistent, then input the cross-scale attention module to obtain S i and S i+1 The cross-layer multi-scale attention weights w between i , 1≤i≤2, and then obtain the multi-scale spatial detail features T1, T2 and T3 weighted by the attention weights.

[0013] As a preferred method, in step 4, the information interaction and refinement between channel features are performed through the spectral prediction module. First, a 1×1 convolution is used to expand the aggregate feature F mer The number of channels is divided into G groups along its channel dimension, namely g k ,1≤k≤G, each set of features is divided into and Among them, each group of features g k In Used with the next set of features Exchange information across channels. and Used for intra-group channel interactive learning.

[0014] Preferably, in step 4, the multi-scale spatial detail features T1, T2 and T3 are injected into the spectral prediction module in stages in the main branch to perform spatial-spectral joint reconstruction to obtain a high-resolution hyperspectral fusion image.

[0015] As a preferred method, in step 4, during the supervised training, the L1 norm loss function is used for optimization. The L1 norm loss function is used to calculate the Loss, which is the label H of N input images in each training iteration. ref And the fusion result H result The average and difference between .

[0016] The beneficial effects of the present invention are:

[0017] 1) The present invention uses the characteristics of multispectral images to purify the spatial high-frequency detail information and discriminative spectral information contained in multi-source input images, thereby merging them into two pure feature components, namely spatial component and spectral component, for subsequent reconstruction; in spectral feature reconstruction, the aggregated feature is used as the main branch, and while multi-scale coding features are used to inject details, the spectral features are interactively inferred through channel expansion and splitting to obtain relevant clues, and finally generate an ideal image.

[0018] 2) The panchromatic and hyperspectral image joint spatial-spectral information reconstruction framework based on multispectral imagery and the novel hyperspectral image spatial-spectral feature migration fusion network proposed in the present invention can overcome the technical bottleneck of multi-source remote sensing spatial-spectral high-fidelity fusion under large spatial spectral resolution differences that existing methods are difficult to achieve, meet the needs of multi-source remote sensing spatial-spectral high-fidelity fusion under large spatial resolution differences and spectral differences, and produce high-resolution hyperspectral image data with clear space and fidelity spectrum. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the integrated fusion method of panchromatic / multispectral / hyperspectral images;

[0020] Figure 2 It is a schematic diagram of the image preprocessing process;

[0021] Figure 3 It is a schematic diagram of the overall structure of the integrated fusion method of panchromatic / multispectral / hyperspectral images;

[0022] Figure 4 It is a schematic diagram of the self-booting module structure;

[0023] Figure 5 This is a schematic diagram of the structure of the spatial-spectral feature migration module;

[0024] Figure 6 Schematic diagram of the cross-scale attention module structure;

[0025] Figure 7 This is a schematic diagram of the spectrum prediction module structure;

[0026] Figure 8 Schematic diagram of the channel interaction unit structure. DETAILED DESCRIPTION

[0027] The present invention is further described below in conjunction with embodiments. The description of the following embodiments is only used to help understand the present invention. It should be noted that for ordinary persons in the art, without departing from the principle of the present invention, the present invention can also be modified in some ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

[0028] Embodiment 1

[0029] As an embodiment, the integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images first pre-processes and performs data enhancement operations on panchromatic, multispectral, and hyperspectral images. Secondly, after extracting spatial and spectral features, the spatial information of the panchromatic image and the spectral information of the hyperspectral image are respectively migrated to the multispectral image according to the feature characteristics and using the spatial-spectral feature migration network to obtain preliminary spatial and spectral components. Then, with the help of the self-guided screened multispectral features and the preliminary extracted features, feature aggregation is performed to obtain the final spatial, spectral, and spatial-spectral feature components. Finally, the output of the consistency component is reconstructed with shared parameters to learn the spatial-spectral intrinsic feature correlation, and cross-scale and level-by-level fusion is performed to produce high-resolution hyperspectral images.

[0030] The following modules are involved:

[0031] An acquisition module, used for acquiring panchromatic images, multispectral images and hyperspectral images, and preprocessing the panchromatic images, multispectral images and hyperspectral images;

[0032] The registration module is used to accurately register the panchromatic image, multispectral image and hyperspectral image, and then perform data enhancement processing on each group of images to ensure that the images to be fused have the same ground object scene information and projection system;

[0033] The feature extraction module is used to compare the pixel-by-pixel feature similarity of the spatial and spectral information of the multispectral image according to the characteristics of the panchromatic and hyperspectral images, so as to obtain the spatial consistency and spectral consistency component outputs;

[0034] The information reconstruction module is used to reconstruct the shared parameters of the consistency component output to learn the spatial-spectral intrinsic feature correlation and fuse them across scales and levels to produce high-resolution hyperspectral fused images.

[0035] like Figure 1 As shown in FIG. 1 , the integrated fusion method of the satellite-borne panchromatic-multispectral-hyperspectral remote sensing images includes the following steps:

[0036] Step 1: Obtain panchromatic, multispectral and hyperspectral images, and preprocess the panchromatic, multispectral and hyperspectral images.

[0037] Multi-sensor data include panchromatic, multispectral and hyperspectral images, such as Figure 2 As shown, the present application first preprocesses the multi-sensor data. The preprocessing before fusing the panchromatic, multispectral and hyperspectral images mainly includes operations such as radiation calibration, atmospheric correction, and orthorectification. In addition, in order to prevent the distorted bands of the hyperspectral image from affecting the accuracy of the fused image, the hyperspectral image also needs to remove bad bands.

[0038] Step 2: Accurately align the panchromatic, multispectral and hyperspectral images, then perform data enhancement on each group of images and ensure that the images to be fused have the same ground scene information and projection system.

[0039] In step 2, according to the original multi-source input images obtained in step 1, due to experimental hardware limitations, a group of original multi-source input images cannot be directly input into the network for training. In addition, the deep learning network training requires a large amount of data. In order to avoid overfitting during network training and improve the robustness and generalization ability of the model, data enhancement processing is used in this embodiment, that is, a group of original multi-source input images is divided into small pieces of images, and each group of images is flipped, rotated, cropped, etc.

[0040] Step 3: According to the characteristics of panchromatic and hyperspectral images, the spatial and spectral information of multispectral images are compared pixel by pixel for similarity, so as to obtain spatial consistency and spectral consistency component outputs, such as Figure 3 , Figure 4 and Figure 5 shown. Figure 3 and Figure 6 In the figure, C represents the feature concatenation step, S represents the sigmoid activation function, ↑ represents the upsampling operation, SPU represents the spectral prediction unit, and CAM represents the cross-scale attention module.

[0041] like Figure 3 As shown on the left, a spatial-spectral feature migration module is established, including feature extraction of different input images, spatial / spectral feature migration, and feature self-guided enhancement of multispectral branches. It should be noted that the spatial and spectral resolutions of multispectral images themselves are at an intermediate level compared to panchromatic and hyperspectral images, and can serve as a bridge to solve the problems of spatial blur and spectral distortion caused by the huge difference in spatial and spectral resolution ratios faced by hyperspectral pan-sharpening. Through step three, with the help of the characteristics of multispectral images, the spatial high-frequency detail information and discriminative spectral information contained in the multi-source input images can be purified, thereby merging into two pure feature components, namely spatial components and spectral components, for subsequent reconstruction.

[0042] Step 4: Reconstruct the shared parameters of the consistent component output to learn the spatial-spectral intrinsic feature correlation, cross-scale and level-by-level fusion, and thus produce high-resolution hyperspectral images, such as Figure 3 , Figure 6 , Figure 7 and Figure 8 As shown in the figure. Conv means convolution, Relu means rectified linear, AvgPool means average pooling, MaxPool means maximum pooling, the C in the diamond box means channel concatenation-convolution-grouping operation, and CIBlock means channel interaction unit.

[0043] like Figure 3 As shown on the right, a spatial-spectral joint reconstruction module is established, including the generation of cross-scale attention feature maps and spectral prediction reconstruction. It should be noted that multi-scale features are particularly important for image fusion, and multi-scale features cannot be extracted using only simple convolution operations. Through the progressive fusion of step four, the spatial-spectral features are first decoded in a multi-scale, shared parameter manner, in which the scale consistency is enforced between different decoder scales through the cross-scale attention module. In the spectral feature reconstruction, the aggregated features are used as the main branch, and the multi-scale encoding features are used to inject details. At the same time, the spectral features are interactively inferred through channel expansion and splitting to obtain upper and lower related clues, and finally generate an ideal high-resolution hyperspectral fusion image.

[0044] Step 5: During the supervised training process, the L1 loss function is used for optimization to make the fused image generated in step 4 closer to the reference image. The reference image is the original high-resolution hyperspectral image. The expression is:

[0045]

[0046] Among them, Loss represents the label H calculated using the L1 norm of N input images in each training iteration ref And the fusion result H result The average and difference between .

[0047] Since it is difficult to obtain a hyperspectral image with real spatial resolution in a real imaging environment, multiple pairs of training data are obtained by simulating spatial and spectral degradation of the training images based on the Wald protocol. Furthermore, in the process of supervised training, the present application uses the L1 norm loss function for optimization to obtain a hyperspectral image with high spatial resolution.

[0048] Embodiment 2

[0049] As another embodiment, this embodiment 2 proposes a more specific integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images based on the embodiment 1, and step 3 is specifically as follows:

[0050] Step 3.1, based on the characteristics of the original multi-source input image, and targeting the spatial and spectral characteristics, 3×3 convolution and 1×1 convolution are used to extract multi-source spatial-spectral features, which can be expressed as:

[0051] F P =Conv3(σ(Conv3(σ(Conv3(P)))))

[0052] F h =Conv1(σ(Conv1(σ(Conv1(H)))))

[0053]

[0054] Where P represents the input panchromatic image, H represents the input hyperspectral image, M represents the input multispectral image, σ(·) represents the Relu activation function, Conv3 and Conv1 represent the convolution kernels of 3×3 and 1×1 respectively, and F P represents the spatial features extracted in the full-color branch, F h represents the spectral features extracted in the hyperspectral branch, and They represent the spatial and spectral features obtained after the corresponding convolution of the multispectral branch, respectively.

[0055] For multi-source input images, panchromatic images contain fine spatial information, hyperspectral images have rich spectral information, and multispectral images contain both spatial and spectral information. Therefore, the panchromatic branch uses 3×3 convolution to extract spatial information, the hyperspectral branch uses 1×1 convolution that pays more attention to channel information to extract information, and the multispectral branch uses both 3×3 and 1×1 convolution to extract information.

[0056] Step 3.2, using the two core information extracted from the three input images in step 3.1, namely, space and spectrum, adopts the idea of ​​feature migration to migrate the high-frequency detail information of the panchromatic image and the discriminative spectral information of the hyperspectral image to the multispectral image, which can greatly improve its spatial detail information and spectral discriminative features, such as Figure 5 shown.

[0057] The expression of step 3.2 is:

[0058]

[0059] M spa =max(M spa,((u,v),(x,y)) )

[0060]

[0061] M spe =max(M spe,(a,b) )

[0062]

[0063] Among them, Cos(·) represents the cosine similarity distance, Norm(·) represents the operation of normalizing the calculation result to the interval (0,1), and max(·) represents the maximum value function. represents element-wise multiplication, represents element-wise addition, for The vector representation after expansion along the channel dimension at the position (u,v), F p,(x,y) For Similarly, M spa represents the spatial weight map, f spa Represents the spatial component output of the enhanced spatial features of the panchromatic image, Indicates that the multispectral image branch is located in the ath column of spectral features, F h,b Indicates the spectral features of the hyperspectral image branch located in the bth column, M spe represents the spectral weight map, f spe Represents the output of the spectral migration of the hyperspectral image.

[0064] The multispectral branch will extract spatial and spectral information, of which the spatial information will be subjected to feature migration operation with the high-frequency detail information of the panchromatic image, and the spectral information will be subjected to feature migration operation with the discriminative spectral information of the hyperspectral image.

[0065] Step 3.3, a self-guided unit is used to screen the importance information of the multispectral image branch. At the same time, the important features in the previous step are reused to avoid the disappearance of important features during the network training process, such as Figure 3 and Figure 4 shown.

[0066] The expression of step 3.3 is:

[0067]

[0068] F spa =Conv(Cat(F p ,f spa ,F))

[0069] F spe =Conv(Cat(F h ,f spe ,F))

[0070] F mer =Conv(Cat(F spa ,F,F spe ))

[0071] Where Sig(·) represents the Sigmoid activation function, Cov(·) represents the convolution operation, Cat(·) represents the operation of superimposing input features in the channel dimension, F represents the output of the multispectral image branch after the self-guided unit, and F spa represents the final spatial component, F spe represents the final spectral component, F mer represents the spatial-spectral aggregation feature.

[0072] The feature F output by the self-guided module and the shallow feature F obtained in step 3.1 p and F h , and the feature f obtained by feature migration in step 3.2 spa With f spa , feature aggregation is performed to obtain the final spatial component, spectral component and space-spectrum aggregation features.

[0073] It should be noted that the parts in this embodiment that are the same or similar to those in the first embodiment can be referenced to each other and will not be described in detail in this application.

[0074] Embodiment 3

[0075] As another embodiment, this embodiment 3 proposes a more specific integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images based on the first and second embodiments, and step 4 is specifically as follows:

[0076] Step 4.1: first, encode the spatial spectrum features obtained in step 3.3 in a parameter sharing manner, and then decode them to obtain the decoded features S1, S2, and S3, and convert the small-scale features S i Upsample to the adjacent large scale S i+1Consistent, then input the cross-scale attention module to obtain the cross-layer multi-scale attention weights w of the two i (1≤i≤2), and then obtain the feature maps T1, T2, and T3 weighted by the attention weights, such as Figure 3 Right and Figure 6 shown.

[0077] The expression of step 4.1 is:

[0078] w i =CAM(S i ,S i+1 )

[0079]

[0080] Among them, CAM(·) represents the cross-scale attention module, Represents element-wise multiplication.

[0081] The cross-layer multi-scale attention module integrates the scale information of different receptive fields by learning the relationship between two different input sizes of the same scene image, making full use of multi-scale feature representation to improve spatial expression capabilities. The cross-scale attention module can learn the spatial correlation between two different input scales, effectively enhancing the ability to aggregate dense features between different scales.

[0082] Step 4.2: After step 4.1, the multi-scale information is injected into the main branch reconstruction, and the information interaction and refinement between channel features are performed through the Spectral Prediction Unit (SPU). First, a 1×1 convolution is used to expand the aggregate feature F. mer The number of channels is divided into G groups along its channel dimension. In addition, each set of features is split into three Among them, the first Used with the next set of features To exchange cross-channel information, the other two features Used for intra-group channel interactive learning, such as Figure 3 Right side, Figure 7 and Figure 8 shown.

[0083] The expression of step 4.2 is:

[0084]

[0085] Where L represents the output feature of SPU, CCS(·) represents the operation of convolution after channel connection and dividing it into two groups of features, and CIBlock(·) represents the channel interaction unit, whose structure is as follows: Figure 8 shown.

[0086] After step 4.1, multi-scale information is injected into the main branch reconstruction. Similar to the spatial multi-scale characteristics, different channels also contain different information. Therefore, it is necessary to explore the valuable clues contained in different channels. Through step 4.2, information interaction and refinement between channel features are carried out, such as Figure 7 As shown, the ability to reconstruct the spectral information of ideal hyperspectral images from coarse-grained to fine-grained is enhanced.

[0087] In step 4.3, the multi-scale spatial detail features T1, T2, and T3 obtained in step 4.1 are injected into the spectral prediction module (SPU) in stages in the main branch to achieve spatial-spectral joint reconstruction, such as Figure 3 Shown on the right.

[0088] The expression of step 4.3 is:

[0089]

[0090] Among them, F out represents the final output fused image, F mer represents the spatial-spectral aggregation feature, and SPU represents the spectral prediction module.

[0091] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 and 2 can be referenced to each other and will not be described in detail in this application.

[0092] Embodiment 4

[0093] As another embodiment, this embodiment 4 proposes, based on embodiments 1 to 3, to use the integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images to perform integrated fusion of multi-source images:

[0094] In this embodiment, the data used are the existing domestic resource 02D (ZY-02D) satellite hyperspectral data and the internationally public Chikusei and Pavia Center hyperspectral images. The computer software ENVI5.3, MATLAB2019a and Pycharm, and the Pytorch framework based on NVIDIA Quadro P5000 GPU are used to realize the automatic operation process. The integrated fusion steps of multi-source images are as follows Figure 3 shown.

[0095] Step 1: Preprocess the data set. First, for the ZY-1 02D images acquired under real conditions, image registration is required. Based on the multi-source image registration process tool of ENVI5.3 software, manual auxiliary point selection correction is added, and finally three sets of images after geo-registration are obtained. In addition, the bad bands of the hyperspectral image are removed to avoid the distortion bands in the hyperspectral image affecting the accuracy of the final fused image. Finally, due to the lack of real images, the corresponding panchromatic-multispectral-hyperspectral images cannot be obtained. The hyperspectral images of Chikusei and Pavia are respectively obtained through the Wald protocol to obtain low-resolution hyperspectral, medium-resolution multispectral, high-resolution panchromatic images and high-resolution hyperspectral images used as references.

[0096] Step 2: Flip, rotate, crop, and other processes on the multi-source image data to expand the data set, so as to avoid problems such as overfitting due to insufficient data during deep learning training.

[0097] Step 3. The spatial and spectral resolutions of multispectral images are at an intermediate level compared to those of panchromatic and hyperspectral images, and can serve as a bridge to solve the problems of spatial blur and spectral distortion caused by the huge difference in spatial and spectral resolution ratios faced by hyperspectral pan-sharpening. According to the characteristics of the original multi-source input images, and targeting the spatial and spectral features, convolutional layers with different convolution kernel sizes are used to extract the fine spatial information of panchromatic images, the rich spectral information of hyperspectral images, and the spatial and spectral information of multispectral images. Then, for space and spectrum, spatial and spectral feature migration units are used to migrate the high-frequency detail information of panchromatic images and the discriminative spectral information of hyperspectral images to multispectral images. Finally, the importance information of the multispectral image branches screened out by the self-guided unit is aggregated with shallow features and migration features to obtain the spatial spectral consistency component output, which reduces the blur and distortion caused by the huge spatial and spectral resolution gap between panchromatic and hyperspectral images on the fused image.

[0098] Step 4: Encode the shared parameters of the consistency component output obtained in step 3, decode it to obtain decoding features of different scales, and then input it into the cross-scale attention module to learn the relationship between two different input sizes of the same scene image, integrate the scale information of different receptive fields, make full use of multi-scale feature representation, and enhance the ability to aggregate dense features between different scales. Then, inject the obtained multi-scale information into the main branch reconstruction in stages, and use the spectral prediction module to interact and refine the information between channel features, realizing spatial-spectral joint reconstruction.

[0099] Step 5: According to the supervised deep network training strategy, the L1 norm is used to establish the loss model of the fused image and the labeled image, and the ADAM optimizer is used for optimization. During the training process, the fusion result is continuously approached to the labeled image to obtain a hyperspectral image with high spatial resolution.

[0100] It should be noted that the parts in this embodiment that are the same or similar to those in Embodiments 1 to 3 can be referenced to each other and will not be described in detail in this application.

[0101] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

Claims

1. A method for integrating the fusion of satellite-borne panchromatic, multispectral and hyperspectral remote sensing images, characterized in that: The following steps are involved: Step 1: Obtain panchromatic images, multispectral images and hyperspectral images, and preprocess them as original multi-source input images; Step 2: align the original multi-source input images and perform data enhancement processing; Step 3: According to the characteristics of panchromatic and hyperspectral images, the spatial and spectral information of the multispectral image in the fused input image are compared for pixel-by-pixel feature similarity, so as to obtain the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs; Step 4: Reconstruct the shared parameters of the consistency component output to learn the spatial-spectral intrinsic feature correlation, perform cross-scale and level-by-level fusion, and thus produce a high-resolution hyperspectral fused image.

2. The integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images according to claim 1 is characterized in that: In step three, according to the characteristics of the original multi-source input image, and for the spatial and spectral characteristics, 3×3 convolution and 1×1 convolution are used to extract multi-source spatial-spectral features; spatial and spectral migration branches are constructed respectively, and combined with multi-source spatial-spectral features, the high-frequency detail information of the panchromatic image and the discriminative spectral information of the hyperspectral image are migrated to the multispectral image, and the spatial-spectral aggregation features of the spatially consistent and spectrally consistent component outputs are obtained.

3. The integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images according to claim 1 is characterized in that: In step 4, the spatial-spectral aggregation features obtained in step 3 are encoded in a parameter sharing manner and then decoded to obtain decoded features S1, S2 and S3. The scales of S1, S2 and S3 are increased in sequence. i Upsample to feature S i+1 Consistent, then input the cross-scale attention module to obtain S i and S i+1 The cross-layer multi-scale attention weights w between i , 1≤i≤2, and then obtain the multi-scale spatial detail features T1, T2 and T3 weighted by the attention weights.

4. The integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images according to claim 3 is characterized in that: In step 4, the spectral prediction module is used to interact and refine the information between channel features. First, a 1×1 convolution is used to expand the aggregate feature F mer The number of channels is divided into G groups along its channel dimension, namely g k , 1≤k≤G, each set of features is divided into and Among them, each group of features.gk Used with the next set of features 1≤j≤3 for cross-channel information exchange, and Used for intra-group channel interactive learning.

5. The integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images according to claim 3 is characterized in that: In step 4, the multi-scale spatial detail features T1, T2 and T3 are injected into the spectral prediction module in stages in the main branch to perform spatial-spectral joint reconstruction to obtain a high-resolution hyperspectral fusion image.

6. The integrated fusion method of satellite-borne panchromatic-multispectral-hyperspectral remote sensing images according to claim 1, characterized in that: In step 4, during the supervised training, the L1 norm loss function is used for optimization. The L1 norm loss function is used to calculate the Loss, which is the label H for each N input images in each training iteration. ref And the fusion result H result The average and difference between .

Images