A hyperspectral and panchromatic image fusion method, system, device and medium based on adaptive feature modulation network

Abstract

The present invention discloses a method, system, device and medium for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network. The method directly obtains more comprehensive panchromatic image detail information from the high-frequency features of the panchromatic image through an octave convolution unit, avoids complex filter design, and fully mines the spatial details of the panchromatic image. The hyperspectral image features are extracted through a spatially and spectrally separable three-dimensional convolution unit, which reduces the number of network parameters, improves the algorithm operation speed, and efficiently captures the multi-scale spatial-spectral features of the hyperspectral image. The adaptive feature modulation module not only effectively utilizes the detail features of the panchromatic image to enhance the spatial information of the fused image, but also can adaptively adjust the injected details to ensure the spectral fidelity of the fusion result, thereby realizing the precise injection of panchromatic image details. The fusion method proposed in the present invention fully mines and utilizes the spatial detail information of the panchromatic image and has clear interpretability.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a hyperspectral and panchromatic image fusion method, system, device and medium based on an adaptive feature modulation network. Background Art

[0002] Hyperspectral images are obtained by sensors simultaneously capturing both spectral and spatial information of ground objects using narrow wavelength intervals across the entire electromagnetic spectrum. These sensors can provide dozens, hundreds, or even thousands of spectral segments with spectral resolutions on the order of nanometers. Hyperspectral images are three-dimensional images that combine two-dimensional spatial scene information with one-dimensional continuous spectral information. Their high spectral resolution and unified image-spectrum integration have led to their widespread application in numerous fields, including civil and military. In practice, due to limitations of imaging systems, hyperspectral images often exhibit low spatial resolution, which limits their accurate interpretation and application. To address this issue, efficient and reliable image fusion methods, leveraging the high spatial resolution of panchromatic images, have been developed to fuse hyperspectral images of the same scene with high spatial resolution panchromatic images. This method generates fused images with both high spectral and spatial resolution. This method is crucial for accurately interpreting hyperspectral image information and effectively improving the value of hyperspectral data.

[0003] Existing hyperspectral image fusion methods can be divided into four categories based on their characteristics: component replacement methods, multiresolution analysis methods, model optimization-based methods, and deep learning-based methods. Component replacement methods use specific transformations to map the hyperspectral image to a new space. In this transformed domain, they replace the spatial information components with the panchromatic image, and then perform an inverse transformation to produce a fused high-resolution image. However, the fused images generated by these methods often suffer from severe spectral distortion. Multiresolution analysis methods use multiresolution decomposition to extract high-frequency spatial detail from the panchromatic image, which is then injected into the upsampled hyperspectral image to produce a fused image. Compared to component replacement methods, the fused images produced by multiresolution analysis methods often exhibit stronger spectral consistency, but are more susceptible to spatial information loss. Model optimization-based methods use a degradation model to establish the degradation relationship between a high-resolution panchromatic image and a low-resolution hyperspectral image to an ideal high-resolution hyperspectral image. By setting prior information and constraints, they obtain a high-resolution hyperspectral image from the known panchromatic and hyperspectral images. These methods require iterative model optimization, which is computationally time-consuming and inefficient.

[0004] Deep learning-based methods are a rapidly emerging remote sensing image fusion technology in recent years, demonstrating excellent fusion performance and strong application potential. For example, G. Masi et al. (Pansharpening by Convolutional Neural Networks. Remote Sensing, 2016, 8(7):594.) proposed a pan-sharpening method based on a convolutional neural network (CNN). The method takes the image block formed by stacking the panchromatic image and the upsampled multispectral image as input, and completes the fusion by learning the mapping relationship between the input and the high-resolution remote sensing image through the network. L.He et al. (HyperPNN: Hyperspectral Pansharpening via Spectrally Predictive Convolutional Neural Networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8): 3092-3100.) proposed two convolutional neural network models for the fusion of hyperspectral and panchromatic images, namely HyperPNN1 and HyperPNN2, both of which showed better spectral preservation performance than traditional methods.

[0005] However, the prior art has the following disadvantages:

[0006] (1) The existing fusion methods based on component replacement and multi-resolution analysis use linear models, which leads to insufficient utilization of the detail information of the full-color image during the fusion process, resulting in spatial blurring, and also causes spectral distortion in the fusion results due to inaccurate detail injection.

[0007] (2) Existing fusion methods based on model optimization are highly dependent on spatial and spectral degradation relationships and prior constraints, and require iterative solutions to highly complex model optimization problems, which are computationally time-consuming and inefficient.

[0008] (3) Existing deep learning-based fusion methods do not fully consider the specific knowledge of the hyperspectral image fusion problem, and the fusion network lacks interpretability. Secondly, the extraction of detail information from panchromatic images relies on complex spatial filter design, which cannot fully mine the detail information of panchromatic images. In addition, the utilization of detail information is insufficient in the fusion process, and the detail injection in the fusion result is not accurate enough, making it difficult to strike a balance between spectral information preservation and spatial quality enhancement. Summary of the Invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a hyperspectral and panchromatic image fusion method, system, device and medium based on an adaptive feature modulation network, which directly obtains more comprehensive panchromatic image detail information from high-frequency features through an octave convolution unit, avoids complex filter design, and fully mines the spatial details of the panchromatic image; extracts hyperspectral image features through a spatially and spectrally separable three-dimensional convolution unit, reduces the number of network parameters, improves the operating speed, and efficiently captures the multi-scale spatial-spectral features of the hyperspectral image; through an adaptive feature modulation module, not only the detail features of the panchromatic image are effectively utilized to enhance the spatial information of the fused image, but also the injected details can be adaptively adjusted to ensure the spectral fidelity of the fusion result, thereby realizing the precise injection of panchromatic image details; compared with the existing fusion method based on deep learning, the fusion method proposed in the present invention fully mines and utilizes the spatial detail information of the panchromatic image and has clear interpretability.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network includes the following steps:

[0012] Step 1: Obtain the public original hyperspectral image and segment the original hyperspectral image in a certain pixel area to obtain several hyperspectral image blocks of the same size and without overlapping;

[0013] Step 2: Based on the Wald protocol, the original hyperspectral image blocks segmented in step 1 are used as reference images, and the reference images are preprocessed to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P. The preprocessed hyperspectral image blocks are then divided into a hyperspectral training set and a hyperspectral test set.

[0014] Step 3: Constructing a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0015] Step 4: Input the hyperspectral training set in step 2 into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network constructed in step 3 for training until the loss function converges, thereby obtaining a trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0016] Step 5: Input the hyperspectral test set in step 2 into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0017] The pre-processing in step 2 is specifically as follows:

[0018] The reference image is sequentially subjected to Gaussian blur filtering and downsampling to obtain a low-resolution hyperspectral image H; at the same time, the spectral bands of the visible light band of the reference image are averaged to obtain a high-resolution panchromatic image P.

[0019] The construction process of the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network in step 3 is as follows:

[0020] Step 3.1: Input the full-color image P obtained in step 2 into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P PD ;

[0021] The octave convolution unit includes a convolution layer with a convolution kernel size of 3×3, three octave convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; the three octave convolution layers are the first octave convolution layer, the second octave convolution layer and the third octave convolution layer, and the convolution kernel size is 3×3;

[0022] Step 3.2: Input the hyperspectral image H in step 2 into the spatial and spectral separable three-dimensional convolution unit for multi-scale feature extraction to obtain the spatial-spectral feature map F of the hyperspectral image HS ;

[0023] The spatially and spectrally separable 3D convolution unit includes three spatially and spectrally separable 3D convolution layers, a feature splicing layer, and a convolution layer with a convolution kernel size of 1×1; each spatially and spectrally separable 3D convolution layer includes two 3D convolution layers, two ReLU function activation layers, and a feature shaping layer. The convolution kernels of the two 3D convolution layers are k×k×1 and 1×1×k, respectively. The values ​​of k in the three spatially and spectrally separable 3D convolution layers are 3, 5, and 7, respectively.

[0024] Step 3.3: Transform the detail feature map F of the full-color image P in step 3.1 into PD and the spatial-spectral feature map F of the hyperspectral image in step 3.2 HS Input the adaptive feature modulation module to perform feature modulation and obtain the modulated feature map F M ;

[0025] The adaptive feature modulation module includes four feature modulation blocks, each of which is composed of two adaptive feature modulation units, two convolution layers and a ReLU function activation layer; each adaptive feature modulation unit includes a mapping function f M and two Sigmoid functions; the mapping function f MIt consists of two parallel three-layer convolutions, the convolution kernel size is 1×1, and the three convolution layers are connected by two ReLU function activation layers;

[0026] Step 3.4: Modulate the feature map F in step 3.2 M Input into the detail reconstruction module for detail reconstruction to obtain the reconstructed hyperspectral image details Hyperspectral image details Compared with the hyperspectral image after upsampling in step 2 Add the corresponding elements to obtain a high-resolution fused hyperspectral image

[0027] The detail reconstruction module consists of three convolutional layers, each with a convolution kernel size of 3×3, and the three convolutional layers are connected by two ReLU function activation layers.

[0028] The process of extracting multi-layer high-frequency detail features in step 3.1 is as follows:

[0029] Step 3.1.1: Input the full-color image P in step 2 into the convolution layer with a convolution kernel size of 3×3 for feature extraction, and obtain the initial feature map X with a dimension of h×w×c ini , h, w, c represent the height, width and number of channels of the feature map respectively;

[0030] Step 3.1.2: Initial feature map X ini Input the first octave convolution layer for convolution to obtain the first high-frequency feature map X with a dimension of h×w×(1-α)c H1 ; At the same time, the initial feature map X ini Perform average pooling and convolution in sequence to obtain a dimension of The first low-frequency feature map X L1 , where α represents the value assigned to the first low-frequency feature map X L1 The ratio of channels, α∈[0,1];

[0031] Step 3.1.3: The first high frequency feature map X H1 and the first low-frequency feature map X L1 Input the second octave convolution layer for same-frequency update and cross-frequency communication to obtain the second high-frequency feature map X H2 and the second low-frequency feature map X L2 ;

[0032] Step 3.1.4: Transform the second high-frequency feature map X H2 and the second low-frequency feature map X L2 Input the third octave convolution layer and perform 3×3 convolution to obtain the second high-frequency feature map X′ after convolution. H2 and the second low-frequency feature map X′ L2, and then the second low-frequency feature map X′ after convolution L2 Upsample to obtain the second low-frequency feature map X′ after upsampling L2U , and then the second high-frequency feature map X′ after convolution H2 and the second low-frequency feature map X′ L2U Add together to get the third high-frequency feature map X output by the third octave convolution layer H3 ;

[0033] Step 3.1.5: Take the first high-frequency feature map X output by the first octave convolutional layer H1 , the second high-frequency feature map X output by the second octave convolution layer H2 And the third high-frequency feature map X output by the third octave convolution layer H3 The input feature splicing layer performs channel dimension splicing of the feature map, and then the spliced ​​feature map is input into the convolution layer with a convolution kernel size of 1×1 for convolution to obtain the detail feature map F of the full-color image P. PD .

[0034] The specific process of the same frequency update is as follows:

[0035] For the first high-frequency feature map X H1 and the first low-frequency feature map X L1 Perform 3×3 convolution to obtain the updated first high-frequency feature map X′ H1 And the updated first low-frequency feature map X′ L1 ;

[0036] The specific process of the cross-frequency communication is as follows:

[0037] The first low-frequency feature map X L1 Perform 3×3 convolution and upsampling in sequence to obtain the first low-frequency feature map X″ after convolution and upsampling L1 , the first low-frequency feature map X″ after upsampling L1 and the updated first high-frequency feature map X′ H1 Add together to get the second high-frequency feature map X H2 ; At the same time, the first high-frequency feature map X H1 Perform average pooling and 3×3 convolution in sequence to obtain the first high-frequency feature map X″ after average pooling and convolution H1 , the first high-frequency feature map X″ H1 and the first low-frequency feature map X′ L1 Add together to get the second low-frequency feature map X L2 .

[0038] The process of multi-scale feature extraction in step 3.2 is as follows:

[0039] Step 3.2.1: Perform bicubic interpolation upsampling on the hyperspectral image H in step 2 so that the low-resolution hyperspectral image H has the same size as the panchromatic image P, and obtain the upsampled hyperspectral image

[0040] Step 3.2.2: Upsample the hyperspectral image in step 3.2.1 The data are input into three spatial and spectral separable 3D convolutional layers for simultaneous feature extraction to obtain the first hyperspectral image spatial-spectral feature map. Second hyperspectral image spatial-spectral feature map And the third hyperspectral image spatial-spectral feature map

[0041] The feature extraction process of each spatially and spectrally separable 3D convolutional layer is as follows:

[0042] The hyperspectral image after upsampling in step 3.2.1 The input convolution kernel is k×k×1 three-dimensional convolution layer for spatial convolution to obtain the hyperspectral image Spatial feature map Use ReLU function to transform the spatial feature map Activate and obtain the activated spatial feature map The activated spatial feature map Input into the three-dimensional convolution layer with a convolution kernel of 1×1×k for spectral convolution to obtain a hyperspectral image Spatial-spectral characteristic diagram of Use ReLU function to transform the spatial-spectral feature map Activate and obtain the activated spatial-spectral feature map The dimension is h×w×d×c, where h, w, d are the height, width and depth of the feature map respectively, and c is the number of channels; the activated spatial-spectral feature map Input to the feature shaping layer for feature shaping operation, reshaping from h×w×d×c dimension to h×w×(d×c) dimension, and obtaining the hyperspectral image space-spectral feature map

[0043] Step 3.2.3: Transform the first hyperspectral image spatial-spectral feature map Second hyperspectral image spatial-spectral feature map And the third hyperspectral image spatial-spectral feature map Input the feature splicing layer to perform feature map channel dimension splicing to obtain the spliced ​​feature map F SC , and then the concatenated feature map F SCInput the convolution layer with a convolution kernel size of 1×1 to perform convolution and obtain the spatial-spectral feature map F of the hyperspectral image HS .

[0044] The process of feature modulation in step 3.3 is as follows:

[0045] Step 3.3.1: Transform the detail feature map F of the full-color image P into PD The first adaptive feature modulation unit input to the first feature modulation block uses the mapping function f M The detail feature map F of the full-color image P PD Mapped into a set of intermediate parameters α′ and β′, α′ and β′ are respectively multiplied by the corresponding elements of α′ and β′ after Sigmoid function activation to obtain the scale parameter α and the translation parameter β;

[0046] Step 3.3.2: Transform the spatial-spectral feature map F of the hyperspectral image into HS Input to the first adaptive feature modulation unit in the first feature modulation block, the spatial-spectral feature map F HS First, the corresponding elements are multiplied by the scale parameter α, and then the corresponding elements are added by the translation parameter β to obtain the modulated feature map F output by the first adaptive feature modulation unit. AFM1 ;

[0047] Step 3.3.3: Modulate the feature map F AFM1 The convolution layer with a convolution kernel size of 3×3 is input in sequence for convolution, and then the ReLU function activation layer is input for activation to obtain the feature map F′ AFM1 ; The detail feature map F of the full-color image P PD and feature map F′ AFM1 Input to the second adaptive feature modulation unit, repeat steps 3.3.1 and 3.3.2, and obtain the modulated feature map F output by the second adaptive feature modulation unit AFM2 ; The feature map F AFM2 Input the convolution layer with a convolution kernel size of 3×3 and perform convolution to obtain the feature map F′ AFM2 ; Use skip connections in each feature modulation block; transform the spatial-spectral feature map F of the hyperspectral image HS and feature map F′ AFM2 Add the corresponding elements to obtain the modulated feature map F output by the first feature modulation block M1 ;

[0048] Step 3.3.4: Transform the detail feature map F of the full-color image P into PD And the modulated feature map F output by the first feature modulation block M1Input to the second feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the second feature modulation block M2 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the second feature modulation block M2 Input to the third feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the third feature modulation block M3 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the third feature modulation block M3 Input to the fourth feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the fourth feature modulation block M4 ; Skip connections are used in the entire adaptive feature modulation module; the spatial-spectral feature map F of the hyperspectral image is converted HS And the modulated feature map F output by the fourth feature modulation block M4 Add the corresponding elements to obtain the feature map F modulated by the adaptive feature modulation module M .

[0049] The present invention also provides a hyperspectral and panchromatic image fusion system based on an adaptive feature modulation network, comprising:

[0050] Image segmentation module: used to segment the original hyperspectral image of a certain pixel area to obtain several hyperspectral image blocks of the same size and non-overlapping;

[0051] Image preprocessing module: used to preprocess the hyperspectral image blocks to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P;

[0052] Model construction module: used to construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0053] Model training module: input the hyperspectral training set into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network for training, and obtain the trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0054] Result output module: The hyperspectral test set is input into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0055] The present invention also provides a hyperspectral and panchromatic image fusion device based on an adaptive feature modulation network, comprising:

[0056] Memory: used for storing a computer program for implementing the above-mentioned hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network;

[0057] Processor: used to implement the above-mentioned hyperspectral and panchromatic image fusion method based on adaptive feature modulation network when executing the computer program.

[0058] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention constructs an octave convolution unit to hierarchically extract the high-frequency details of the full-color image, directly and effectively obtains more comprehensive full-color image detail information from the high-frequency features of the full-color image, and effectively improves the fusion performance.

[0061] 2. The present invention extracts hyperspectral image features by constructing a spatially and spectrally separable three-dimensional convolution unit, which reduces the number of network parameters, improves the algorithm operation speed, and efficiently captures the multi-scale spatial-spectral characteristics of hyperspectral images.

[0062] 3. The present invention constructs an adaptive feature modulation module and adds a gating mechanism in the adaptive feature modulation unit. By taking the full-color image detail information as a priori and adaptively modulating the hyperspectral image features, it can not only effectively combine the full-color image detail features to enhance the spatial information of the hyperspectral image, but also adaptively adjust the injected detail information to ensure the spectral fidelity of the fused image, thereby realizing the accurate injection of full-color image detail information.

[0063] In summary, the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network proposed in the present invention fully utilizes the detail information of the panchromatic image and realizes the accurate injection of details; the fused image has high quality, high fidelity of spectral information, and good preservation of spatial-spectral information; it strikes a balance between spectral information preservation and spatial quality enhancement, and can not only effectively combine the detail information of the panchromatic image, but also adaptively adjust the injected details to reduce spectral distortion, ensuring the accurate injection of detail information. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a flow chart of the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network provided by the present invention.

[0065] Figure 2 This is the fusion result diagram of the different fusion methods provided by the present invention on the 10th hyperspectral image block in the Washington DC Mall dataset, where: Figure 2 (a) in the figure is the reference image. Figure 2 (b) in the figure is GSA. Figure 2 (c) in the equation is PCA. Figure 2 (d) in the equation is HPF, Figure 2 (e) in is MGH, Figure 2 (f) in the figure is DiCNN. Figure 2 (g) in is HyperPNN, Figure 2 The (h) in the figure is AFMN.

[0066] Figure 3 The spectral angle mapping between the fusion results of the different fusion methods provided by the present invention on the 10th hyperspectral image block in the Washington DC Mall dataset and the reference image; wherein, Figure 3 (a) is the reference figure. Figure 3 (b) in the figure is GSA. Figure 3 (c) in the equation is PCA. Figure 3 (d) in the equation is HPF, Figure 3 (e) in is MGH, Figure 3 (f) in the figure is DiCNN. Figure 3 (g) in is HyperPNN, Figure 3 The (h) in the figure is AFMN. DETAILED DESCRIPTION

[0067] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0068] A hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network includes the following steps:

[0069] Step 1: Obtain the public original hyperspectral image and segment the original hyperspectral image in a certain pixel area to obtain several hyperspectral image blocks of the same size and without overlapping;

[0070] Step 2: Based on the Wald protocol, the original hyperspectral image blocks segmented in step 1 are used as reference images, and the reference images are preprocessed to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P. The preprocessed hyperspectral image blocks are then divided into a hyperspectral training set and a hyperspectral test set.

[0071] Specifically, the reference image is sequentially subjected to Gaussian blur filtering and downsampling to obtain a low-resolution hyperspectral image H. At the same time, the spectral bands of the visible light band of the reference image are averaged to obtain a high-resolution panchromatic image P. The preprocessed hyperspectral image blocks are divided into a hyperspectral training set and a hyperspectral test set, where the hyperspectral training set accounts for 70% of the hyperspectral dataset and the hyperspectral test set accounts for 30% of the hyperspectral dataset.

[0072] Step 3: Constructing a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0073] Step 3.1: Input the full-color image P obtained in step 2 into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P PD The octave convolution unit includes a convolution layer with a convolution kernel size of 3×3, three octave convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; the three octave convolution layers are the first octave convolution layer, the second octave convolution layer and the third octave convolution layer, and the convolution kernel size is 3×3;

[0074] Step 3.1.1: Input the full-color image P in step 2 into the convolution layer with a convolution kernel size of 3×3 for feature extraction, and obtain the initial feature map X with a dimension of h×w×c ini , h, w, c represent the height, width and number of channels of the feature map respectively;

[0075] Step 3.1.2: Initial feature map X ini Input the first octave convolution layer for convolution to obtain the first high-frequency feature map X with a dimension of h×w×(1-α)c H1 ; At the same time, the initial feature map X ini Perform average pooling and convolution in sequence to obtain a dimension of The first low-frequency feature map X L1 , where α represents the value assigned to the first low-frequency feature map X L1 The ratio of channels, α∈[0,1];

[0076] Step 3.1.3: The first high frequency feature map X H1 and the first low-frequency feature map X L1 Input the second octave convolution layer for same-frequency update and cross-frequency communication to obtain the second high-frequency feature map X H2 and the second low-frequency feature map X L2 ;

[0077] The specific process of the same frequency update is as follows:

[0078] For the first high-frequency feature map X H1 and the first low-frequency feature map X L1 Perform 3×3 convolution to obtain the updated first high-frequency feature map X′ H1 And the updated first low-frequency feature map X′ L1 ;

[0079] The specific process of the cross-frequency communication is as follows:

[0080] The first low-frequency feature map X L1 Perform 3×3 convolution and upsampling in sequence to obtain the first low-frequency feature map X″ after convolution and upsampling L1 , the first low-frequency feature map X″ after upsampling L1 and the updated first high-frequency feature map X′ H1 Add together to get the second high-frequency feature map X H2 ; At the same time, the first high-frequency feature map X H1 Perform average pooling and 3×3 convolution in sequence to obtain the first high-frequency feature map X″ after average pooling and convolution H1 , the first high-frequency feature map X″ H1 and the first low-frequency feature map X′ L1 Add together to get the second low-frequency feature map X L2 ;

[0081] Step 3.1.4: Transform the second high-frequency feature map X H2 and the second low-frequency feature map X L2 Input the third octave convolution layer and perform 3×3 convolution to obtain the second high-frequency feature map X′ after convolution. H2 and the second low-frequency feature map X′ L2 , and then the second low-frequency feature map X′ after convolution L2 Upsample to obtain the second low-frequency feature map X′ after upsampling L2U , and then the second high-frequency feature map X′ after convolution H2 and the second low-frequency feature map X′ L2U Add together to get the third high-frequency feature map X output by the third octave convolution layer H3 ;

[0082] Step 3.1.5: Take the first high-frequency feature map X output by the first octave convolutional layer H1 , the second high-frequency feature map X output by the second octave convolution layer H2 And the third high-frequency feature map X output by the third octave convolution layer H3 The input feature splicing layer performs channel dimension splicing of the feature map, and then the spliced ​​feature map is input into the convolution layer with a convolution kernel size of 1×1 for convolution to obtain the detail feature map F of the full-color image P. PD ;

[0083] Step 3.2: Input the hyperspectral image H in step 2 into the spatial and spectral separable three-dimensional convolution unit for multi-scale feature extraction to obtain the spatial-spectral feature map F of the hyperspectral image HS The spatially and spectrally separable three-dimensional convolution unit includes three spatially and spectrally separable three-dimensional convolution layers, a feature splicing layer, and a convolution layer with a convolution kernel size of 1×1. Each spatially and spectrally separable three-dimensional convolution layer includes two three-dimensional convolution layers, two ReLU function activation layers, and a feature shaping layer. The convolution kernels of the two three-dimensional convolution layers are k×k×1 and 1×1×k, respectively. The values ​​of k in the three spatially and spectrally separable three-dimensional convolution layers are 3, 5, and 7, respectively.

[0084] Step 3.2.1: Perform bicubic interpolation upsampling on the hyperspectral image H in step 2 so that the low-resolution hyperspectral image H has the same size as the panchromatic image P, and obtain the upsampled hyperspectral image

[0085] Step 3.2.2: Upsample the hyperspectral image in step 3.2.1 The data are input into three spatial and spectral separable 3D convolutional layers for simultaneous feature extraction to obtain the first hyperspectral image spatial-spectral feature map. Second hyperspectral image spatial-spectral feature map And the third hyperspectral image spatial-spectral feature map

[0086] The specific process of synchronous feature extraction is as follows:

[0087] The hyperspectral image after upsampling in step 3.2.1 When input into the first spatial and spectral separable 3D convolution layer, a 3D convolution layer with a convolution kernel of 3×3×1 is used for spatial convolution to obtain a hyperspectral image. Spatial feature map Use ReLU function to transform the spatial feature map Activate and obtain the activated spatial feature map The activated spatial feature map Input into the 3D convolution layer with a convolution kernel of 1×1×3 for spectral convolution to obtain a hyperspectral image Spatial-spectral characteristic diagram of Use ReLU function to transform the spatial-spectral feature map Activate and obtain the activated spatial-spectral feature map The dimension is h×w×d×c, where h, w, d are the height, width and depth of the feature map respectively, and c is the number of channels; the activated spatial-spectral feature map Input to the feature reshaping layer for feature reshaping operation, reshaping from h×w×d×c dimension to h×w×(d×c) dimension, and obtaining the first hyperspectral image space-spectral feature map

[0088] At the same time, the hyperspectral image after upsampling in step 3.2.1 is When input into the second spatial and spectral separable three-dimensional convolution layer, it passes through the three-dimensional convolution layer with a convolution kernel of 5×5×1, the ReLU function activation layer, the three-dimensional convolution layer with a convolution kernel of 1×1×5, the ReLU function activation layer and the feature shaping layer in sequence to obtain the second hyperspectral image spatial-spectral feature map

[0089] At the same time, the hyperspectral image after upsampling in step 3.2.1 is When input into the third spatial and spectral separable three-dimensional convolution layer, it passes through the three-dimensional convolution layer with a convolution kernel of 7×7×1, the ReLU function activation layer, the three-dimensional convolution layer with a convolution kernel of 1×1×7, the ReLU function activation layer and the feature shaping layer in sequence to obtain the third hyperspectral image spatial-spectral feature map

[0090] Step 3.2.3: Transform the first hyperspectral image spatial-spectral feature map Second hyperspectral image spatial-spectral feature map And the third hyperspectral image spatial-spectral feature map Input the feature splicing layer to perform feature map channel dimension splicing to obtain the spliced ​​feature map F SC , and then the concatenated feature map F SC Input the convolution layer with a convolution kernel size of 1×1 to perform convolution and obtain the spatial-spectral feature map F of the hyperspectral image HS ;

[0091] Step 3.3: Transform the detail feature map F of the full-color image P in step 3.1 into PD and the spatial-spectral feature map F of the hyperspectral image in step 3.2 HS Input the adaptive feature modulation module to perform feature modulation and obtain the modulated feature map F M The adaptive feature modulation module includes four feature modulation blocks, each of which is composed of two adaptive feature modulation units, two convolution layers and a ReLU function activation layer; each adaptive feature modulation unit includes a mapping function f M and two Sigmoid functions; the mapping function f M It consists of two parallel three-layer convolutions, the convolution kernel size is 1×1, and the three convolution layers are connected by two ReLU function activation layers;

[0092] Step 3.3.1: Transform the detail feature map F of the full-color image P into PD The first adaptive feature modulation unit input to the first feature modulation block uses the mapping function f M The detail feature map F of the full-color image P PD Mapped into a set of intermediate parameters α′ and β′, α′ and β′ are respectively multiplied by the corresponding elements of α′ and β′ after Sigmoid function activation to obtain the scale parameter α and the translation parameter β;

[0093] Step 3.3.2: Transform the spatial-spectral feature map F of the hyperspectral image into HS Input to the first adaptive feature modulation unit in the first feature modulation block, the spatial-spectral feature map F HS First, the corresponding elements are multiplied by the scale parameter α, and then the corresponding elements are added by the translation parameter β to obtain the modulated feature map F output by the first adaptive feature modulation unit. AFM1 ;

[0094] Step 3.3.3: Modulate the feature map F AFM1 The convolution layer with a convolution kernel size of 3×3 is input in sequence for convolution, and then the ReLU function activation layer is input for activation to obtain the feature map F′ AFM1 ; The detail feature map F of the full-color image P PD and feature map F′ AFM1 Input to the second adaptive feature modulation unit, repeat steps 3.3.1 and 3.3.2, and obtain the modulated feature map F output by the second adaptive feature modulation unit AFM2 ; The feature map F AFM2 Input the convolution layer with a convolution kernel size of 3×3 and perform convolution to obtain the feature map F′ AFM2 ; Use skip connections in each feature modulation block to strengthen cross-layer communication of features; transform the spatial-spectral feature map F of the hyperspectral image HS and feature map F′ AFM2 Add the corresponding elements to obtain the modulated feature map F output by the first feature modulation block M1 ;

[0095] Step 3.3.4: Transform the detail feature map F of the full-color image P into PD And the modulated feature map F output by the first feature modulation block M1 Input to the second feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the second feature modulation block M2 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the second feature modulation block M2Input to the third feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the third feature modulation block M3 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the third feature modulation block M3 Input to the fourth feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the fourth feature modulation block M4 ; Skip connections are used in the entire adaptive feature modulation module to enhance cross-layer communication of features; the spatial-spectral feature map F of the hyperspectral image is converted into HS And the modulated feature map F output by the fourth feature modulation block M4 Add the corresponding elements to obtain the feature map F modulated by the adaptive feature modulation module M ;

[0096] Step 3.4: Modulate the feature map F in step 3.2 M Input into the detail reconstruction module for detail reconstruction to obtain the reconstructed hyperspectral image details Hyperspectral image details Compared with the hyperspectral image after upsampling in step 2 Add the corresponding elements to obtain a high-resolution fused hyperspectral image

[0097] The detail reconstruction module consists of three convolutional layers, each with a convolution kernel size of 3×3, and the three convolutional layers are connected by two ReLU function activation layers;

[0098] Specifically, the feature map F modulated in step 3.2 is M The data is sequentially input into the convolution layer with a convolution kernel size of 3×3 and the ReLU function activation layer for detail reconstruction to obtain the reconstructed hyperspectral image details.

[0099] Step 4: Input the hyperspectral training set in step 2 into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network constructed in step 3 for training until the loss function converges, thereby obtaining a trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0100] Specifically, the loss function L1 is used to minimize the fusion of hyperspectral images. and the reference image H G The error between the two, using a learning rate of 10 -4 The Adam algorithm minimizes the loss function L1, and the batch size and number of training cycles are 16 and 2×10 respectively. 3 ;

[0101] The loss function L1 is defined as:

[0102]

[0103] Among them, N p is the number of training samples, j is the index of the training sample, For high-resolution fusion of hyperspectral images, is the reference image;

[0104] Step 5: Input the hyperspectral test set in step 2 into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0105] A hyperspectral and panchromatic image fusion system based on an adaptive feature modulation network, comprising:

[0106] Image segmentation module: used to segment the original hyperspectral image of a certain pixel area to obtain several hyperspectral image blocks of the same size and non-overlapping;

[0107] Image preprocessing module: used to preprocess the hyperspectral image blocks to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P;

[0108] Model construction module: used to construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0109] Model training module: input the hyperspectral training set into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network for training, and obtain the trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0110] Result output module: The hyperspectral test set is input into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0111] A hyperspectral and panchromatic image fusion device based on an adaptive feature modulation network, comprising:

[0112] Memory: used for storing a computer program for implementing the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network;

[0113] Processor: used to implement the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network when executing the computer program.

[0114] A computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network.

[0115] Simulation experiment

[0116] To comprehensively evaluate the performance of the proposed hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network, the Washington DC Mall dataset was selected for experimentation and analysis. The Washington DC Mall hyperspectral data, captured by the HYDICE sensor, depicts a shopping mall in Washington, D.C., consisting of 1280×307 pixels and 191 bands, covering the spectral range from 400 to 2500 nanometers. The 1200×240 pixel image in the upper right corner was selected and divided into 20 non-overlapping hyperspectral image cubes of 120×120×191 pixels. The preprocessing described in step 1 was then performed to obtain a low-resolution hyperspectral image of 30×30 spatial size and a high-resolution panchromatic image of 120×120 spatial size.

[0117] The proposed hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network (AFMN) is compared with six existing fusion methods. The six existing fusion methods include traditional component substitution (CS) and multi-resolution analysis (MRA)-based methods, namely the adaptive Schmidt quadrature (GSA) method, the principal component analysis (PCA) method, the high-pass filtering (HPF) method, the high-pass modulation (MGH) method based on the generalized Laplacian pyramid of the modulation transfer function, the convolutional neural network-based method, namely the detail injection-based convolutional neural network (DiCNN) method, and the convolutional neural network for hyperspectral and panchromatic image fusion (HyperPNN) method. The quality of the fused image is quantified using the evaluation metrics correlation coefficient (CC), spectral angle mapping (SAM), root mean square error (RMSE), and relative global synthesis error (ERGAS). A larger CC value indicates better fused image quality, a smaller SAM value indicates higher spectral fidelity of the fused image, a smaller RMSE value indicates smaller differences between the fused image and the reference image, and a smaller ERGAS value indicates better spatial and spectral quality of the fused image. The objective quantitative evaluation results of different fusion methods on the Washington DC Mall dataset are shown in Table 1, with the best results highlighted in bold.

[0118] Table 1 Objective quantitative evaluation results of different fusion methods

[0119]

[0120] As shown in Table 1, the proposed AFMN method achieves the best fusion performance, especially in terms of the correlation coefficient (CC) and relative global error of synthesis (ERGAS). This fully demonstrates that the proposed AFMN method can effectively extract and inject spatial detail information of panchromatic images while better preserving spectral characteristics.

[0121] Figure 2 The fusion results of different methods on the 10th hyperspectral image cube (named "Washington_Patch10") in the Washington DC Mall dataset are obtained. Figure 2 In each sub-graph, a region marked by a green frame is enlarged to facilitate visual comparison. Figure 2 It can be observed that the fusion results generated by the GSA, PCA, HPF, MGH, DiCNN and HyperPNN methods in the prior art have obvious chromatic aberration and spatial distortion in the magnified area; compared with the above-mentioned fusion methods in the prior art, the AFMN method proposed in the present invention presents clearer local structural information and less spectral distortion in the magnified area.

[0122] Figure 3 Spectral angle mapping between the fusion results of different methods on "Washington_Patch10" and the reference image; Figure 3 The scale in the reference image (a) represents the spectral angle mapping (SAM) value. The closer to the dark blue, the smaller the SAM value of the area. Figure 3 As can be seen from (h) in the figure, the dark blue coverage area of ​​the spectral angle map obtained by the AFMN method is consistent with Figure 3 The reference image (a) in the figure is closest, indicating that the SAM value of the fused image obtained by the AFMN method is the smallest. This conclusion is consistent with the objective evaluation results in Table 1, further illustrating that the fusion method proposed in this invention can better maintain the spectral characteristics.

Claims

1. A hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network, characterized in that: The steps include: Step 1: Obtain the public original hyperspectral image and segment the original hyperspectral image in a certain pixel area to obtain several hyperspectral image blocks of the same size and without overlapping; Step 2: Based on the Wald protocol, the original hyperspectral image block segmented in step 1 is used as the reference image, and the reference image is preprocessed to obtain a low-resolution hyperspectral image H and a high-resolution panchromatic image P respectively; Then, the preprocessed hyperspectral image blocks are divided into a hyperspectral training set and a hyperspectral test set; Step 3: Constructing a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module; The full-color image P obtained in step 2 is input into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P. PD The octave convolution unit includes a convolution layer with a convolution kernel size of 3×3, three octave convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; the three octave convolution layers are the first octave convolution layer, the second octave convolution layer and the third octave convolution layer, and the convolution kernel size is 3×3; The process of extracting multi-layer high-frequency detail features is as follows: (1): Input the full-color image P in step 2 into the convolution layer with a convolution kernel size of 3×3 for feature extraction, and obtain the initial feature map X with a dimension of h×w×c ini , h, w, c represent the height, width and number of channels of the feature map respectively; (2): The initial feature map X ini Input the first octave convolution layer for convolution to obtain the first high-frequency feature map X with a dimension of h×w×(1-α)c H1 ; At the same time, the initial feature map X ini Perform average pooling and convolution in sequence to obtain a dimension of The first low-frequency feature map X L1 , where α represents the value assigned to the first low-frequency feature map X L1 The ratio of channels, α∈[0,1]; (3): The first high-frequency feature map X H1 and the first low-frequency feature map X L1 Input the second octave convolution layer for same-frequency update and cross-frequency communication to obtain the second high-frequency feature map X H2 and the second low-frequency feature map X L2 ; (4): The second high-frequency feature map X H2 and the second low-frequency feature map X L2 Input the third octave convolution layer and perform 3×3 convolution to obtain the second high-frequency feature map X′ after convolution. H2 and the second low-frequency feature map X′ L2 , and then the second low-frequency feature map X′ after convolution L2 Upsample to obtain the second low-frequency feature map X′ after upsampling L2U , and then the convolution of the second high-frequency feature map X′ H2 and the second low-frequency feature map X′ L2U Add together to get the third high-frequency feature map X output by the third octave convolution layer H3 ; (5): The first high-frequency feature map X output by the first octave convolution layer H1 , the second high-frequency feature map X output by the second octave convolution layer H2 And the third high-frequency feature map X output by the third octave convolution layer H3 The input feature splicing layer performs channel dimension splicing of the feature map, and then the spliced ​​feature map is input into the convolution layer with a convolution kernel size of 1×1 for convolution to obtain the detail feature map F of the full-color image P. PD ; Step 4: Input the hyperspectral training set in step 2 into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network constructed in step 3 for training until the loss function converges, thereby obtaining a trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network; Step 5: Input the hyperspectral test set in step 2 into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

2. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 1, characterized in that: The pre-processing in step 2 is specifically as follows: The reference image is sequentially subjected to Gaussian blur filtering and downsampling to obtain a low-resolution hyperspectral image H; at the same time, the spectral bands of the visible light band of the reference image are averaged to obtain a high-resolution panchromatic image P.

3. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 1, characterized in that: The construction process of the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network in step 3 is as follows: Step 3.1: Input the full-color image P obtained in step 2 into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P PD ; Step 3.2: Input the hyperspectral image H in step 2 into the spatial and spectral separable three-dimensional convolution unit for multi-scale feature extraction to obtain the spatial-spectral feature map F of the hyperspectral image HS ; The spatially and spectrally separable 3D convolution unit includes three spatially and spectrally separable 3D convolution layers, a feature splicing layer, and a convolution layer with a convolution kernel size of 1×1; each spatially and spectrally separable 3D convolution layer includes two 3D convolution layers, two ReLU function activation layers, and a feature shaping layer. The convolution kernels of the two 3D convolution layers are k×k×1 and 1×1×k, respectively. The values ​​of k in the three spatially and spectrally separable 3D convolution layers are 3, 5, and 7, respectively. Step 3.3: Transform the detail feature map F of the full-color image P in step 3.1 into PD and the spatial-spectral feature map F of the hyperspectral image in step 3.2 HS Input the adaptive feature modulation module to perform feature modulation and obtain the modulated feature map F M ; The adaptive feature modulation module includes four feature modulation blocks, each of which is composed of two adaptive feature modulation units, two convolution layers and a ReLU function activation layer; each adaptive feature modulation unit includes a mapping function f M and two Sigmoid functions; the mapping function f M It consists of two parallel three-layer convolutions, the convolution kernel size is 1×1, and the three convolution layers are connected by two ReLU function activation layers; Step 3.4: Modulate the feature map F in step 3.2 M Input into the detail reconstruction module for detail reconstruction to obtain the reconstructed hyperspectral image details Hyperspectral image details Compared with the hyperspectral image after upsampling in step 2 Add the corresponding elements to obtain a high-resolution fused hyperspectral image The detail reconstruction module consists of three convolutional layers, each with a convolution kernel size of 3×3, and the three convolutional layers are connected by two ReLU function activation layers.

4. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 1, characterized in that: The specific process of the same frequency update is as follows: For the first high-frequency feature map X H1 and the first low-frequency feature map X L1 Perform 3×3 convolution to obtain the updated first high-frequency feature map X′ H1 And the updated first low-frequency feature map X′ L1 ; The specific process of the cross-frequency communication is as follows: The first low-frequency feature map X L1 Perform 3×3 convolution and upsampling in sequence to obtain the first low-frequency feature map X″ after convolution and upsampling L1 , the first low-frequency feature map X″ after upsampling L1 and the updated first high-frequency feature map X′ H1 Add together to get the second high-frequency feature map X H2 ; At the same time, the first high-frequency feature map X H1 Perform average pooling and 3×3 convolution in sequence to obtain the first high-frequency feature map X″ after average pooling and convolution H1 , the first high-frequency feature map X″ H1 and the first low-frequency feature map X′ L1 Add together to get the second low-frequency feature map X L2 .

5. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 3, characterized in that: The process of multi-scale feature extraction in step 3.2 is as follows: Step 3.2.1: Perform bicubic interpolation upsampling on the hyperspectral image H in step 2 so that the low-resolution hyperspectral image H has the same size as the panchromatic image P, and obtain the upsampled hyperspectral image Step 3.2.2: Upsample the hyperspectral image in step 3.2.1 The data are input into three spatial and spectral separable 3D convolutional layers for synchronous feature extraction to obtain the first hyperspectral image spatial-spectral feature map. Second hyperspectral image spatial-spectral feature map And the third hyperspectral image spatial-spectral feature map The feature extraction process of each spatially and spectrally separable 3D convolutional layer is as follows: The hyperspectral image after upsampling in step 3.2.1 The input convolution kernel is k×k×1 three-dimensional convolution layer for spatial convolution to obtain the hyperspectral image Spatial feature map Use ReLU function to transform the spatial feature map Activate and obtain the activated spatial feature map The activated spatial feature map Input into the three-dimensional convolution layer with a convolution kernel of 1×1×k for spectral convolution to obtain a hyperspectral image Spatial-spectral characteristic diagram of Use ReLU function to transform the spatial-spectral feature map Activate and obtain the activated spatial-spectral feature map The dimension is h×w×d×c, where h, w, d are the height, width and depth of the feature map respectively, and c is the number of channels; the activated spatial-spectral feature map Input to the feature shaping layer for feature shaping operation, reshaping from h×w×d×c dimension to h×w×(d×c) dimension, and obtaining the hyperspectral image space-spectral feature map Step 3.2.3: Transform the first hyperspectral image spatial-spectral feature map Second hyperspectral image spatial-spectral feature map And the third hyperspectral image spatial-spectral feature map Input the feature splicing layer to perform feature map channel dimension splicing to obtain the spliced ​​feature map F SC , and then the concatenated feature map F SC Input the convolution layer with a convolution kernel size of 1×1 to perform convolution and obtain the spatial-spectral feature map F of the hyperspectral image HS .

6. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 3, characterized in that: The process of feature modulation in step 3.3 is as follows: Step 3.3.1: Transform the detail feature map F of the full-color image P into PD The first adaptive feature modulation unit input to the first feature modulation block uses the mapping function f M The detail feature map F of the full-color image P PD Mapped into a set of intermediate parameters α′ and β′, α′ and β′ are respectively multiplied by the corresponding elements of α′ and β′ after Sigmoid function activation to obtain the scale parameter α and the translation parameter β; Step 3.3.2: Transform the spatial-spectral feature map F of the hyperspectral image into HS Input to the first adaptive feature modulation unit in the first feature modulation block, the spatial-spectral feature map F HS First, the corresponding elements are multiplied by the scale parameter α, and then the corresponding elements are added by the translation parameter β to obtain the modulated feature map F output by the first adaptive feature modulation unit. AFM1 ; Step 3.3.3: Modulate the feature map F AFM1 The convolution layer with a convolution kernel size of 3×3 is input in sequence for convolution, and then the ReLU function activation layer is input for activation to obtain the feature map F′ AFM1 ; The detail feature map F of the full-color image P PD and feature map F′ AFM1 Input to the second adaptive feature modulation unit, repeat steps 3.3.1 and 3.3.2, and obtain the modulated feature map F output by the second adaptive feature modulation unit AFM2 ; The feature map F AFM2 Input the convolution layer with a convolution kernel size of 3×3 and perform convolution to obtain the feature map F′ AFM2 ; Use skip connections in each feature modulation block; transform the spatial-spectral feature map F of the hyperspectral image HS and feature map F′ AFM2 Add the corresponding elements to obtain the modulated feature map F output by the first feature modulation block M1 ; Step 3.3.4: Transform the detail feature map F of the full-color image P into PD And the modulated feature map F output by the first feature modulation block M1 Input to the second feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the second feature modulation block M2 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the second feature modulation block M2 Input to the third feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the third feature modulation block M3 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the third feature modulation block M3 Input to the fourth feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the fourth feature modulation block M4 ; Skip connections are used in the entire adaptive feature modulation module; the spatial-spectral feature map F of the hyperspectral image is converted HS And the modulated feature map F output by the fourth feature modulation block M4 Add the corresponding elements to obtain the feature map F modulated by the adaptive feature modulation module M .

7. A hyperspectral and panchromatic image fusion system based on an adaptive feature modulation network according to any one of claims 1 to 6, characterized in that: include: Image segmentation module: used to segment the original hyperspectral image of a certain pixel area to obtain several hyperspectral image blocks of the same size and non-overlapping; Image preprocessing module: used to preprocess the hyperspectral image blocks to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P; Model construction module: used to construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module; Model training module: input the hyperspectral training set into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network for training, and obtain the trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network; Result output module: The hyperspectral test set is input into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

8. A hyperspectral and panchromatic image fusion device based on an adaptive feature modulation network, characterized in that: include: Memory: used to store a computer program for implementing the hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network according to claims 1 to 6; Processor: used to implement the hyperspectral and panchromatic image fusion method based on adaptive feature modulation network of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement claims 1-6, a method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network.

Images