Adaptive feature modulation network-based hyperspectral and panchromatic image fusion method, system and device, and medium

Through the adaptive feature modulation network, the convolution unit can be separated by octave convolution and spatial spectrum, combined with the adaptive feature modulation module, the problems of spectral distortion and spatial information loss in hyperspectral image fusion are solved, and the efficient hyperspectral is achieved Fusion with full-color images improves fusion performance and interpretability.

CN118967473AActive Publication Date: 2024-11-15XIDIAN UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411030964.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

The existing hyperspectral image fusion methods have problems such as spectral distortion, spatial information loss, low computational efficiency and lack of interpretability, especially in the use of full-color image detail information.

Method used

Using the method based on adaptive feature modulation network, the high-frequency details of the full-color image are extracted through the octave convolution unit, and the three-dimensional convolution unit can be separated in space and spectral 3D convolution unit to extract the high-spectral image features. Combined with the adaptive feature modulation module, the network parameters are reduced , improve the running speed, and adaptively adjust the details during the fusion process to ensure spectral fidelity.

Benefits of technology

Achieve high-quality hyperspectral and full-color images, improve fusion performance, ensure the fidelity and spatial quality of spectral information, and balance the spectral information maintenance and spatial quality enhancement, providing clear and interpretable sex.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967473A_ABST
    Figure CN118967473A_ABST
Patent Text Reader

Abstract

The invention discloses a hyperspectral and panchromatic image fusion method, system and device based on an adaptive feature modulation network, and a medium, and aims to directly obtain more comprehensive panchromatic image detail information from panchromatic image high-frequency features through an octave convolution unit, thereby avoiding complex filter design. Full mining of full-color image space details is realized; the hyperspectral image features are extracted through the space and spectrum separable three-dimensional convolution unit, the network parameter quantity is reduced, the algorithm operation speed is improved, and the multi-scale space-spectrum features of the hyperspectral image are efficiently captured; through the adaptive feature modulation module, not only is the detail features of the panchromatic image effectively utilized to enhance the spatial information of the fused image, but also the injected details can be adaptively adjusted to ensure the spectral fidelity of the fusion result, and accurate injection of the details of the panchromatic image is realized; the fusion method provided by the invention fully excavates and utilizes the spatial detail information of the panchromatic image, and has clear interpretability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a method, system, device and medium for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network. Background Art

[0002] Hyperspectral images are obtained by sensors that simultaneously capture the spectral information and spatial information of ground objects using narrow wavelength intervals within the entire electromagnetic spectrum. They can provide dozens, hundreds or even thousands of spectral segments, with a spectral resolution of nanometers. Hyperspectral images are three-dimensional images that integrate two-dimensional spatial scene information and one-dimensional continuous spectral information. They have the advantages of high spectral resolution and unified image spectrum, and have been widely used in many fields such as civil and military. In practice, due to the limitations of the imaging system, hyperspectral images often present low spatial resolution, which limits the accurate interpretation and application of hyperspectral images. To address this problem, based on the high spatial resolution of panchromatic images, by studying efficient and reliable image fusion methods, the hyperspectral image of the same scene and the panchromatic image with high spatial resolution are fused to generate a fused image with both high spectral and high spatial resolution, which is of great significance for accurately interpreting hyperspectral image information and effectively improving the value of hyperspectral data.

[0003] According to the characteristics of different methods, the existing hyperspectral image fusion methods can be divided into four categories: component replacement method, multi-resolution analysis method, model optimization-based method and deep learning-based method. The component replacement method uses a specific transformation to map the hyperspectral image to a new space, and uses the panchromatic image to replace the spatial information component in this transformation domain and then performs an inverse transformation to obtain a fused high-resolution image. The fused image generated by this type of method often has a more serious spectral distortion problem. The multi-resolution analysis method obtains the high-frequency spatial detail information of the panchromatic image through multi-resolution decomposition, and then injects it into the upsampled hyperspectral image to generate a fused image. Compared with the component replacement method, the fused image obtained by the multi-resolution analysis method often has stronger spectral consistency, but it is more likely to have spatial information loss. The model optimization-based method uses a degradation model to establish the degradation relationship between the high-resolution panchromatic image and the low-resolution hyperspectral image to the ideal high-resolution hyperspectral image. By setting prior information and constraints, the high-resolution hyperspectral image is obtained based on the known panchromatic image and hyperspectral image. This type of method requires iterative solution of the model optimization problem, which is time-consuming and inefficient.

[0004] The deep learning-based method is a remote sensing image fusion technology that has rapidly emerged in recent years, showing excellent fusion performance and strong application potential. For example, G.Masi et al. (Pansharpening by Convolutional Neural Networks. Remote Sensing, 2016, 8(7): 594.) proposed a panchromatic sharpening method based on a convolutional neural network (CNN), which takes the image block formed by stacking the panchromatic image and the upsampled multispectral image as input, and completes the fusion by learning the mapping relationship between the input and the high-resolution remote sensing image through the network. L. He et al. (HyperPNN: Hyperspectral Pansharpening via Spectrally Predictive Convolutional Neural Networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(8): 3092-3100.) proposed two convolutional neural network models for the fusion of hyperspectral and panchromatic images, namely HyperPNN1 and HyperPNN2, both of which showed better spectral preservation performance than traditional methods.

[0005] However, the prior art has the following disadvantages:

[0006] (1) The existing fusion methods based on component replacement and multi-resolution analysis use linear models, which leads to insufficient utilization of the detail information of the full-color image during the fusion process, resulting in spatial blurring, and also causes spectral distortion in the fusion result due to inaccurate detail injection.

[0007] (2) Existing fusion methods based on model optimization are highly dependent on spatial and spectral degradation relationships and prior constraints, and require iterative solutions to highly complex model optimization problems, which are computationally time-consuming and inefficient.

[0008] (3) Existing deep learning-based fusion methods do not fully consider the specific knowledge of the hyperspectral image fusion problem, and the fusion network lacks interpretability. Secondly, the extraction of detail information of the panchromatic image relies on complex spatial filter design, which cannot fully mine the detail information of the panchromatic image. In addition, the use of detail information is not sufficient in the fusion process, and the detail injection in the fusion result is not accurate enough, making it difficult to strike a balance between spectral information preservation and spatial quality enhancement. Summary of the invention

[0009] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a method, system, device and medium for the fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network, which directly obtains more comprehensive panchromatic image detail information from high-frequency features through an octave convolution unit, avoids complex filter design, and fully mines the spatial details of the panchromatic image; extracts hyperspectral image features through a spatially and spectrally separable three-dimensional convolution unit, reduces the amount of network parameters, improves the operating speed, and efficiently captures the multi-scale spatial-spectral features of the hyperspectral image; through an adaptive feature modulation module, not only the detail features of the panchromatic image are effectively utilized to enhance the spatial information of the fused image, but also the injected details can be adaptively adjusted to ensure the spectral fidelity of the fusion result, thereby realizing the precise injection of panchromatic image details; compared with the existing fusion method based on deep learning, the fusion method proposed in the present invention fully mines and utilizes the spatial detail information of the panchromatic image, and has clear interpretability.

[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0011] A method for fusing hyperspectral and panchromatic images based on an adaptive feature modulation network comprises the following steps:

[0012] Step 1: Obtain the original public hyperspectral image, segment the original hyperspectral image of a certain pixel area, and obtain several hyperspectral image blocks of the same size and non-overlapping;

[0013] Step 2: Based on the Wald protocol, the original hyperspectral image blocks segmented in step 1 are used as reference images, and the reference images are preprocessed to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P, respectively; and the preprocessed hyperspectral image blocks are divided into a hyperspectral training set and a hyperspectral test set;

[0014] Step 3: construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0015] Step 4: Input the hyperspectral training set in step 2 into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network constructed in step 3 for training until the loss function converges, thereby obtaining a trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0016] Step 5: Input the hyperspectral test set in step 2 into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0017] The pre-processing in step 2 is specifically as follows:

[0018] The reference image is subjected to Gaussian blur filtering and downsampling in turn to obtain a low-resolution hyperspectral image H; meanwhile, the spectral bands of the visible light band of the reference image are averaged to obtain a high-resolution panchromatic image P.

[0019] The construction process of the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network in step 3 is as follows:

[0020] Step 3.1: Input the full-color image P obtained in step 2 into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P PD ;

[0021] The octave convolution unit includes a convolution layer with a convolution kernel size of 3×3, three octave convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; the three octave convolution layers are respectively the first octave convolution layer, the second octave convolution layer and the third octave convolution layer, and the convolution kernel size is 3×3;

[0022] Step 3.2: Input the hyperspectral image H in step 2 into the spatial and spectral separable three-dimensional convolution unit for multi-scale feature extraction to obtain the spatial-spectral feature map F of the hyperspectral image HS ;

[0023] The spatial and spectrally separable three-dimensional convolution unit includes three spatial and spectrally separable three-dimensional convolution layers, a feature concatenation layer and a convolution layer with a convolution kernel size of 1×1; each spatial and spectrally separable three-dimensional convolution layer includes two three-dimensional convolution layers, two ReLU function activation layers and a feature shaping layer, the convolution kernels of the two three-dimensional convolution layers are k×k×1 and 1×1×k respectively, and the values ​​of k in the three spatial and spectrally separable three-dimensional convolution layers are 3, 5 and 7 respectively;

[0024] Step 3.3: Transform the detail feature map F of the full-color image P in step 3.1 PD and the spatial-spectral characteristic map F of the hyperspectral image in step 3.2 HS Input the adaptive feature modulation module to perform feature modulation and obtain the modulated feature map F M ;

[0025] The adaptive feature modulation module includes four feature modulation blocks, each of which is composed of two adaptive feature modulation units, two convolution layers and a ReLU function activation layer; each adaptive feature modulation unit includes a mapping function f M and two Sigmoid functions; the mapping function f MIt consists of two parallel three-layer convolutions, the convolution kernel size is 1×1, and the three convolution layers are connected by two ReLU function activation layers;

[0026] Step 3.4: The feature map F modulated in step 3.2 M Input to the detail reconstruction module for detail reconstruction to obtain the reconstructed hyperspectral image details Hyperspectral image details Compared with the hyperspectral image after upsampling in step 2 Add the corresponding elements to obtain a high-resolution fused hyperspectral image

[0027] The detail reconstruction module consists of three convolutional layers, the convolution kernel size is 3×3, and the three convolutional layers are connected by two ReLU function activation layers.

[0028] The process of extracting multi-layer high-frequency detail features in step 3.1 is as follows:

[0029] Step 3.1.1: Input the full-color image P in step 2 into the convolution layer with a convolution kernel size of 3×3 for feature extraction, and obtain the initial feature map X with a dimension of h×w×c ini , h, w, c represent the height, width and number of channels of the feature map respectively;

[0030] Step 3.1.2: Initial feature map X ini Input the first octave convolution layer for convolution to obtain the first high-frequency feature map X with a dimension of h×w×(1-α)c H1 ; At the same time, the initial feature map X ini Perform average pooling and convolution in sequence to obtain a dimension of The first low-frequency feature map X L1 , where α represents the value assigned to the first low-frequency feature map X L1 The ratio of channels, α∈[0,1];

[0031] Step 3.1.3: The first high frequency feature map X H1 and the first low-frequency feature map X L1 Input the second octave convolution layer for same-frequency update and cross-frequency communication to obtain the second high-frequency feature map X H2 and the second low-frequency feature map X L2 ;

[0032] Step 3.1.4: Transform the second high-frequency feature map X H2 and the second low-frequency feature map X L2 The input is respectively subjected to 3×3 convolution in the third octave convolution layer to obtain the second high-frequency feature map X′ after convolution. H2 and the second low-frequency feature map X′ L2, and then the second low-frequency feature map X′ after convolution L2 Upsample to obtain the second low-frequency feature map X′ after upsampling L2U , and then the second high-frequency feature map X′ after convolution H2 and the second low-frequency feature map X′ L2U Add together to get the third high-frequency feature map X output by the third octave convolution layer H3 ;

[0033] Step 3.1.5: Take the first high-frequency feature map X output by the first octave convolutional layer H1 , the second high-frequency feature map X output by the second octave convolution layer H2 And the third high-frequency feature map X output by the third octave convolution layer H3 The input feature splicing layer performs channel dimension splicing of the feature map, and then the spliced ​​feature map is input into the convolution layer with a convolution kernel size of 1×1 for convolution to obtain the detail feature map F of the full-color image P. PD .

[0034] The specific process of the same frequency update is as follows:

[0035] For the first high-frequency feature map X H1 and the first low-frequency feature map X L1 Perform 3×3 convolution to obtain the updated first high-frequency feature map X′ H1 And the updated first low-frequency feature map X L '1;

[0036] The specific process of the cross-frequency communication is as follows:

[0037] The first low-frequency feature map X L1 Perform 3×3 convolution and upsampling in sequence to obtain the first low-frequency feature map X after convolution and upsampling. L ″1, the first low-frequency feature map X after upsampling L ″1 and the updated first high-frequency feature map X′ H1 Add together to get the second high-frequency feature map X H2 ; At the same time, the first high-frequency feature map X H1 Perform average pooling and 3×3 convolution in sequence to obtain the first high-frequency feature map X″ after average pooling and convolution H1 , the first high-frequency feature map X″ H1 With the first low-frequency feature map X′ L1 Add together to get the second low-frequency feature map X L2 .

[0038] The process of multi-scale feature extraction in step 3.2 is as follows:

[0039] Step 3.2.1: Perform bicubic interpolation upsampling on the hyperspectral image H in step 2 so that the low-resolution hyperspectral image H has the same size as the panchromatic image P, and obtain the upsampled hyperspectral image

[0040] Step 3.2.2: Upsample the hyperspectral image in step 3.2.1 The images are input into three spatial and spectral separable 3D convolutional layers for simultaneous feature extraction to obtain the first hyperspectral image spatial-spectral feature map. Second hyperspectral image spatial-spectral feature map And the third hyperspectral image space-spectral feature map

[0041] The feature extraction process of each spatially and spectrally separable 3D convolutional layer is as follows:

[0042] The hyperspectral image after upsampling in step 3.2.1 The input convolution kernel is k×k×1, and the three-dimensional convolution layer is used for spatial convolution to obtain the hyperspectral image. Spatial feature map Use ReLU function to transform the spatial feature map Activate and obtain the activated spatial feature map The activated spatial feature map Input to the three-dimensional convolution layer with a convolution kernel of 1×1×k for spectral convolution to obtain a hyperspectral image Spatial-spectral characteristics of Use ReLU function to transform the spatial-spectral feature map Activate and obtain the activated spatial-spectral feature map The dimension is h×w×d×c, where h, w, d are the height, width and depth of the feature map, and c is the number of channels. The input is sent to the feature shaping layer for feature shaping operation, and the h×w×d×c dimension is reshaped into the h×w×(d×c) dimension to obtain the hyperspectral image space-spectral feature map

[0043] Step 3.2.3: Transform the spatial-spectral feature map of the first hyperspectral image Second hyperspectral image spatial-spectral feature map And the third hyperspectral image space-spectral feature map Input the feature concatenation layer to concatenate the feature map channel dimensions and obtain the concatenated feature map F SC , and then the concatenated feature map F SCInput the convolution layer with a convolution kernel size of 1×1 to obtain the spatial-spectral feature map F of the hyperspectral image. HS .

[0044] The process of feature modulation in step 3.3 is as follows:

[0045] Step 3.3.1: Transform the detail feature map F of the full-color image P PD The first adaptive feature modulation unit input to the first feature modulation block uses the mapping function f M The detail feature map F of the full-color image P PD Mapped into a set of intermediate parameters α′ and β′, α′ and β′ are respectively multiplied by the corresponding elements of α′ and β′ after Sigmoid function activation to obtain the scale parameter α and the translation parameter β;

[0046] Step 3.3.2: Transform the spatial-spectral feature map F of the hyperspectral image HS Input to the first adaptive feature modulation unit in the first feature modulation block, the spatial-spectral feature map F HS First, the corresponding elements are multiplied by the scale parameter α, and then the corresponding elements are added by the translation parameter β to obtain the modulated feature map F output by the first adaptive feature modulation unit. AFM1 ;

[0047] Step 3.3.3: The modulated feature map F AFM1 The convolutional layers with a convolution kernel size of 3×3 are input in sequence for convolution, and then the ReLU function activation layer is input for activation to obtain the feature map F′ AFM1 ; The detail feature map F of the full-color image P PD and feature map F′ AFM1 Input to the second adaptive feature modulation unit, repeat steps 3.3.1 and 3.3.2, and obtain the modulated feature map F output by the second adaptive feature modulation unit AFM2 ; The feature map F AFM2 Input the convolution layer with a convolution kernel size of 3×3 for convolution to obtain the feature map F′ AFM2 ; Skip connections are used in each feature modulation block; the spatial-spectral feature map F of the hyperspectral image is transformed HS and feature map F′ AFM2 Add the corresponding elements to obtain the modulated feature map F output by the first feature modulation block M1 ;

[0048] Step 3.3.4: Transform the detail feature map F of the full-color image P into PD And the modulated feature map F output by the first feature modulation block M1Input to the second feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the second feature modulation block M2 ; The detail feature map F of the full-color image P PD The modulated feature map F output by the second feature modulation block M2 Input to the third feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the third feature modulation block M3 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the third feature modulation block M3 Input to the fourth feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the fourth feature modulation block M4 ; Skip connections are used in the entire adaptive feature modulation module; the spatial-spectral feature map F of the hyperspectral image HS And the modulated feature map F output by the fourth feature modulation block M4 Add the corresponding elements to obtain the feature map F modulated by the adaptive feature modulation module M .

[0049] The present invention also provides a hyperspectral and panchromatic image fusion system based on an adaptive feature modulation network, comprising:

[0050] Image segmentation module: used to segment the original hyperspectral image of a certain pixel area to obtain several hyperspectral image blocks of the same size and non-overlapping;

[0051] Image preprocessing module: used to preprocess the hyperspectral image blocks to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P;

[0052] Model construction module: used to construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0053] Model training module: input the hyperspectral training set into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network for training, and obtain the trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0054] Result output module: The hyperspectral test set is input into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0055] The present invention also provides a hyperspectral and panchromatic image fusion device based on an adaptive feature modulation network, comprising:

[0056] Memory: used for storing a computer program for realizing the above-mentioned hyperspectral and panchromatic image fusion method based on adaptive feature modulation network;

[0057] Processor: used to implement the above-mentioned hyperspectral and panchromatic image fusion method based on adaptive feature modulation network when executing the computer program.

[0058] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the above-mentioned hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention constructs an octave convolution unit to hierarchically extract the high-frequency details of the full-color image, directly and effectively obtains more comprehensive full-color image detail information from the high-frequency features of the full-color image, and effectively improves the fusion performance.

[0061] 2. The present invention extracts hyperspectral image features by constructing a spatially and spectrally separable three-dimensional convolution unit, which reduces the amount of network parameters, improves the algorithm operation speed, and efficiently captures the multi-scale spatial-spectral features of hyperspectral images.

[0062] 3. The present invention constructs an adaptive feature modulation module and adds a gating mechanism in the adaptive feature modulation unit. By taking the panchromatic image detail information as a priori and adaptively modulating the hyperspectral image features, it can not only effectively combine the panchromatic image detail features to enhance the spatial information of the hyperspectral image, but also adaptively adjust the injected detail information to ensure the spectral fidelity of the fused image, thereby realizing the accurate injection of the panchromatic image detail information.

[0063] In summary, the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network proposed in the present invention makes full use of the detail information of the panchromatic image and realizes the accurate injection of details; the fused image has high quality, high fidelity of spectral information, and good preservation of spatial-spectral information; it strikes a balance between spectral information preservation and spatial quality enhancement, which can not only effectively combine the detail information of the panchromatic image, but also adaptively adjust the injected details to reduce spectral distortion, thereby ensuring the accurate injection of detail information. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flow chart of a hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network provided by the present invention.

[0065] Figure 2 This is a fusion result diagram of different fusion methods provided by the present invention on the 10th hyperspectral image block in the Washington DC Mall dataset, where: Figure 2 (a) is the reference image, Figure 2 (b) for GSA, Figure 2 (c) is PCA, Figure 2 (d) is HPF, Figure 2 (e) is MGH, Figure 2 (f) is DiCNN, Figure 2 (g) is HyperPNN, Figure 2 (h) is AFMN.

[0066] Figure 3 The spectral angle mapping between the fusion results of the different fusion methods provided by the present invention on the 10th hyperspectral image block in the Washington DC Mall dataset and the reference image; wherein, Figure 3 (a) is the reference picture, Figure 3 (b) for GSA, Figure 3 (c) is PCA, Figure 3 (d) is HPF, Figure 3 (e) is MGH, Figure 3 (f) is DiCNN, Figure 3 (g) is HyperPNN, Figure 3 (h) is AFMN. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0068] A method for fusing hyperspectral and panchromatic images based on an adaptive feature modulation network comprises the following steps:

[0069] Step 1: Obtain the original public hyperspectral image, segment the original hyperspectral image of a certain pixel area, and obtain several hyperspectral image blocks of the same size and non-overlapping;

[0070] Step 2: Based on the Wald protocol, the original hyperspectral image blocks segmented in step 1 are used as reference images, and the reference images are preprocessed to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P, respectively; and the preprocessed hyperspectral image blocks are divided into a hyperspectral training set and a hyperspectral test set;

[0071] Specifically, the reference image is subjected to Gaussian blur filtering and downsampling in sequence to obtain a low-resolution hyperspectral image H; at the same time, the spectral bands of the visible light band of the reference image are averaged to obtain a high-resolution panchromatic image P; the preprocessed hyperspectral image blocks are divided into a hyperspectral training set and a hyperspectral test set, where the hyperspectral training set accounts for 70% of the hyperspectral data set, and the hyperspectral test set accounts for 30% of the hyperspectral data set;

[0072] Step 3: construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0073] Step 3.1: Input the full-color image P obtained in step 2 into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P PD ; The octave convolution unit includes a convolution layer with a convolution kernel size of 3×3, three octave convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; the three octave convolution layers are respectively the first octave convolution layer, the second octave convolution layer and the third octave convolution layer, and the convolution kernel size is 3×3;

[0074] Step 3.1.1: Input the full-color image P in step 2 into the convolution layer with a convolution kernel size of 3×3 for feature extraction, and obtain the initial feature map X with a dimension of h×w×c ini , h, w, c represent the height, width and number of channels of the feature map respectively;

[0075] Step 3.1.2: Initial feature map X ini Input the first octave convolution layer for convolution to obtain the first high-frequency feature map X with a dimension of h×w×(1-α)c H1 ; At the same time, the initial feature map X ini Perform average pooling and convolution in sequence to obtain a dimension of The first low-frequency feature map X L1 , where α represents the value assigned to the first low-frequency feature map X L1 The ratio of channels, α∈[0,1]

[0076] Step 3.1.3: The first high frequency feature map X H1 and the first low-frequency feature map X L1 Input the second octave convolution layer for same-frequency update and cross-frequency communication to obtain the second high-frequency feature map X H2 and the second low-frequency feature map X L2 ;

[0077] The specific process of the same frequency update is as follows:

[0078] For the first high-frequency feature map X H1 and the first low-frequency feature map X L1 Perform 3×3 convolution to obtain the updated first high-frequency feature map X′ H1 And the updated first low-frequency feature map X′ L1 ;

[0079] The specific process of the cross-frequency communication is as follows:

[0080] The first low-frequency feature map X L1 Perform 3×3 convolution and upsampling in sequence to obtain the first low-frequency feature map X after convolution and upsampling. L ″1, the first low-frequency feature map X after upsampling L ″1 and the updated first high-frequency feature map X′ H1 Add together to get the second high-frequency feature map X H2 ; At the same time, the first high-frequency feature map X H1 Perform average pooling and 3×3 convolution in sequence to obtain the first high-frequency feature map X″ after average pooling and convolution H1 , the first high-frequency feature map X″ H1 With the first low-frequency feature map X′ L1 Add together to get the second low-frequency feature map X L2 ;

[0081] Step 3.1.4: Transform the second high-frequency feature map X H2 and the second low-frequency feature map X L2 The input is respectively subjected to 3×3 convolution in the third octave convolution layer to obtain the second high-frequency feature map X′ after convolution. H2 and the second low-frequency feature map X′ L2 , and then the second low-frequency feature map X′ after convolution L2 Upsample to obtain the second low-frequency feature map X′ after upsampling L2U , and then the second high-frequency feature map X′ after convolution H2 and the second low-frequency feature map X′ L2U Add together to get the third high-frequency feature map X output by the third octave convolution layer H3 ;

[0082] Step 3.1.5: Take the first high-frequency feature map X output by the first octave convolutional layer H1 , the second high-frequency feature map X output by the second octave convolution layer H2 And the third high-frequency feature map X output by the third octave convolution layer H3 The input feature splicing layer performs channel dimension splicing of the feature map, and then the spliced ​​feature map is input into the convolution layer with a convolution kernel size of 1×1 for convolution to obtain the detail feature map F of the full-color image P. PD ;

[0083] Step 3.2: Input the hyperspectral image H in step 2 into the spatial and spectral separable three-dimensional convolution unit for multi-scale feature extraction to obtain the spatial-spectral feature map F of the hyperspectral image HS ; The spatial and spectrally separable three-dimensional convolution unit includes three spatial and spectrally separable three-dimensional convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; each spatial and spectrally separable three-dimensional convolution layer includes two three-dimensional convolution layers, two ReLU function activation layers and a feature shaping layer, the convolution kernels of the two three-dimensional convolution layers are k×k×1 and 1×1×k respectively, and the values ​​of k in the three spatial and spectrally separable three-dimensional convolution layers are 3, 5, and 7 respectively;

[0084] Step 3.2.1: Perform bicubic interpolation upsampling on the hyperspectral image H in step 2 so that the low-resolution hyperspectral image H has the same size as the panchromatic image P, and obtain the upsampled hyperspectral image

[0085] Step 3.2.2: Upsample the hyperspectral image in step 3.2.1 The images are input into three spatial and spectral separable 3D convolutional layers for simultaneous feature extraction to obtain the first hyperspectral image spatial-spectral feature map. Second hyperspectral image spatial-spectral feature map And the third hyperspectral image space-spectral feature map

[0086] The specific process of synchronous feature extraction is as follows:

[0087] The hyperspectral image after upsampling in step 3.2.1 When input to the first spatial and spectral separable 3D convolution layer, a 3D convolution layer with a convolution kernel of 3×3×1 is used for spatial convolution to obtain a hyperspectral image. Spatial feature map Use ReLU function to transform the spatial feature map Activate and obtain the activated spatial feature map The activated spatial feature map Input to the three-dimensional convolution layer with a convolution kernel of 1×1×3 for spectral convolution to obtain a hyperspectral image Spatial-spectral characteristics of Use ReLU function to transform the spatial-spectral feature map Activate and obtain the activated spatial-spectral feature map The dimension is h×w×d×c, where h, w, d are the height, width and depth of the feature map, and c is the number of channels. Input to the feature reshaping layer for feature reshaping operation, reshape from h×w×d×c dimension to h×w×(d×c) dimension, and obtain the first hyperspectral image space-spectral feature map

[0088] At the same time, the hyperspectral image upsampled in step 3.2.1 is When input into the second spatial and spectral separable three-dimensional convolution layer, it passes through a three-dimensional convolution layer with a convolution kernel of 5×5×1, a ReLU function activation layer, a three-dimensional convolution layer with a convolution kernel of 1×1×5, a ReLU function activation layer, and a feature shaping layer in sequence to obtain the second hyperspectral image spatial-spectral feature map

[0089] At the same time, the hyperspectral image upsampled in step 3.2.1 is When input into the third spatial and spectral separable three-dimensional convolution layer, it passes through the three-dimensional convolution layer with a convolution kernel of 7×7×1, the ReLU function activation layer, the three-dimensional convolution layer with a convolution kernel of 1×1×7, the ReLU function activation layer and the feature shaping layer in sequence to obtain the third hyperspectral image spatial-spectral feature map

[0090] Step 3.2.3: Transform the spatial-spectral feature map of the first hyperspectral image Second hyperspectral image spatial-spectral feature map And the third hyperspectral image space-spectral feature map Input the feature concatenation layer to concatenate the feature map channel dimensions and obtain the concatenated feature map F SC , and then the concatenated feature map F SC Input the convolution layer with a convolution kernel size of 1×1 to obtain the spatial-spectral feature map F of the hyperspectral image. HS ;

[0091] Step 3.3: Transform the detail feature map F of the full-color image P in step 3.1 PD and the spatial-spectral characteristic map F of the hyperspectral image in step 3.2 HS Input the adaptive feature modulation module to perform feature modulation and obtain the modulated feature map F M The adaptive feature modulation module includes four feature modulation blocks, each of which is composed of two adaptive feature modulation units, two convolutional layers and a ReLU function activation layer; each adaptive feature modulation unit includes a mapping function f M and two Sigmoid functions; the mapping function f M It consists of two parallel three-layer convolutions, the convolution kernel size is 1×1, and the three convolution layers are connected by two ReLU function activation layers;

[0092] Step 3.3.1: Transform the detail feature map F of the full-color image P PD The first adaptive feature modulation unit input to the first feature modulation block uses the mapping function f M The detail feature map F of the full-color image P PD Mapped into a set of intermediate parameters α′ and β′, α′ and β′ are respectively multiplied by the corresponding elements of α′ and β′ after Sigmoid function activation to obtain the scale parameter α and the translation parameter β;

[0093] Step 3.3.2: Transform the spatial-spectral feature map F of the hyperspectral image HS Input to the first adaptive feature modulation unit in the first feature modulation block, the spatial-spectral feature map F HS First, the corresponding elements are multiplied by the scale parameter α, and then the corresponding elements are added by the translation parameter β to obtain the modulated feature map F output by the first adaptive feature modulation unit. AFM1 ;

[0094] Step 3.3.3: The modulated feature map F AFM1 The convolutional layers with a convolution kernel size of 3×3 are input in sequence for convolution, and then the ReLU function activation layer is input for activation to obtain the feature map F′ AFM1 ; The detail feature map F of the full-color image P PD and feature map F′ AFM1 Input to the second adaptive feature modulation unit, repeat steps 3.3.1 and 3.3.2, and obtain the modulated feature map F output by the second adaptive feature modulation unit AFM2 ; The feature map F AFM2 Input the convolution layer with a convolution kernel size of 3×3 for convolution to obtain the feature map F′ AFM2 ; Skip connections are used in each feature modulation block to enhance cross-layer communication of features; the spatial-spectral feature map F of the hyperspectral image is transformed into HS and feature map F′ AFM2 Add the corresponding elements to obtain the modulated feature map F output by the first feature modulation block M1 ;

[0095] Step 3.3.4: Transform the detail feature map F of the full-color image P into PD And the modulated feature map F output by the first feature modulation block M1 Input to the second feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the second feature modulation block M2 ; The detail feature map F of the full-color image P PD The modulated feature map F output by the second feature modulation block M2Input to the third feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the third feature modulation block M3 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the third feature modulation block M3 Input to the fourth feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the fourth feature modulation block M4 ; Skip connections are used in the entire adaptive feature modulation module to enhance cross-layer communication of features; the spatial-spectral feature map F of the hyperspectral image is transformed into HS And the modulated feature map F output by the fourth feature modulation block M4 Add the corresponding elements to obtain the feature map F modulated by the adaptive feature modulation module M ;

[0096] Step 3.4: The feature map F modulated in step 3.2 M Input to the detail reconstruction module for detail reconstruction to obtain the reconstructed hyperspectral image details Hyperspectral image details Compared with the hyperspectral image after upsampling in step 2 Add the corresponding elements to obtain a high-resolution fused hyperspectral image

[0097] The detail reconstruction module consists of three convolutional layers, each with a convolution kernel size of 3×3, and the three convolutional layers are connected by two ReLU function activation layers;

[0098] Specifically, the feature map F modulated in step 3.2 M The data is sequentially input into the convolution layer with a convolution kernel size of 3×3 and the ReLU function activation layer for detail reconstruction to obtain the reconstructed hyperspectral image details.

[0099] Step 4: Input the hyperspectral training set in step 2 into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network constructed in step 3 for training until the loss function converges, thereby obtaining a trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0100] Specifically, the loss function L1 is used to minimize the fusion of hyperspectral images. and the reference image H G The error between the two is 10. -4 The Adam algorithm minimizes the loss function L1, and the batch size and number of training cycles are 16 and 2×10 respectively. 3 ;

[0101] The loss function L1 is defined as:

[0102]

[0103] Among them, N p is the number of training samples, j is the index of the training sample, To fuse hyperspectral images with high resolution, is the reference image;

[0104] Step 5: Input the hyperspectral test set in step 2 into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0105] A hyperspectral and panchromatic image fusion system based on an adaptive feature modulation network, comprising:

[0106] Image segmentation module: used to segment the original hyperspectral image of a certain pixel area to obtain several hyperspectral image blocks of the same size and non-overlapping;

[0107] Image preprocessing module: used to preprocess the hyperspectral image blocks to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P;

[0108] Model construction module: used to construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module;

[0109] Model training module: input the hyperspectral training set into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network for training, and obtain the trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network;

[0110] Result output module: The hyperspectral test set is input into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

[0111] A hyperspectral and panchromatic image fusion device based on an adaptive feature modulation network, comprising:

[0112] Memory: used for storing a computer program for implementing the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network;

[0113] Processor: used to implement the hyperspectral and panchromatic image fusion method based on the adaptive feature modulation network when executing the computer program.

[0114] A computer-readable storage medium stores a computer program, which, when executed by a processor, can implement a hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network.

[0115] Simulation experiment

[0116] In order to comprehensively evaluate the performance of the hyperspectral and panchromatic image fusion method based on adaptive feature modulation network proposed in this invention, the Washington DC Mall dataset is selected for experiments and analysis. The Washington DC Mall hyperspectral data is a scene of a shopping mall in Washington DC taken by the HYDICE sensor, containing 1280×307 pixels and 191 bands, covering the spectral region from 400 to 2500 nanometers. The image of 1200×240 pixels in the upper right corner is selected and divided into 20 non-overlapping hyperspectral image cubes of 120×120×191 in size. The preprocessing of step 1 is performed to obtain a low-resolution hyperspectral image with a spatial size of 30×30 and a high-resolution panchromatic image with a spatial size of 120×120.

[0117] The hyperspectral and panchromatic image fusion method based on adaptive feature modulation network (AFMN) proposed in the present invention is compared with six existing fusion methods. The six existing fusion methods include traditional component substitution (CS)-based and multi-resolution analysis (MRA)-based methods, namely, adaptive Schmidt orthogonal (GSA) method, principal component analysis (PCA) method, high-pass filtering (HPF) method, high-pass modulation (MGH) method based on generalized Laplacian pyramid of modulation transfer function, convolutional neural network-based methods, namely, detail injection-based convolutional neural network (DiCNN) method, and convolutional neural network (HyperPNN) method for hyperspectral and panchromatic image fusion. The quality of the fused image is quantified by the evaluation indicators of correlation coefficient (CC), spectral angle mapping (SAM), root mean square error (RMSE) and relative global synthesis error (ERGAS); the larger the value of correlation coefficient (CC), the better the quality of the fused image; the smaller the value of spectral angle mapping (SAM), the higher the fidelity of the spectral information of the fused image; the smaller the value of root mean square error (RMSE), the smaller the difference between the fused image and the reference image; the smaller the value of relative global synthesis error (ERGAS), the better the spatial and spectral quality of the fused image. The objective quantitative evaluation results of different fusion methods on the Washington DC Mall dataset are shown in Table 1, where the best results are highlighted in bold.

[0118] Table 1 Objective quantitative evaluation results of different fusion methods

[0119]

[0120] As can be seen from Table 1, the AFMN method proposed in the present invention achieves the best fusion performance, especially in terms of the evaluation results of the correlation coefficient (CC) and the relative global synthesis error (ERGAS). This fully proves that the AFMN method proposed in the present invention can effectively extract and inject the spatial detail information of the full-color image while better maintaining the spectral characteristics.

[0121] Figure 2 The fusion results of different methods on the 10th hyperspectral image cube (named "Washington_Patch10") in the Washington DC Mall dataset are shown in Figure 2. Figure 2 In each sub-image, a region marked by a green frame is enlarged to facilitate visual comparison. Figure 2 It can be observed that the fusion results generated by the GSA, PCA, HPF, MGH, DiCNN and HyperPNN methods in the prior art have obvious chromatic aberration and spatial distortion in the magnified area; compared with the above-mentioned fusion methods in the prior art, the AFMN method proposed in the present invention presents clearer local structural information and less spectral distortion in the magnified area.

[0122] Figure 3 The spectral angle mapping diagram between the fusion results of different methods on "Washington_Patch10" and the reference image; Figure 3 (a) The scale in the reference image represents the spectral angle mapping (SAM) value. The closer to the dark blue, the smaller the SAM value of the area. Figure 3 (h) It can be seen that the dark blue coverage area of ​​the spectral angle map obtained by the AFMN method is consistent with Figure 3 (a) is closest to the reference image, indicating that the SAM value of the fused image obtained by the AFMN method is the smallest. This conclusion is consistent with the objective evaluation results in Table 1, which further illustrates that the fusion method proposed in the present invention can better maintain the spectral characteristics.

Claims

1. A hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network, characterized in that: The steps include: Step 1: Obtain the original public hyperspectral image, segment the original hyperspectral image of a certain pixel area, and obtain several hyperspectral image blocks of the same size and non-overlapping; Step 2: Based on the Wald protocol, the original hyperspectral image block segmented in step 1 is used as the reference image, and the reference image is preprocessed to obtain a low-resolution hyperspectral image H and a high-resolution panchromatic image P respectively; And divide the preprocessed hyperspectral image blocks into a hyperspectral training set and a hyperspectral test set; Step 3: construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module; Step 4: Input the hyperspectral training set in step 2 into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network constructed in step 3 for training until the loss function converges, thereby obtaining a trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network; Step 5: Input the hyperspectral test set in step 2 into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

2. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 1, characterized in that: The pre-processing in step 2 is specifically as follows: The reference image is subjected to Gaussian blur filtering and downsampling in turn to obtain a low-resolution hyperspectral image H; meanwhile, the spectral bands of the visible light band of the reference image are averaged to obtain a high-resolution panchromatic image P.

3. The method for fusion of hyperspectral and panchromatic images based on adaptive feature modulation network according to claim 1, characterized in that: The construction process of the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network in step 3 is as follows: Step 3.1: Input the full-color image P obtained in step 2 into the octave convolution unit for multi-layer high-frequency detail feature extraction to obtain the detail feature map F of the full-color image P PD ; The octave convolution unit includes a convolution layer with a convolution kernel size of 3×3, three octave convolution layers, a feature splicing layer and a convolution layer with a convolution kernel size of 1×1; the three octave convolution layers are respectively the first octave convolution layer, the second octave convolution layer and the third octave convolution layer, and the convolution kernel size is 3×3; Step 3.2: Input the hyperspectral image H in step 2 into the spatial and spectral separable three-dimensional convolution unit for multi-scale feature extraction to obtain the spatial-spectral feature map F of the hyperspectral image HS ; The spatial and spectrally separable three-dimensional convolution unit includes three spatial and spectrally separable three-dimensional convolution layers, a feature concatenation layer and a convolution layer with a convolution kernel size of 1×1; each spatial and spectrally separable three-dimensional convolution layer includes two three-dimensional convolution layers, two ReLU function activation layers and a feature shaping layer, the convolution kernels of the two three-dimensional convolution layers are k×k×1 and 1×1×k respectively, and the values ​​of k in the three spatial and spectrally separable three-dimensional convolution layers are 3, 5 and 7 respectively; Step 3.3: Transform the detail feature map F of the full-color image P in step 3.1 PD and the spatial-spectral characteristic map F of the hyperspectral image in step 3.2 HS Input the adaptive feature modulation module to perform feature modulation and obtain the modulated feature map F M ; The adaptive feature modulation module includes four feature modulation blocks, each of which is composed of two adaptive feature modulation units, two convolution layers and a ReLU function activation layer; each adaptive feature modulation unit includes a mapping function f M and two Sigmoid functions; the mapping function f M It consists of two parallel three-layer convolutions, the convolution kernel size is 1×1, and the three convolution layers are connected by two ReLU function activation layers; Step 3.4: The feature map F modulated in step 3.2 M Input to the detail reconstruction module for detail reconstruction to obtain the reconstructed hyperspectral image details Hyperspectral image details Compared with the hyperspectral image after upsampling in step 2 Add the corresponding elements to obtain a high-resolution fused hyperspectral image The detail reconstruction module consists of three convolutional layers, the convolution kernel size is 3×3, and the three convolutional layers are connected by two ReLU function activation layers.

4. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 3, characterized in that: The process of extracting multi-layer high-frequency detail features in step 3.1 is as follows: Step 3.1.1: Input the full-color image P in step 2 into the convolution layer with a convolution kernel size of 3×3 for feature extraction, and obtain the initial feature map X with a dimension of h×w×c ini , h, w, c represent the height, width and number of channels of the feature map respectively; Step 3.1.2: Initial feature map X ini Input the first octave convolution layer for convolution to obtain the first high-frequency feature map X with a dimension of h×w×(1-α)c H1 ; At the same time, the initial feature map X ini Perform average pooling and convolution in sequence to obtain a dimension of The first low-frequency feature map X L1 , where α represents the value assigned to the first low-frequency feature map X L1 The ratio of channels, α∈[0,1]; Step 3.1.3: The first high frequency feature map X H1 and the first low-frequency feature map X L1 Input the second octave convolution layer for same-frequency update and cross-frequency communication to obtain the second high-frequency feature map X H2 and the second low-frequency feature map X L2 ; Step 3.1.4: Transform the second high-frequency feature map X H2 and the second low-frequency feature map X L2 The input is respectively subjected to 3×3 convolution in the third octave convolution layer to obtain the second high-frequency feature map X′ after convolution. H2 and the second low-frequency feature map X L ′2, then the second low-frequency feature map X after convolution L ′2 is upsampled to obtain the second low-frequency feature map X after upsampling L ' 2U , and then the second high-frequency feature map X′ after convolution H2 With the second low frequency feature map X L ' 2U Add together to get the third high-frequency feature map X output by the third octave convolution layer H3 ; Step 3.1.5: Take the first high-frequency feature map X output by the first octave convolutional layer H1 , the second high-frequency feature map X output by the second octave convolution layer H2 And the third high-frequency feature map X output by the third octave convolution layer H3 The input feature splicing layer performs channel dimension splicing of the feature map, and then the spliced ​​feature map is input into the convolution layer with a convolution kernel size of 1×1 for convolution to obtain the detail feature map F of the full-color image P. PD .

5. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 4, characterized in that: The specific process of the same frequency update is as follows: For the first high-frequency feature map X H1 and the first low-frequency feature map X L1 Perform 3×3 convolution to obtain the updated first high-frequency feature map X′ H1 And the updated first low-frequency feature map X L '1; The specific process of the cross-frequency communication is as follows: The first low-frequency feature map X L1 Perform 3×3 convolution and upsampling in sequence to obtain the first low-frequency feature map X after convolution and upsampling. L ″1, the first low-frequency feature map X after upsampling L ″1 and the updated first high-frequency feature map X′ H1 Add together to get the second high-frequency feature map X H2 ; At the same time, the first high-frequency feature map X H1 Perform average pooling and 3×3 convolution in sequence to obtain the first high-frequency feature map X″ after average pooling and convolution H1 , the first high-frequency feature map X″ H1 With the first low-frequency feature map X′ L1 Add together to get the second low-frequency feature map X L2 .

6. The method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network according to claim 3, characterized in that: The process of multi-scale feature extraction in step 3.2 is as follows: Step 3.2.1: Perform bicubic interpolation upsampling on the hyperspectral image H in step 2 so that the low-resolution hyperspectral image H has the same size as the panchromatic image P, and obtain the upsampled hyperspectral image Step 3.2.2: Upsample the hyperspectral image in step 3.2.1 The images are input into three spatial and spectral separable 3D convolutional layers for synchronous feature extraction to obtain the first hyperspectral image spatial-spectral feature map. Second hyperspectral image spatial-spectral feature map And the third hyperspectral image space-spectral feature map The process of feature extraction for each spatially and spectrally separable 3D convolutional layer is as follows: The hyperspectral image after upsampling in step 3.2.1 The input convolution kernel is k×k×1, and the three-dimensional convolution layer is used for spatial convolution to obtain the hyperspectral image. Spatial feature map Use ReLU function to transform the spatial feature map Activate and obtain the activated spatial feature map The activated spatial feature map Input to the three-dimensional convolution layer with a convolution kernel of 1×1×k for spectral convolution to obtain the spatial-spectral feature map of the hyperspectral image H~ Use ReLU function to transform the spatial-spectral feature map Activate and obtain the activated spatial-spectral feature map The dimension is h×w×d×c, where h, w, d are the height, width and depth of the feature map, and c is the number of channels. The input is sent to the feature shaping layer for feature shaping operation, and the h×w×d×c dimension is reshaped into the h×w×(d×c) dimension to obtain the hyperspectral image space-spectral feature map Step 3.2.3: Transform the spatial-spectral feature map of the first hyperspectral image Second hyperspectral image spatial-spectral feature map And the third hyperspectral image space-spectral feature map Input the feature concatenation layer to concatenate the feature map channel dimensions and obtain the concatenated feature map F SC , and then the concatenated feature map F SC Input the convolution layer with a convolution kernel size of 1×1 to obtain the spatial-spectral feature map F of the hyperspectral image. HS .

7. The method for fusion of hyperspectral and panchromatic images based on adaptive feature modulation network according to claim 3, characterized in that: The process of feature modulation in step 3.3 is as follows: Step 3.3.1: Transform the detail feature map F of the full-color image P into PD The first adaptive feature modulation unit input to the first feature modulation block uses the mapping function f M The detail feature map F of the full-color image P PD Mapped into a set of intermediate parameters α′ and β′, α′ and β′ are respectively multiplied by the corresponding elements of α′ and β′ after Sigmoid function activation to obtain the scale parameter α and the translation parameter β; Step 3.3.2: Transform the spatial-spectral feature map F of the hyperspectral image HS Input to the first adaptive feature modulation unit in the first feature modulation block, the spatial-spectral feature map F HS First, the corresponding elements are multiplied by the scale parameter α, and then the corresponding elements are added by the translation parameter β to obtain the modulated feature map F output by the first adaptive feature modulation unit. AFM1 ; Step 3.3.3: The modulated feature map F AFM1 The convolutional layers with a convolution kernel size of 3×3 are input in sequence for convolution, and then the ReLU function activation layer is input for activation to obtain the feature map F. A ' FM1 ; The detail feature map F of the full-color image P PD and feature map F A ' FM1 Input to the second adaptive feature modulation unit, repeat steps 3.3.1 and 3.3.2, and obtain the modulated feature map F output by the second adaptive feature modulation unit AFM2 ; The feature map F AFM2 Input the convolution layer with a convolution kernel size of 3×3 for convolution to obtain the feature map F A ' FM2 ; Skip connections are used in each feature modulation block; the spatial-spectral feature map F of the hyperspectral image is transformed HS and feature map F A ' FM2 Add the corresponding elements to obtain the modulated feature map F output by the first feature modulation block M1 ; Step 3.3.4: Transform the detail feature map F of the full-color image P PD And the modulated feature map F output by the first feature modulation block M1 Input to the second feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the second feature modulation block M2 ; The detail feature map F of the full-color image P PD The modulated feature map F output by the second feature modulation block M2 Input to the third feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the third feature modulation block M3 ; The detail feature map F of the full-color image P PD And the modulated feature map F output by the third feature modulation block M3 Input to the fourth feature modulation block, repeat steps 3.3.1, 3.3.2 and 3.3.3, and obtain the modulated feature map F output by the fourth feature modulation block M4 ; Skip connections are used in the entire adaptive feature modulation module; the spatial-spectral feature map F of the hyperspectral image HS And the modulated feature map F output by the fourth feature modulation block M4 Add the corresponding elements to obtain the feature map F modulated by the adaptive feature modulation module M .

8. A hyperspectral and panchromatic image fusion system based on an adaptive feature modulation network, characterized in that: include: Image segmentation module: used to segment the original hyperspectral image of a certain pixel area to obtain several hyperspectral image blocks of the same size and non-overlapping; Image preprocessing module: used to preprocess the hyperspectral image blocks to obtain low-resolution hyperspectral images H and high-resolution panchromatic images P; Model construction module: used to construct a hyperspectral and panchromatic image fusion model based on an adaptive feature modulation network; the hyperspectral and panchromatic image fusion model includes an octave convolution unit, a spatially and spectrally separable three-dimensional convolution unit, an adaptive feature modulation module, and a detail reconstruction module; Model training module: input the hyperspectral training set into the hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network for training, and obtain the trained hyperspectral and panchromatic image fusion model based on the adaptive feature modulation network; Result output module: The hyperspectral test set is input into the trained hyperspectral and panchromatic image fusion model based on adaptive feature modulation network for testing to obtain a high-resolution fused hyperspectral image.

9. A hyperspectral and panchromatic image fusion device based on an adaptive feature modulation network, characterized in that: include: Memory: used to store a computer program for implementing the hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network according to claims 1 to 7; Processor: used to implement the hyperspectral and panchromatic image fusion method based on an adaptive feature modulation network of claims 1-7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement claims 1-7, a method for fusion of hyperspectral and panchromatic images based on an adaptive feature modulation network.

Citation Information

Patent Citations

  • Facial expression recognition method and device

    CN114612958A

  • Text recognition method and device, storage medium and electronic equipment

    CN116508075A

  • Method and system for normalizing smoothing feature of time space for behavior recognition

    JP2021103519A