Hyperspectral image fusion method, system and device based on depth prior and double-attention residual network, and medium

Through the deep prior and double attention residual network methods, hyperspectral images are upsampled and feature extraction, which solves the problems of poor upsampling effect and inflexible feature processing in the prior art, and achieves high precision Spectral image fusion.

CN118967476APending Publication Date: 2024-11-15XIDIAN UNIV

Patent Information

Application Number
CN202411037939.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing hyperspectral image fusion technology cannot effectively restore detailed information during the upsampling process, and lacks discrimination and flexibility for different types of features, resulting in limited fusion performance.

Method used

Using a method based on depth prior and double attention residual network, low-resolution hyperspectral images are upsampled, and features are extracted adaptively through deep hyperspectral prior algorithm and channel and space dual attention residual network to improve fusion The representation and discriminatory ability of the network.

Benefits of technology

Without the need for a large amount of data training, the spatial resolution of the hyperspectral image is improved, the spectral information is maintained, and the hyperspectral image fusion is achieved with higher accuracy, reducing the dependence on data volume.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967476A_ABST
    Figure CN118967476A_ABST
Patent Text Reader

Abstract

A hyperspectral image fusion method, system, device and medium based on depth prior and double-attention residual network, the method comprises the following steps: firstly obtaining an image data set and preprocessing to obtain a low-resolution hyperspectral image and a panchromatic image, then performing up-sampling on the low-resolution hyperspectral image through a depth hyperspectral prior algorithm to obtain a full-color image; then splicing the panchromatic image and the up-sampling hyperspectral image in a spectral dimension to obtain an input image, and inputting the input image into a channel and space double-attention residual network for reconstruction to obtain a residual image; and finally, carrying out addition operation on the up-sampling hyperspectral image and the reconstructed residual image to obtain a fused hyperspectral image. The system, the device and the medium are used for realizing the hyperspectral image fusion method based on the depth prior and the double-attention residual network. According to the method, the dependence on the data volume is reduced, and meanwhile, the hyperspectral image fusion precision is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a hyperspectral image fusion method, system, device and medium based on deep prior and dual attention residual network. Background Art

[0002] Due to the trade-off of the imaging system, the spatial resolution and spectral resolution of remote sensing images restrict each other. Compared with extremely high spectral resolution, the spatial resolution of hyperspectral images is generally low. Using hyperspectral image fusion technology, the hyperspectral image and the panchromatic image with high spatial resolution of the same scene are fused to obtain a fused image with both high spectral and high spatial resolution, which is an effective means to improve the spatial resolution of hyperspectral images. In recent years, with the increasing demand for high-resolution hyperspectral images in various remote sensing tasks such as environmental monitoring, agricultural production, and military reconnaissance, hyperspectral image fusion technology has received more and more attention. In addition, as remote sensing enters the era of big data, multi-source remote sensing image data such as panchromatic, multispectral and hyperspectral have exploded, and hyperspectral image fusion is facing new challenges and opportunities. The development of more intelligent fusion methods has become an inevitable trend.

[0003] Commonly used hyperspectral image fusion methods can be mainly divided into two categories: traditional fusion methods and deep learning-based fusion methods.

[0004] Traditional hyperspectral image fusion methods mainly include: component replacement method, multi-resolution analysis method, and model optimization-based method.

[0005] 1) The component substitution (CS) method first uses a specific transformation to map the hyperspectral image to a new space, and then divides it into a spatial structure component and a spectral information component in this transformation domain. The spatial information component is then replaced with a full-color image and then inversely transformed to obtain a fused high-resolution image. The spectral fidelity of the fusion result obtained by the component substitution method is often poor.

[0006] 2) The multiresolution analysis (MRA) method first obtains the high-frequency spatial detail information of the panchromatic image through multiresolution decomposition, and then injects it into the upsampled hyperspectral image according to certain rules. Typical fusion methods in this category include the high-pass filter (HPF) method and the modulation transfer function-based generalized Laplacian pyramid (MTF-GLP) method. Methods based on multiresolution analysis usually have the problem of spatial information loss, and the implementation process is relatively complicated.

[0007] 3) Model-based Optimization (MBO) methods regard fusion as solving an inverse optimization problem, which mainly relies on the observation model between the ideal high-resolution hyperspectral image and the actual low-resolution hyperspectral image and high-resolution panchromatic image to construct and solve the energy functional. Representative methods include the non-negative matrix factorization (NMF) method and the coupled non-negative matrix factorization (CNMF) method. The fusion performance of the model-based optimization method is limited by the representation ability of the model and is highly dependent on prior constraints.

[0008] With the continuous development of artificial intelligence technology, the method based on deep learning (DL) is a rapidly emerging remote sensing image fusion technology in recent years, showing excellent fusion performance and strong application potential, and is receiving more and more attention. This is because deep learning technology can automatically learn the multi-level features of remote sensing image data and effectively model the complex mapping relationship between the fused image and the high-resolution remote sensing image.

[0009] In recent years, a large number of remote sensing image fusion methods based on Convolutional Neural Network (CNN) and Residual Network (ResNet) have been proposed, showing better performance than traditional fusion methods.

[0010] K. Li et al. proposed a detail-based deep Laplacian hyperspectral and panchromatic image fusion method DDLPS in the paper DDLPS: Detail-based Deep Laplacian Pansharpening for Hyperspectral Imagery. (K. Li. DDLPS: Detail-based Deep Laplacian Pansharpening for Hyperspectral Imagery [J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57 (10): 8011-8025.) However, the DDLPS method uses the LapSRN algorithm to upsample low-resolution hyperspectral images. The upsampling performance is highly dependent on the ability of the super-resolution network to learn priors from a large number of RGB image data sets. In addition, the LapSRN algorithm upsamples each band of the low-resolution hyperspectral image in turn, ignoring the strong correlation between the hyperspectral image bands, resulting in spectral distortion in the upsampling results.

[0011] The Chinese patent application with publication number CN112634137A discloses a method for fusion of hyperspectral and panchromatic images based on AE to extract multi-scale spatial spectrum features. However, the invention only uses the channel attention mechanism to enhance the extracted features, ignoring the features in the spatial position that are more useful for enhancing the detailed information of the fused image, resulting in limited fusion performance.

[0012] In summary, the shortcomings of the prior art are summarized as follows:

[0013] 1) Existing hyperspectral image fusion technology mainly uses image interpolation or large-scale data training strategies to upsample low-resolution hyperspectral images. However, the interpolation process cannot effectively restore the missing detail information. The effect of large-scale data training depends largely on the amount of data and the learning process. The quality of the upsampling results needs to be improved.

[0014] 2) Existing hyperspectral image fusion technologies often regard the fusion process as a black box problem, lacking the discriminability and flexibility of processing different types of features, and are unable to fully extract and utilize the complementary information of hyperspectral and panchromatic images, resulting in limited fusion performance. Summary of the invention

[0015] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a hyperspectral image fusion method, system, device and medium based on deep prior and dual attention residual network. By upsampling low-resolution hyperspectral images through a deep hyperspectral prior algorithm, the spatial resolution of hyperspectral images can be effectively improved without the need for a large amount of data training, and the spectral information can be better maintained to provide high-quality upsampled hyperspectral images for the fusion network; the constructed channel and spatial dual attention residual network is used to adaptively and simultaneously extract the features in the spectral channel and spatial position that are more critical to improving the fusion performance, effectively improving the representation and discrimination capabilities of the fusion network, and achieving higher-precision hyperspectral image fusion.

[0016] In order to achieve the above object, the technical solution adopted by the present invention is:

[0017] A hyperspectral image fusion method based on deep prior and dual attention residual network includes the following steps:

[0018] Step 1: Obtain an image dataset and preprocess the data to obtain a low-resolution hyperspectral image H lr and a full-color image P;

[0019] Step 2: The low-resolution hyperspectral image H obtained in step 1 is processed by a deep hyperspectral prior algorithm. lr Upsample to obtain the upsampled hyperspectral image H up ;

[0020] Step 3: The full-color image P obtained in step 1 and the upsampled hyperspectral image H obtained in step 2 are up The input image H is obtained by splicing in the spectral dimension in , and then the input image H in Input into the channel and space dual attention residual network for reconstruction to obtain the reconstructed residual image

[0021] Step 4: Upsample the hyperspectral image H obtained in step 2 up And the reconstructed residual image obtained in step 3 Perform sum operation to obtain the fused hyperspectral image H fus .

[0022] The step 1 specifically comprises the following steps:

[0023] Step 1.1: Obtain the original hyperspectral image from the public dataset as the reference image H ref ;

[0024] Step 1.2, according to the Wald protocol, by using the reference image H obtained in step 1.1 refUse Gaussian blur and downsample by 4 times to get a low-resolution hyperspectral image H lr , for the reference image H obtained in step 1.1 ref The spectral bands in the visible light range are averaged to obtain a high-resolution full-color image P.

[0025] The step 2 specifically includes the following steps:

[0026] The low-resolution hyperspectral image H obtained in step 1 lr , the U-Net network architecture of the deep hyperspectral prior algorithm is upsampled to the same spatial size as the panchromatic image P, and a higher quality upsampled hyperspectral image H is obtained. up :

[0027] The U-Net network architecture includes five downsampling units d[p], five upsampling units u[p] and five jump connection units s[p], where p = 1, 2, ..., 5. The U-Net network takes a tensor Z filled with uniform noise as input. The spatial dimension of the tensor Z is consistent with the full-color image P, and the spectral dimension of the tensor Z is consistent with the low-resolution hyperspectral image H. lr Consistent, the U-Net network output is the upsampled hyperspectral image H up ;

[0028] Each of the downsampling units d contains a Lanczos2 downsampling module, batch normalization (BatchNormalization, BN), a LeakyReLU activation function and a convolution layer with a convolution kernel of 3×3, as an encoder for extracting effective features;

[0029] Each of the upsampling units u consists of a bilinear upsampling module and two convolutional layers, has the same convolution kernel size as the downsampling unit d, and acts as a decoder to reconstruct the image;

[0030] The jump connection unit s is added between each of the downsampling units d and each of the upsampling units u, and each of the jump connection units s includes a convolution layer with a convolution kernel of 1×1, batch normalization (Batch Normalization, BN) and a LeakyReLU activation function.

[0031] The deep hyperspectral prior algorithm includes upsampling the hyperspectral image H up Do downsampling to get the upsampled hyperspectral image after downsampling The L1 loss function is used to convert the downsampled upsampled hyperspectral image and the low-resolution hyperspectral image H obtained in step 1 lr Minimize the error between them.

[0032] The step 3 specifically comprises the following steps:

[0033] Step 3.1: Upsample the hyperspectral image H obtained in step 2 up After concatenating the full-color image P obtained in step 1 in the spectral dimension, we get the input image H of the channel and spatial dual attention residual network. in , input image H in After passing through a feature extraction network consisting of a convolutional layer with a convolution kernel of 3×3, the initial features are extracted:

[0034] F0=F FEN (H in ),

[0035] Among them, F FEN (·) represents the feature extraction network, and F0 represents the initial feature extracted;

[0036] Step 3.2, the initial feature F0 extracted in step 3.1 is fed to the feature mapping network, which stacks N channels and spatial dual attention residual blocks to enhance the representation ability of the network:

[0037] F N =f FMN (F0) = f R,N (f R,N-1 (…f R,1 (F0)…)),

[0038] Among them, F N represents the output features of the Nth channel and spatial dual attention residual block in the feature mapping network, f FMN (·) represents the feature mapping network, f R,N (·) represents the Nth channel and spatial dual attention residual block; the channel and spatial dual attention residual block integrates the channel attention (CA) module and the spatial attention (SA) module into the residual block to adaptively rescale the feature map; the input feature of the Nth channel and spatial dual attention residual block is F N-1 , input feature F N-1First, a more refined feature U is generated through two convolutional layers with a kernel size of 3×3 connected by a ReLU activation function. Then, the feature U is fed to the channel attention (CA) module and the spatial attention (SA) module for attention weighting. Then, the input features of the Nth channel and spatial dual attention residual block, the features output by the channel attention (CA) module, and the features output by the spatial attention (SA) module are summed.

[0039] When the feature U is fed to the Channel Attention (CA) module:

[0040] Step 3.2.1, let U = [u1,u2,…,u C ] represents the feature of the input channel attention (CA) module. The feature U contains C feature maps with a spatial size of H×W, where H and W represent the height and width of the feature map, respectively. The global average pooling layer is used to obtain the vector v∈R C , the c-th element of vector v is:

[0041]

[0042] Among them, u c (i,j) is the cth channel u c The value at position (i,j) in the c is the cth element of the vector v obtained by the global average pooling layer of feature U;

[0043] Step 3.2.2, feed the vector v obtained in step 3.2.1 to two convolutional layers with kernel size of 1×1, apply ReLU activation function after the first convolutional layer, and apply Sigmoid activation function after the second convolutional layer to obtain the channel attention mask M CA :

[0044] M CA =σ(Ω CA,2 (δ(Ω CA,1 v))),

[0045] Among them, δ(·) represents the ReLU activation function, σ(·) represents the Sigmoid activation function, and Ω CA,1 and Ω CA,2are the weights of the first and second convolutional layers in the channel attention (CA) module, M CA is the obtained channel attention mask, vector v∈R C The number of channels is reduced in the first convolutional layer by a reduction rate r, and the number of channels is reduced to After being activated by the ReLU activation function, the second convolutional layer increases to C channels;

[0046] Step 3.2.3: Add the channel attention mask M obtained in step 3.2.2 CA Multiply the corresponding elements with the feature U fed to the Channel Attention (CA) module to obtain the feature U output by the Channel Attention (CA) module CA :

[0047]

[0048] in, Indicates the multiplication of corresponding elements, U CA Features output by the Channel Attention (CA) module;

[0049] When the feature U is fed into the Spatial Attention (SA) module:

[0050] Step 3.2.4, let U = [u 1,1 ,u 1,2 ,…,u i,j ,…,u H,W ] represents the input feature of the SpatialAttention (SA) module, u i,j ∈R C Represents the feature vector at the spatial position (i, j) of the feature map. H and W represent the height and width of the feature map respectively. A convolution layer with a convolution kernel size of 1×1 is used to convolve the feature U. The convolved feature is activated using the Sigmoid activation function to obtain the spatial attention mask M. SA :

[0051] M SA =σ(Ω SA U),

[0052] Among them, Ω SA is the weight of the convolutional layer in the Spatial Attention (SA) module, σ(·) represents the Sigmoid activation function, and M SA is the obtained spatial attention mask;

[0053] Step 3.2.5: The spatial attention mask M obtained in step 3.2.4 is SA Multiply the corresponding elements with the feature U fed to the Spatial Attention (SA) module to obtain the feature U output by the Spatial Attention (SA) module SA :

[0054]

[0055] in, Indicates the multiplication of corresponding elements, U SA Features output by the Spatial Attention (SA) module;

[0056] In step 3.2.6, a skip connection is used in each channel and spatial dual attention residual block to add the input features of the Nth channel and spatial dual attention residual block, the features of the output of the channel attention (CA) module obtained in step 3.2.3, and the features of the output of the spatial attention (SA) module obtained in step 3.2.5 to obtain the output features of the Nth channel and spatial dual attention residual block:

[0057] F N =F N-1 +U CA +U SA ,

[0058] Among them, F N-1 represents the input features of the Nth channel and spatial dual attention residual block, F N represents the output features of the Nth channel and spatial dual attention residual block;

[0059] Step 3.3: Map the output features F of the Nth channel and spatial dual attention residual block in the feature mapping network obtained in step 3.2 N The residual reconstruction network is fed into the residual reconstruction network, which uses a convolution layer with a convolution kernel size of 3×3 to reconstruct the residual image.

[0060]

[0061] Among them, f RRN (·) represents the residual reconstruction network, is the reconstructed residual image.

[0062] The channel and spatial dual attention residual network includes an L1 loss function, which is defined as:

[0063]

[0064] Among them, l is the number of training samples, k is the index of training samples, H res Denotes the reference image H ref With the upsampled image H up The residual image between them is obtained by optimizing the channel and spatial dual attention residual network using the L1 loss function to minimize the reconstructed residual image. and the residual image H res The error between .

[0065] The upsampled hyperspectral image H is up and the reconstructed residual image Perform sum operation to obtain the fused hyperspectral image H fus :

[0066]

[0067] Among them, H fus Represents the obtained fused hyperspectral image.

[0068] A system for hyperspectral image fusion method based on deep prior and dual attention residual network, comprising:

[0069] The preprocessing module obtains the image data set and preprocesses the data to obtain a low-resolution hyperspectral image H lr and a full-color image P;

[0070] The upsampling module uses a deep hyperspectral prior algorithm to upsample the low-resolution hyperspectral image H lr Upsample to obtain the upsampled hyperspectral image H up ;

[0071] The reconstruction module combines the panchromatic image P and the upsampled hyperspectral image H up The input image H is obtained by splicing in the spectral dimension in , and then the input image H in Input into the channel and space dual attention residual network for reconstruction to obtain the reconstructed residual image

[0072] Fusion module, upsampled hyperspectral image H up and the reconstructed residual image Perform sum operation to obtain the fused hyperspectral image H fus .

[0073] A hyperspectral image fusion device based on deep prior and dual attention residual network, comprising:

[0074] Memory: used to store a computer program for implementing the hyperspectral image fusion method based on deep prior and dual attention residual network;

[0075] Processor: used to implement the hyperspectral image fusion method based on deep prior and dual attention residual network when executing the computer program.

[0076] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a hyperspectral image fusion method based on deep prior and dual attention residual network.

[0077] Compared with the prior art, the beneficial effects of the present invention are:

[0078] 1. The present invention proposes a method for upsampling low-resolution hyperspectral images using a deep hyperspectral prior algorithm, which can generate super-resolution results with better preserved spectral information without the need for a large amount of data training, and provide higher quality upsampled hyperspectral images for the image fusion network.

[0079] 2. The present invention proposes a method for reconstructing residual images based on a channel and space dual attention residual network. The constructed channel and space dual attention residual network adaptively and simultaneously extracts features in spectral channels and spatial positions that are more useful for improving reconstruction and fusion performance, thereby effectively improving the representation and discrimination capabilities of the fusion network and achieving higher-precision hyperspectral image fusion.

[0080] In summary, the present invention reduces the dependence on data volume while improving the accuracy of hyperspectral image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is a flow chart of the hyperspectral image fusion method based on deep prior and dual attention residual network described in the present invention.

[0082] Figure 2 Schematic diagram of the U-Net architecture of the deep hyperspectral prior algorithm adopted in the present invention.

[0083] Figure 3 1 is a comparison diagram of upsampling results of different methods on the Pavia Center dataset “PaviaCenter_Patch11” image in the embodiment of the present invention, where (a) is the reference image H ref ; (b) is the upsampling result diagram of the Bicubic method; (c) is the upsampling result diagram of the LSE method; (d) is the upsampling result diagram of the TSE method; (e) is the upsampling result diagram of the LapSRN method; (f) is the upsampling result diagram of the DHP method proposed in the present invention.

[0084] Figure 4 The upsampling results of different methods on “PaviaCenter_Patch11” in the embodiments of the present invention and the reference image H ref Spectral angle mapping between , where (a) is the reference spectral angle mapping; (b) is the upsampling result of the Bicubic method and the reference image H ref (c) The upsampling result of the LSE method and the reference image H ref Spectral angle mapping between the two images; (d) The upsampling result of the TSE method and the reference image H ref (e) The spectral angle mapping between the LapSRN method and the reference image H ref (f) is the spectral angle mapping between the upsampling result of the DHP method proposed in the present invention and the reference image H ref Spectral angle mapping between.

[0085] Figure 5 Schematic diagram of the structure of the channel and spatial dual attention residual block used in the present invention.

[0086] Figure 6 This is a comparison of the fusion results of the Pavia Center dataset “PaviaCenter_Patch21” image using different methods in the simulation experiment of the present invention, where (a) is the reference image H ref ; (b) is the fusion result diagram of the GSA method; (c) is the fusion result diagram of the PCA method; (d) is the fusion result diagram of the SFIM method; (e) is the fusion result diagram of the MGH method; (f) is the fusion result diagram of the GFPCA method; (g) is the fusion result diagram of the BNA method; (h) is the fusion result diagram of the BSP method; (i) is the fusion result diagram of the CNMF method; (j) is the fusion result diagram of the DDLPS method; (k) is the fusion result diagram of the DRCNN method; (l) is the fusion result diagram of the DHP-DARN method proposed in the present invention.

[0087] Figure 7 The difference curves between the spectra at four pixel positions and the reference image spectra are obtained by using different methods in the simulation experiment of the present invention on "PaviaCenter_Patch21", where (a) is at the (10,72) pixel position, (b) is at the (56,52) pixel position, (c) is at the (80,108) pixel position, and (d) is at the (105,38) pixel position.

[0088] Figure 8 This is the fusion result of different methods on the Botswana dataset “Botswana_Patch15” image in the simulation experiment of the present invention, where (a) is the reference image Href ; (b) is the fusion result diagram of the GSA method; (c) is the fusion result diagram of the PCA method; (d) is the fusion result diagram of the SFIM method; (e) is the fusion result diagram of the MGH method; (f) is the fusion result diagram of the GFPCA method; (g) is the fusion result diagram of the BNA method; (h) is the fusion result diagram of the BSP method; (i) is the fusion result diagram of the CNMF method; (j) is the fusion result diagram of the DDLPS method; (k) is the fusion result diagram of the DRCNN method; (l) is the fusion result diagram of the DHP-DARN method proposed in the present invention.

[0089] Fig. 9 The fusion results of different methods on "Botswana_Patch15" in the simulation experiment of the present invention are compared with the reference image H ref The error image between them, where (a) is the reference error image; (b) is the fusion result of the GSA method and the reference image H ref (c) is the error diagram between the fusion result of the PCA method and the reference image H ref (d) is the error diagram between the fusion result of SFIM method and the reference image H ref (e) is the error diagram between the fusion result of the MGH method and the reference image H ref (f) is the error diagram between the fusion result of GFPCA method and the reference image H ref (g) is the error diagram between the fusion result of the BNA method and the reference image H ref (h) is the error diagram between the fusion result of the BSP method and the reference image H ref The error diagram between them; (i) is the fusion result of CNMF method and the reference image H ref (j) is the error diagram between the fusion result of DDLPS method and the reference image H ref (k) is the error diagram between the fusion result of DRCNN method and the reference image H ref (l) is the error diagram between the fusion result of the DHP-DARN method proposed in the present invention and the reference image H ref The error diagram between . DETAILED DESCRIPTION

[0090] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0091] like Figure 1 As shown, a hyperspectral image fusion method based on deep prior and dual attention residual network includes the following steps:

[0092] Step 1: Obtain an image dataset and preprocess the data to obtain a low-resolution hyperspectral image H lr and a full-color image P;

[0093] Step 2: The low-resolution hyperspectral image H obtained in step 1 is processed by a deep hyperspectral prior algorithm. lr Upsample to obtain the upsampled hyperspectral image H up ;

[0094] Step 3: The full-color image P obtained in step 1 and the upsampled hyperspectral image H obtained in step 2 are up The input image H is obtained by splicing in the spectral dimension in , and then the input image H in Input into the channel and space dual attention residual network for reconstruction to obtain the reconstructed residual image

[0095] Step 4: Upsample the hyperspectral image H obtained in step 2 up And the reconstructed residual image obtained in step 3 Perform sum operation to obtain the fused hyperspectral image H fus .

[0096] The step 1 specifically comprises the following steps:

[0097] Step 1.1: Obtain the original hyperspectral image from the public dataset as the reference image H ref ;

[0098] Step 1.2, according to the Wald protocol, by using the reference image H obtained in step 1.1 ref Use Gaussian blur and downsample by 4 times to get a low-resolution hyperspectral image H lr , for the reference image H obtained in step 1.1 ref The spectral bands in the visible light range are averaged to obtain a high-resolution full-color image P.

[0099] The step 2 specifically includes the following steps:

[0100] The low-resolution hyperspectral image H obtained in step 1 lr , upsampled to the same spatial size as the full-color image P through the U-Net network architecture of the deep hyperspectral prior algorithm, such as Figure 2 As shown, a higher quality upsampled hyperspectral image H is obtained. up :

[0101] The U-Net network architecture includes five downsampling units d[p], five upsampling units u[p] and five jump connection units s[p], where p = 1, 2, ..., 5. The U-Net network takes a tensor Z filled with uniform noise as input. The spatial dimension of the tensor Z is consistent with the full-color image P, and the spectral dimension of the tensor Z is consistent with the low-resolution hyperspectral image H. lr Consistent, the U-Net network output is the upsampled hyperspectral image H up ;

[0102] Each of the downsampling units d contains a Lanczos2 downsampling module, batch normalization (BatchNormalization, BN), a LeakyReLU activation function and a convolution layer with a convolution kernel of 3×3, as an encoder for extracting effective features;

[0103] Each of the upsampling units u consists of a bilinear upsampling module and two convolutional layers, has the same convolution kernel size as the downsampling unit d, and acts as a decoder to reconstruct the image;

[0104] In order to alleviate the gradient vanishing problem and improve the reconstruction accuracy, the jump connection unit s is added between each of the downsampling units d and each of the upsampling units u, and each of the jump connection units s includes a convolution layer with a convolution kernel of 1×1, batch normalization (Batch Normalization, BN) and a LeakyReLU activation function.

[0105] The deep hyperspectral prior algorithm includes upsampling the hyperspectral image H up Do downsampling to get the upsampled hyperspectral image after downsampling The L1 loss function is used to convert the downsampled upsampled hyperspectral image and the low-resolution hyperspectral image H obtained in step 1 lr Minimize the error between them.

[0106] Different from the method using L2 loss function in the previous deep image prior (DIP) algorithm, the present invention adopts L1 loss function to minimize and low-resolution hyperspectral image H lrThe error between the two, the performance of the DHP upsampling method trained with the L2 loss function (denoted as DHP-L2) and the DHP upsampling method trained with the L1 loss function (denoted as DHP-L1) are tested on the Pavia Center dataset, and the corresponding objective quantitative evaluation results are listed in Table 1, where the best value is highlighted in bold. The hyperspectral image upsampled by the DHP-L1 method has better evaluation results in terms of the cross correlation coefficient (CC), spectral angle mapping (SAM), root mean square error (RMSE) and relative global synthesis error (ERGAS), especially in terms of spectral angle mapping (SAM), indicating that the DHP-L1 method proposed in the present invention has better spectral preservation ability;

[0107] Table 1 Objective quantitative evaluation results of DHP-L2 and DHP-L1 methods on the Pavia Center dataset

[0108]

[0109] Step 2.3, verify the superiority of the DHP upsampling method in terms of spatial and spectral quality:

[0110] Experimental tests and analysis are carried out on the Pavia Center dataset, and the deep hyperspectral prior (DHP) method proposed in the present invention is compared with the bicubic interpolation (Bicubic), local self-similarity (LSE), transform self-similarity (TSE), and Laplacian pyramid super-resolution network (LapSRN) algorithms. The upsampling results of different methods on the 11th hyperspectral image cube (named "PaviaCenter_Patch11") in the Pavia Center dataset are shown in Figure 2. Figure 3 As shown, Figure 3 (a) is the reference image H ref , compared to Figure 3 (b) Bicubic, (c) LSE, (d) TSE, and (e) LapSRN upsampling algorithms. Figure 3 (f) The DHP method proposed in the present invention shows clearer detail information and less spatial blur;

[0111] In order to further compare the spectrum preservation performance of different upsampling methods, Figure 4 The upsampling results of different methods on "PaviaCenter_Patch11" and the reference image H are presented. ref Spectral Angle Mapping (SAM) between Figure 4 (a) is the reference spectral angle mapping. The closer the image is to dark blue, the smaller the SAM value and the better the spectral retention performance. Figure 4(b) Bicubic, (c) LSE, (d) TSE, (e) LapSRN upsampling algorithms, Figure 4 In (f), the DHP method proposed in the present invention achieves a relatively smaller SAM value, especially in the area marked by the red frame line, which verifies that the DHP upsampling method proposed in the present invention has better spectral preservation ability.

[0112] The step 3 specifically includes the following steps:

[0113] Step 3.1: Upsample the hyperspectral image H obtained in step 2 up After concatenating the full-color image P obtained in step 1 in the spectral dimension, we get the input image H of the channel and spatial dual attention residual network. in , input image H in After passing through a feature extraction network consisting of a convolutional layer with a convolution kernel of 3×3, the initial features are extracted:

[0114] F0=f FEN (H in ),

[0115] Among them, f FEN (·) represents the feature extraction network, and F0 represents the initial feature extracted;

[0116] Step 3.2, the initial feature F0 extracted in step 3.1 is fed to the feature mapping network, which stacks N channels and spatial dual attention residual blocks to enhance the representation ability of the network:

[0117] F N =f FMN (F0) = f R,N (f R,N-1 (…f R,1 (F0)…)),

[0118] Among them, F N represents the output features of the Nth channel and spatial dual attention residual block in the feature mapping network, f FMN (·) represents the feature mapping network, f R,N (·) represents the Nth channel and spatial dual attention residual block, such as Figure 5 As shown in Figure 2, the channel and spatial dual attention residual block integrates the channel attention (CA) module and the spatial attention (SA) module into the residual block to adaptively rescale the feature map. The input feature of the Nth channel and spatial dual attention residual block is F N-1 , input feature F N-1First, a more refined feature U is generated through two convolutional layers with a kernel size of 3×3 connected by a ReLU activation function. Then, the feature U is fed to the Channel Attention (CA) module and the Spatial Attention (SA) module for attention weighting. Then, the input features of the Nth channel and spatial dual attention residual block, the features output by the Channel Attention (CA) module, and the features output by the Spatial Attention (SA) module are added together.

[0119] When the feature U is fed to the Channel Attention (CA) module:

[0120] Step 3.2.1, let U = [u1,u2,…,u C ] represents the feature of the input channel attention (CA) module. The feature U contains C feature maps with a spatial size of H×W, where H and W represent the height and width of the feature map, respectively. A global average pooling layer is used to obtain the vector v∈R C , the c-th element of vector v is:

[0121]

[0122] Among them, u c (i,j) is the cth channel u c The value at position (i,j) in the c is the cth element of the vector v obtained by the global average pooling layer of feature U. The global spatial information is compressed into the vector v.

[0123] Step 3.2.2, feed the vector v obtained in step 3.2.1 to two convolutional layers with kernel size of 1×1, apply ReLU activation function after the first convolutional layer, and apply Sigmoid activation function after the second convolutional layer to obtain the channel attention mask M CA :

[0124] M CA =σ(Ω CA,2 (δ(Ω CA,1 v))),

[0125] Among them, δ(·) represents the ReLU activation function, σ(·) represents the Sigmoid activation function, and Ω CA,1 and Ω CA,2are the weights of the first and second convolutional layers in the Channel Attention (CA) module, M CA is the obtained channel attention mask, vector v∈R C The number of channels is reduced in the first convolutional layer by a reduction rate r, and the number of channels is reduced to After being activated by the ReLU activation function, the second convolutional layer increases to C channels;

[0126] Step 3.2.3: Add the channel attention mask M obtained in step 3.2.2 CA Multiply the corresponding elements with the feature U fed to the Channel Attention (CA) module to obtain the feature U output by the Channel Attention (CA) module CA :

[0127]

[0128] in, Indicates the multiplication of corresponding elements, U CA Features output by the Channel Attention (CA) module;

[0129] When the feature U is fed into the Spatial Attention (SA) module:

[0130] Step 3.2.4, let U = [u 1,1 ,u 1,2 ,…,u i,j ,…,u H,W ] represents the input feature of the SpatialAttention (SA) module, u i,j ∈R C Represents the feature vector at the spatial position (i, j) of the feature map. H and W represent the height and width of the feature map respectively. A convolution layer with a convolution kernel size of 1×1 is used to convolve the feature U. The convolved feature is activated using the Sigmoid activation function to obtain the spatial attention mask M. SA :

[0131] M SA =σ(Ω SA U),

[0132] Among them, Ω SA is the weight of the convolutional layer in the Spatial Attention (SA) module, σ(·) represents the Sigmoid activation function, and M SA is the obtained spatial attention mask;

[0133] Step 3.2.5: The spatial attention mask M obtained in step 3.2.4 is SA Multiply the corresponding elements with the feature U fed to the Spatial Attention (SA) module to obtain the feature U output by the Spatial Attention (SA) module SA :

[0134]

[0135] in, Indicates the multiplication of corresponding elements, U SA Features output by the Spatial Attention (SA) module;

[0136] In step 3.2.6, a skip connection is used in each channel and spatial dual attention residual block to add the input features of the Nth channel and spatial dual attention residual block, the features of the output of the channel attention (CA) module obtained in step 3.2.3, and the features of the output of the spatial attention (SA) module obtained in step 3.2.5 to obtain the output features of the Nth channel and spatial dual attention residual block:

[0137] F N =F N-1 +U CA +U SA ,

[0138] Among them, F N-1 represents the input features of the Nth channel and spatial dual attention residual block, F N represents the output features of the Nth channel and spatial dual attention residual block;

[0139] Step 3.3: The output feature F of the Nth channel and spatial dual attention residual block in the feature mapping network obtained in step 3.2 is N The residual reconstruction network is fed into the residual reconstruction network, which uses a convolution layer with a convolution kernel size of 3×3 to reconstruct the residual image.

[0140]

[0141] Among them, f RRN (·) represents the residual reconstruction network, is the reconstructed residual image;

[0142] The channel and spatial dual attention residual network includes an L1 loss function, which is defined as:

[0143]

[0144] Among them, l is the number of training samples, k is the index of training samples, H res Denotes the reference image H ref With the upsampled image H up The residual image between them is obtained by optimizing the channel and spatial dual attention residual network using the L1 loss function to minimize the reconstructed residual image. and the residual image H res The error between .

[0145] The upsampled hyperspectral image H is up and the reconstructed residual image Perform sum operation to obtain the fused hyperspectral image H fus :

[0146]

[0147] Among them, H fus Represents the obtained fused hyperspectral image.

[0148] A system for hyperspectral image fusion method based on deep prior and dual attention residual network, comprising:

[0149] The preprocessing module obtains the image data set and preprocesses the data to obtain a low-resolution hyperspectral image H lr and a full-color image P, for implementing step 1 of a hyperspectral image fusion method based on deep prior and dual attention residual network;

[0150] The upsampling module uses a deep hyperspectral prior algorithm to upsample the low-resolution hyperspectral image H lr Upsample to obtain the upsampled hyperspectral image H up , used to implement step 2 of a hyperspectral image fusion method based on deep prior and dual attention residual network;

[0151] The reconstruction module combines the panchromatic image P and the upsampled hyperspectral image H up The input image H is obtained by splicing in the spectral dimension in , and then the input image H in Input into the channel and space dual attention residual network for reconstruction to obtain the reconstructed residual image Step 3 for implementing a hyperspectral image fusion method based on deep prior and dual attention residual network;

[0152] Fusion module, upsampled hyperspectral image H up and the reconstructed residual image Perform sum operation to obtain the fused hyperspectral image Hfus , used to implement step 4 of a hyperspectral image fusion method based on deep prior and dual attention residual network.

[0153] A hyperspectral image fusion device based on deep prior and dual attention residual network, comprising:

[0154] Memory: used to store a computer program for implementing the hyperspectral image fusion method based on deep prior and dual attention residual network;

[0155] Processor: used to implement the hyperspectral image fusion method based on deep prior and dual attention residual network when executing the computer program.

[0156] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a hyperspectral image fusion method based on deep prior and dual attention residual network.

[0157] The effect of the present invention is further described below in conjunction with simulation experiments.

[0158] 1. Experimental conditions

[0159] In order to evaluate the performance of the proposed hyperspectral image fusion method based on deep prior and dual attention residual network, a series of experiments and comparative analyses were carried out on two simulated hyperspectral datasets.

[0160] The first simulated dataset used in the experiment is the Pavia Center dataset. The spatial resolution of the Pavia Center image data is 1.3 meters, and its original size is 1096×715×102, that is, it has 1096×715 spatial pixels and 102 bands, and the spectral range is 430 to 860 nanometers. In order to facilitate the experimental design, we selected the 960×640 part of the upper left corner of the original image for subsequent experiments, and divided it into 24 non-overlapping hyperspectral image cubes of size 160×160×102, thus forming the Pavia Center dataset. The image in the Pavia Center dataset was used as the reference image. Based on the Wald protocol, the reference image was Gaussian blurred and downsampled 4 times to simulate the generation of a low-resolution hyperspectral image with a spatial size of 40×40. The spectral bands in the visible light range of the reference image were averaged to simulate a high-resolution panchromatic image with a spatial size of 160×160. The spatial resolution ratio between the panchromatic image and the low-resolution hyperspectral image is 4:1.

[0161] The second simulated dataset used in the experiment is the Botswana dataset. The Botswana hyperspectral image contains 242 spectral bands in the wavelength range of 400 to 2500 nanometers. After removing several noise bands, 145 bands remain for research. We selected the 1200×240 size portion of the upper left corner of the original image for the experiment and divided it into 20 non-overlapping hyperspectral image cubes of size 120×120×145 to form the Botswana dataset. The image in the Botswana dataset was used as the reference image. A low-resolution hyperspectral image with a spatial size of 40×40 was simulated by applying Gaussian blur and 3x downsampling on the reference image. A high-resolution panchromatic image with a spatial size of 120×120 was generated by averaging the spectral bands in the visible light range of the reference image. The spatial resolution ratio between the panchromatic image and the low-resolution hyperspectral image is 3:1.

[0162] 2. Experimental Setup

[0163] In the channel attention (CA) module of the channel and spatial dual attention residual network, in order to reduce the amount of computation and increase the nonlinearity of the model, the number of feature channels is first reduced by a reduction rate r in the first convolutional layer, and then increased to the original number of channels by the second convolutional layer after activation by the ReLU activation function. In the experiment, we set the reduction rate r to 16.

[0164] N channel and spatial dual attention residual blocks are stacked in the feature mapping network of the channel and spatial dual attention residual network. The number of channel and spatial dual attention residual blocks N directly determines the depth of the channel and spatial dual attention residual network. Considering the fusion performance and time cost, we set the number of channel and spatial dual attention residual blocks to N = 4 in the experiment.

[0165] During the training of the channel and spatial dual attention residual networks, each training pair is cropped into image patches of size 32×32 as network input, and the batch size is set to 128. We use a learning rate of 1×10 -3 The Adam optimizer is used to minimize the L1 loss function. The training cycle (Epoch) is set to 1.5×10 3 In the testing phase, we take the input image H from the test set in As input, the final fused image is compared with the corresponding reference image H ref Comparisons are made to verify the performance of the fusion method.

[0166] 3. Experimental Content

[0167] Experiment 1: The first experiment was conducted on the Pavia Center dataset, using the method of the present invention (DHP-DARN) and ten existing fusion methods to fuse low-resolution hyperspectral images with panchromatic images. Table 2 lists the objective quantitative evaluation results of different fusion methods on the Pavia Center dataset test set, where the optimal value is highlighted in bold. It can be found that the method proposed in the present invention (DHP-DARN) achieved the largest CC value and the smallest SAM, RMSE and ERGAS values, indicating that the DHP-DARN method achieved superior spatial and spectral quality compared to the comparison algorithm. In addition, the quantitative evaluation results of the proposed method are far superior to the other two CNN-based hyperspectral image fusion algorithms, which verifies the effectiveness of the channel and spatial attention modules proposed in the present invention.

[0168] Table 2 Objective quantitative evaluation results of different methods on the Pavia Center dataset

[0169]

[0170] For subjective visual assessment, Figure 6 The fusion results of different methods on the 21st hyperspectral image cube (named “PaviaCenter_Patch21”) in the PaviaCenter dataset are shown. Figure 6 Each sub-image in the figure includes an area marked by a red frame and is enlarged to facilitate the comparison of the spatial details of the fused image. Figure 6 (a) is the reference image H ref In traditional fusion methods, Figure 6 (b) GSA method, (c) PCA method, (f) GFPCA method and Figure 6 (i) There is a significant color difference between the fused image obtained by the CNMF method and the reference image. Figure 6 (d) SFIM method and Figure 6 (e) The fused image obtained by the MGH method suffers from the problem of spatial information loss. Figure 6 (g) BNA method and Figure 6 The fusion result generated by the (h)BSP method is very blurry. In the fusion method based on deep learning, Figure 6 (j) DDLPS method and Figure 6 The (k)DRCNN method has obvious spatial and spectral distortion in the magnified area. Figure 6 (l) The DHP-DARN method proposed in the present invention can reconstruct more accurate spatial detail information, such as Figure 6 The roof edge location of the enlarged area in (l).

[0171] In order to further evaluate the spectral fidelity of different fusion methods, Figure 7 The difference curves between the spectra at four pixel positions of the fusion results of different methods on "PaviaCenter_Patch21" and the reference image spectrum are plotted. The four pixel coordinates are (10,72), (56,52), (80,108) and (105,38). The positions of these four pixels have been Figure 6 The part (a) in the figure is highlighted in yellow. Figure 7 As shown in (a) to (d), each figure has a black dotted line as the baseline. The closer the curve is to the baseline, the smaller the spectral distortion of the fused image is. It can be clearly seen that the DHP-DARN method proposed in the present invention obtains the spectral curve closest to the reference image in the spectral domain, which verifies that the fusion method proposed in the present invention has better spectral fidelity performance.

[0172] Experiment 2: The second experiment was conducted on the Botswana dataset. For the Botswana dataset, Table 3 lists the objective quantitative evaluation results of different fusion methods on their test sets, with the best results marked in bold. It can be found that among all the comparison methods, the DHP-DARN method proposed in this invention obtains the best values ​​of CC, SAM, RMSE and ERGAS evaluation indicators, and achieves objective performance that exceeds traditional fusion methods and fusion methods based on deep learning.

[0173] Table 3 Objective quantitative evaluation results of different methods on the Botswana dataset

[0174]

[0175]

[0176] In order to conduct subjective visual evaluation of the fusion results, Figure 8 The fusion results of different methods on the 15th hyperspectral image cube in the Botswana dataset (named “Botswana_Patch15”) are shown. It can be observed that in the traditional fusion method, Figure 8 The fusion results of (b) GSA method, (c) PCA method, (f) GFPCA method, (g) BNA method, (h) BSP method and (i) CNMF method are compared with Figure 8 (a) Reference image H ref There is a clear color difference compared to Figure 8 (d) SFIM method and Figure 8 (e) The fusion result of the MGH method has obvious spatial fuzziness. Among the fusion methods based on deep learning, the DHP-DARN method proposed in this paper shows a better performance than Figure 8 (j) DDLPS method and Figure 8 The (k) DRCNN method has better visual effects, showing clearer spatial detail information and less spectral loss. The fusion results Figure 8 (l) is closest to Figure 8 (a) Reference image H ref .

[0177] In order to further evaluate the fusion performance of the DHP-DARN method proposed in this paper on the Botswana dataset, we Fig. 9 The error images between the fusion results of different methods on "Botswana_Patch15" and the reference image are shown in the figure. Bright yellow indicates a larger error value, and dark blue indicates a smaller error. Fig. 9 (a) is the reference error diagram. It can be seen that compared with Fig. 9 Among them, (b) GSA method, (c) PCA method, (d) SFIM method, (e) MGH method, (f) GFPCA method, (g) BNA method, (h) BSP method, (i) CNMF method, (j) DDLPS method and (k) DRCNN method, the present invention proposes Fig. 9 The DHP-DARN method in (l) generates the fusion result with the smallest error, which is consistent with Figure 8 This result is consistent with the fusion result of (l) in , which once again verifies the excellent fusion performance of the method of the present invention.

[0178] 4. Experimental Conclusion

[0179] The superiority of the DHP-DARN fusion method proposed in the present invention is verified on two hyperspectral datasets of different scenes. The spatial resolution ratios between the panchromatic images and the hyperspectral images of the two datasets are different, which fully demonstrates the generalization ability of the hyperspectral image fusion method proposed in the present invention.

Claims

1. A hyperspectral image fusion method based on deep prior and dual attention residual network, characterized in that: The following steps are involved: Step 1: Obtain an image dataset and preprocess the data to obtain a low-resolution hyperspectral image H lr and a full-color image P; Step 2: The low-resolution hyperspectral image H obtained in step 1 is processed by a deep hyperspectral prior algorithm. lr Upsample to obtain the upsampled hyperspectral image H up ; Step 3: The full-color image P obtained in step 1 and the upsampled hyperspectral image H obtained in step 2 are up The input image H is obtained by splicing in the spectral dimension in , and then the input image H in Input into the channel and space dual attention residual network for reconstruction to obtain the reconstructed residual image Step 4: Upsample the hyperspectral image H obtained in step 2 up And the reconstructed residual image obtained in step 3 Perform sum operation to obtain the fused hyperspectral image H fus .

2. The hyperspectral image fusion method based on deep prior and dual attention residual network according to claim 1 is characterized in that: The step 1 specifically comprises the following steps: Step 1.1: Obtain the original hyperspectral image from the public dataset as the reference image H ref ; Step 1.2, according to the Wald protocol, by using the reference image H obtained in step 1.1 ref Use Gaussian blur and downsample by 4 times to get a low-resolution hyperspectral image H lr , for the reference image H obtained in step 1.1 ref The spectral bands in the visible light range are averaged to obtain a high-resolution full-color image P.

3. The hyperspectral image fusion method based on deep prior and dual attention residual network according to claim 1 is characterized in that: The step 2 specifically includes the following steps: The low-resolution hyperspectral image H obtained in step 1 lr , the U-Net network architecture of the deep hyperspectral prior algorithm is upsampled to the same spatial size as the panchromatic image P, and a higher quality upsampled hyperspectral image H is obtained. up : The U-Net network architecture includes five downsampling units d[p], five upsampling units u[p] and five jump connection units s[p], where p = 1, 2, ..., 5. The U-Net network takes a tensor Z filled with uniform noise as input. The spatial dimension of the tensor Z is consistent with the full-color image P, and the spectral dimension of the tensor Z is consistent with the low-resolution hyperspectral image H. lr Consistent, the U-Net network output is the upsampled hyperspectral image H up ; Each of the downsampling units d contains a Lanczos2 downsampling module, batch normalization (BatchNormalization, BN), a LeakyReLU activation function and a convolution layer with a convolution kernel of 3×3, as an encoder for extracting effective features; Each of the upsampling units u consists of a bilinear upsampling module and two convolutional layers, has the same convolution kernel size as the downsampling unit d, and acts as a decoder to reconstruct the image; The jump connection unit s is added between each of the downsampling units d and each of the upsampling units u, and each of the jump connection units s includes a convolution layer with a convolution kernel of 1×1, batch normalization (Batch Normalization, BN) and a LeakyReLU activation function.

4. The hyperspectral image fusion method based on deep prior and dual attention residual network according to claim 3 is characterized in that: The deep hyperspectral prior algorithm includes upsampling the hyperspectral image H up Do downsampling to get the upsampled hyperspectral image after downsampling The L1 loss function is used to convert the downsampled upsampled hyperspectral image and the low-resolution hyperspectral image H obtained in step 1 lr Minimize the error between them.

5. The hyperspectral image fusion method based on deep prior and dual attention residual network according to claim 1 is characterized in that: The step 3 specifically includes the following steps: Step 3.1: Upsample the hyperspectral image H obtained in step 2 up After concatenating the full-color image P obtained in step 1 in the spectral dimension, we get the input image H of the channel and spatial dual attention residual network. in , input image H in After passing through a feature extraction network consisting of a convolutional layer with a convolution kernel of 3×3, the initial features are extracted: F0=f FEN (H in ), Among them, f FEN (·) represents the feature extraction network, and F0 represents the initial feature extracted; Step 3.2, the initial feature F0 extracted in step 3.1 is fed to the feature mapping network, which stacks N channels and spatial dual attention residual blocks to enhance the representation ability of the network: F N =f FMN (F0)=f R,N (f R,N-1 (…f R,1 (F0)…)), Among them, F N represents the output features of the Nth channel and spatial dual attention residual block in the feature mapping network, f FMN (·) represents the feature mapping network, f R,N (·) represents the Nth channel and spatial dual attention residual block; the channel and spatial dual attention residual block integrates the channel attention (CA) module and the spatial attention (SA) module into the residual block to adaptively rescale the feature map; the input feature of the Nth channel and spatial dual attention residual block is F N-1 , input feature F N-1 First, a more refined feature U is generated through two convolutional layers with a kernel size of 3×3 connected by a ReLU activation function. Then, the feature U is fed to the Channel Attention (CA) module and the Spatial Attention (SA) module for attention weighting. Then, the input features of the Nth channel and spatial dual attention residual block, the features output by the Channel Attention (CA) module, and the features output by the Spatial Attention (SA) module are added together. When the feature U is fed to the ChannelAttention (CA) module: Step 3.2.1, let U = [u1,u2,…,u C ] represents the feature of the input channel attention (CA) module. The feature U contains C feature maps with a spatial size of H×w, where H and W represent the height and width of the feature map, respectively. A global average pooling layer is used to obtain the vector v∈R C , the c-th element of vector v is: Among them, u c (i,j) is the cth channel u c The value at position (i,j) in the c is the cth element of the vector v obtained by the global average pooling layer of feature U; Step 3.2.2, feed the vector v obtained in step 3.2.1 to two convolutional layers with kernel size of 1×1, apply ReLU activation function after the first convolutional layer, and apply Sigmoid activation function after the second convolutional layer to obtain the channel attention mask M CA : M CA =σ(Ω CA,2 (d(Ω CA,1 v))), Among them, δ(·) represents the ReLU activation function, σ(·) represents the Sigmoid activation function, and Ω CA,1 and Ω CA,2 are the weights of the first and second convolutional layers in the Channel Attention (CA) module, respectively. CA is the obtained channel attention mask, vector v∈R C The number of channels is reduced in the first convolutional layer by a reduction rate r, and the number of channels is reduced to After being activated by the ReLU activation function, the second convolutional layer increases to C channels; Step 3.2.3: Add the channel attention mask M obtained in step 3.2.2 CA Multiply the corresponding elements with the feature U fed to the Channel Attention (CA) module to obtain the feature U output by the Channel Attention (CA) module CA : in, Indicates the multiplication of corresponding elements, U CA Features output by the Channel Attention (CA) module; When the feature U is fed to the SpatialAttention (SA) module: Step 3.2.4, let U = [u 1,1 ,u 1,2 ,…,u i,j ,…,u H,W ] represents the input feature of the SpatialAttention (SA) module, u i,j ∈R C Represents the feature vector at the spatial position (i, j) of the feature map. H and W represent the height and width of the feature map respectively. A convolution layer with a convolution kernel size of 1×1 is used to convolve the feature U. The convolved feature is activated using the Sigmoid activation function to obtain the spatial attention mask M. SA : M SA =σ(Ω SA U), Among them, Ω SA is the weight of the convolutional layer in the Spatial Attention (SA) module, σ(·) represents the Sigmoid activation function, and M SA is the obtained spatial attention mask; Step 3.2.5: The spatial attention mask M obtained in step 3.2.4 is SA Multiply the corresponding elements with the feature U fed to the SpatialAttention (SA) module to obtain the feature U output by the SpatialAttention (SA) module SA : in, Indicates the multiplication of corresponding elements, U SA Features output by the Spatial Attention (SA) module; In step 3.2.6, a skip connection is used in each channel and spatial dual attention residual block to add the input features of the Nth channel and spatial dual attention residual block, the features of the output of the channel attention (CA) module obtained in step 3.2.3, and the features of the output of the spatial attention (SA) module obtained in step 3.2.5 to obtain the output features of the Nth channel and spatial dual attention residual block: F N =F N-1 +U CA +U SA , Among them, F N-1 represents the input features of the Nth channel and spatial dual attention residual block, F N represents the output features of the Nth channel and spatial dual attention residual block; Step 3.3: The output feature F of the Nth channel and spatial dual attention residual block in the feature mapping network obtained in step 3.2 is N The residual reconstruction network is fed into the residual reconstruction network, which uses a convolution layer with a convolution kernel size of 3×3 to reconstruct the residual image. Among them, f RRN (·) represents the residual reconstruction network, is the reconstructed residual image.

6. The hyperspectral image fusion method based on deep prior and dual attention residual network according to claim 5 is characterized in that: The channel and spatial dual attention residual network includes an L1 loss function, which is defined as: Among them, l is the number of training samples, k is the index of training samples, H res Denotes the reference image H ref With the upsampled image H up The residual image between them is obtained by optimizing the channel and spatial dual attention residual network using the L1 loss function to minimize the reconstructed residual image. and the residual image H res The error between .

7. The hyperspectral image fusion method based on deep prior and dual attention residual network according to claim 1 is characterized in that: In step 4, the upsampled hyperspectral image H up and the reconstructed residual image Perform sum operation to obtain the fused hyperspectral image H fus : Among them, H fus Represents the obtained fused hyperspectral image.

8. A system for a hyperspectral image fusion method based on deep prior and dual attention residual network according to any one of claims 1 to 7, characterized in that: include: The preprocessing module obtains the image data set and preprocesses the data to obtain a low-resolution hyperspectral image H lr and a full-color image P; The upsampling module uses a deep hyperspectral prior algorithm to upsample the low-resolution hyperspectral image H lr Upsample to obtain the upsampled hyperspectral image H up ; The reconstruction module combines the panchromatic image P and the upsampled hyperspectral image H up The input image H is obtained by splicing in the spectral dimension in , and then the input image H in Input into the channel and space dual attention residual network for reconstruction to obtain the reconstructed residual image Fusion module, upsampled hyperspectral image H up and the reconstructed residual image Perform sum operation to obtain the fused hyperspectral image H fus .

9. A hyperspectral image fusion device based on deep prior and dual attention residual network, characterized in that: include: Memory: used to store a computer program for implementing a hyperspectral image fusion method based on deep prior and dual attention residual network as described in any one of claims 1 to 7; Processor: used to implement a hyperspectral image fusion method based on deep prior and dual attention residual network as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a hyperspectral image fusion method based on deep prior and dual attention residual network as described in any one of claims 1 to 7 are implemented.

Citation Information

Patent Citations

  • Hyper-spectral and panchromatic image fusion method for extracting multi-scale spatial-spectral characteristics based on AE

    CN112634137A

Cited By

  • Multispectral image fusion method and system based on spatial spectrum difference prior guidance

    CN120510047A

  • Multispectral image fusion method and system based on spatial spectral difference prior guidance

    CN120510047B

  • Hyperspectral and multispectral image fusion method and system based on Reformer residual iteration

    CN120543374A

  • Hyperspectral image fusion method and system based on variance guidance and heavy tail estimation

    CN121563805A