A method for simultaneous spatial-spectral super-resolution reconstruction based on multi-task interactive learning
Through the multi-task interactive learning of the simultaneous super-resolution reconstruction method, the SSFIN network architecture and weight decay strategy are used to solve the problem of high spatial resolution and high spectral resolution image reconstruction in the existing technology, and the efficient space-spectral characteristics are achieved Interaction, better than the performance of existing methods.
Patent Information
- Application Number
- CN202111338491.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-12
AI Technical Summary
The prior art is difficult to realize the reconstruction of high spatial resolution and high spectral resolution images at the same time, and the existing deep neural network methods are cost-effective and inefficient in computing, so they cannot effectively utilize the intrinsic relationship between spatial and spectral information.
Using multi-task interactive learning, the simultaneous super-resolution reconstruction method is used to establish an SSFIN network architecture, including spatial super-segment branches, spectral super-segment branches and spatial-spectral feature interaction modules, and the network is trained using weight attenuation strategies to make the space and spectrum Feature interaction gradually reduces the weight of auxiliary tasks in the total loss and improves feature learning ability.
Achieve efficient space-spectral joint super-resolution reconstruction, which is better than the most advanced two-stage method, improves the network's feature learning ability and performance, and achieves the most advanced performance on multiple public data sets.
Smart Images

Figure CN114219708B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a spatial-spectral simultaneous super-resolution reconstruction method based on multi-task interactive learning, and belongs to the field of image processing and computer vision. Background Art
[0002] High spatial resolution and high spectral resolution images (HR-HSI) are widely used in earth science, medical diagnosis and other fields. However, how to obtain images with both high spatial resolution and high spectral resolution remains an unsolved problem.
[0003] Joint spatial-spectral super-resolution (SSSR) aims to reconstruct high spatial resolution hyperspectral images (HR-HSI) from low spatial resolution multispectral images (LR-MSI). Hyperspectral images (HSI) have high spectral resolution and are widely used in various computer vision tasks, including target recognition and tracking, medical image processing, and remote sensing. Although HSI can achieve high spectral resolution through continuous narrow bands, its spatial resolution is usually much coarser than that of multispectral images (MSI). This is because the dense spectral channels in hyperspectral sensors allow a limited number of photons to arrive evenly in a narrow spectral window. The low spatial resolution of hyperspectral images (LR-HSI) captured by the sensor greatly limits its application value. In addition, hyperspectral capture often requires a long capture time and cannot be used to capture dynamic scenes. Therefore, how to reconstruct HR-HSI from LR-MSI is an important and challenging problem.
[0004] Recently, deep neural networks (DNNs) have achieved great success in numerous visual recognition and image restoration tasks. Many DNN-based single image super-resolution (SISR) and spectral super-resolution (SSR) methods have been widely studied and achieved very good performance. To solve the problem of joint spatial-spectral super-resolution, a straightforward approach is to execute the SISR network and the SSR network sequentially in a two-stage manner or in reverse order. It should be noted that when we complete SSR first, LR-HSI is obtained, and we have two ways to obtain the super-resolved target HR-HSI. One method is to reconstruct each HR spectral image separately through SISR, and the other is to jointly estimate the target HR-HSI through hyperspectral image super-resolution (HSISR).
[0005] However, spatial super-resolution reconstruction and spectral prediction in SSSR are interrelated and they should be mutually beneficial. Running these two algorithms independently cannot fully exploit the intrinsic relationship between spectral and spatial information, which limits the performance of SSSR. In addition, to predict the target HR-HSI, both the state-of-the-art SISR / HSISR and SSR networks require a large number of parameters. Therefore, the two-stage SSSR neural network is computationally expensive and extremely inefficient. A learning-based method, called SSJSR, was proposed to solve the SSSR problem. They first trained a 3-layer spatial super-resolution network, followed by a 5-layer spectral super-resolution network. This two-stage training strategy divides SSSR into two independent tasks and completely ignores the correlation between the two tasks. Summary of the invention
[0006] The present invention proposes a spatial-spectral simultaneous super-resolution reconstruction method based on multi-task interactive learning, which solves the problems existing in the prior art.
[0007] A method for simultaneous super-resolution reconstruction of space and spectrum based on multi-task interactive learning, comprising the following steps:
[0008] Step 1: Establish the SSFIN network architecture;
[0009] Step 2: Use the weight decay strategy to train the SSFIN network.
[0010] Furthermore, in step one, the following steps are included:
[0011] Step 11: Establish three branches in the SSFIN network: SISR network, SSR network and spatial spectral fusion network. The three branches generate three images respectively: HR-MSI, LR-HSI and HR-HSI. The three images are used as the output of the SSFIN network, and LR-MSI is used as the input of the SSFIN network.
[0012] Step 1 and 2: Spatial super-resolution branch:
[0013] First, convolutional layers are used to extract shallow spatial features from LR-MSI input. Then, N spatial residual groups SpaRGs are used to extract deep spatial features. A long skip connection is added to retain the original low-frequency information of LR-MSI. Each spatial residual group consists of several residual spatial attention blocks RSAB and a short skip connection to explore the spatial relationship of feature maps. Finally, a spatial super-resolution head module consisting of two convolutional layers and an upsampling model is used to reconstruct HR-MSI.
[0014] The SISR network is optimized using a loss defined as L1:
[0015]
[0016] Among them is the output of the SISR network, and U is the ground truth HR-MSI;
[0017] Step 13. Spectral super-resolution branch:
[0018] First, a convolutional layer is used to extract shallow spectral features, and then N spectral residual groups (SpeRGs) are used to extract deep spectral features. Each spectral residual group consists of several residual channel attention blocks (RCABs). The residual channel attention block applies a global average pooling layer to learn the weights of each channel. Finally, a spectral super-resolution head module composed of a convolutional layer is used to reconstruct the LR-HSI.
[0019] Apply the L1 loss to train the SSR network:
[0020]
[0021] Among them is the output of the SSR network, and V is the ground truth LR-HSI;
[0022] Step 14. Spatial-spectral feature interaction module:
[0023] In the k-th SSFIB:
[0024]
[0025]
[0026]
[0027] where Conv represents the convolution operation, Concat represents the concatenation operator, and H SpaRG 、H SpeRG and H fm represent the spatial residual group, the spectral residual group, and the fusion module respectively. and represent the outputs of the k-th spatial residual group, spectral residual group, and fusion module respectively.
[0028] Set To fuse the features extracted from the spatial residual group, the spectral residual group, and the previous fusion module, first concatenate and , where 0 < n < k. Then, use a fusion module containing two convolutional layers and two rectified linear units (ReLUs) to integrate the spatial and spectral features. Next, apply a convolutional layer to extract the unique features of the SISR network and the SSR network respectively. Finally, add their unique features to the original features.
[0029] Step 15: Spatial-spectral fusion network:
[0030] First, a convolutional layer is used to integrate the features from the three branches. Then, an upsampling model is used to make these features have the same spatial scale as HR-HSI. Then, G space-spectral residual groups SSRGs are applied to reconstruct the target HR-HSI. The dual residual attention block RDAB is used to connect spatial attention and channel attention. In the space-spectral residual group, the dual residual attention block is applied to extract more useful spatial and spectral features at the same time. After that, a convolutional layer is used to integrate the features extracted by the network at different stages. This is because the deep neural network extracts features with different information at different depths and shares the parameters of each empty spectrum residual group. After G empty spectrum residual groups, a convolutional layer is used to reconstruct the output target.
[0031] Use L1 loss to train the spatial-spectral fusion network:
[0032]
[0033] in is the output of the fusion network, and X is the ground-truth HR-HSI.
[0034] Furthermore, in step 2, specifically, the spatial super-resolution network and the spectral super-resolution network are regarded as two regularized branches, and the weight decay strategy is used to train SSFIN. The total loss function of SSFIN is:
[0035]
[0036] Where M and m represent the total number of training rounds and the current number of training rounds, respectively. α and β are trade-off parameters. From the definition of the loss function, we know that once L spa and L spe The weight of is decayed to zero, that is, the learned spatial and spectral features are used to help the joint reconstruction of HSI.
[0037] Furthermore, during the training process, specifically, in the spatial super-resolution branch, the number of spaRGs is set to N=3, in each spaRG, the number of RSABs is set to 5, in the spectral super-resolution branch, the number of speRGs is set to N=3, in each speRG, the number of RCABs is set to 5, in the spatial-spectral fusion branch network, the number of SSRGs is set to 4, each SSRG has 5 RDABs, the convolution kernel sizes of all convolutional layers are set to 3×3, α and β are set to 0.5, M is set to 100, the network is trained using the Adam optimizer for 250,000 iterations, the batch size is 8, each epoch is set to 2500 iterations, the learning rate is initialized to 0.0008, and decayed by 2 times when reaching 5, 15, 30 and 60 epochs, and the network is implemented and trained using the Pytorch framework and NVIDIA Tesla V100 GPU.
[0038] The present invention has the following beneficial effects: A spatial-spectral simultaneous super-resolution reconstruction method based on multi-task interactive learning of the present invention designs a spatial-spectral feature interaction block (SSFIB) so that the spatial super-resolution branch and the spectral super-resolution branch can benefit from each other. The reconstruction module aims to integrate features from the spatial and spectral SR branches to predict the target HR-HSI. In this multi-task framework, the weights of the spatial super-resolution and spectral super-resolution tasks in the total loss are gradually reduced (they will be reduced to zero at the end of training). Similar to the current feature contrast representation learning framework, a small neural network head, namely the spatial / spectral super-resolution head, is introduced to remove information that may only be useful for downstream tasks (i.e., spatial / spectral super-resolution), but retain more spatial and spectral information for the main spatial-spectral joint super-resolution task. Therefore, the spatial SR and spectral SR tasks can be regarded as regularization branches, which will greatly improve the feature learning ability of the network. Experimental results show that the proposed SSSR framework outperforms the most advanced two-stage method. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 The overall architecture of the proposed SSFIN network;
[0040] Figure 2 is the detailed network architecture of the spatial residual group, where Figure 2 (a) is the spatial residual group, Figure 2 (b) is the spectral residual group, Figure 2 (c) is the spatial spectrum residual group;
[0041] Figure 3 is the spatial super-resolution head and the spectral super-resolution head, where Figure 3 (a) is the spatial super-resolution head; Figure 3 (b) is the spectrum super-resolution head;
[0042] Figure 4 Qualitative results for the CAVE dataset;
[0043] Figure 5 Qualitative results and corresponding error images for channel 51 of the Pavia Centre dataset;
[0044] Figure 6 Reconstructed spectral features of four randomly selected locations in the Pavia Centre dataset. DETAILED DESCRIPTION
[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0046] This paper proposes a spatial-spectral feature interaction network (SSFIN) to learn spatial-spectral joint super-resolution in a multi-task framework. Since directly recovering HR-HSI from LR-MSI is a serious ill-posed problem, we introduce two auxiliary tasks (i.e., spatial super-resolution and spectral super-resolution) to help the network better recover HR-HSI. In particular, the backbone architecture of our SSFIN network consists of three branches, one for spatial super-resolution and one for spectral super-resolution, and the main branch is to integrate the spatial and spectral representations learned from the spatial and spectral super-resolution branches.
[0047] Since the spatial super-resolution task and the spectral super-resolution task are interrelated and should be mutually beneficial rather than independent, we design a spatial-spectral feature interaction block (SSFIB) so that the spatial super-resolution branch and the spectral super-resolution branch can benefit from each other. Finally, the reconstruction module aims to integrate the features from the spatial and spectral SR branches to predict the target HR-HSI. In this multi-task framework, we gradually reduce the weights of the spatial super-resolution and spectral super-resolution tasks in the total loss (which will be reduced to zero at the end of training). Similar to the current feature contrastive representation learning framework, we introduce a small neural network head, namely: the spatial / spectral super-resolution head, to remove information that may only be useful for the downstream task (i.e., spatial / spectral super-resolution), but retain more spatial and spectral information for the main spatial-spectral joint super-resolution task. Therefore, the spatial SR and spectral SR tasks can be regarded as regularization branches, which will greatly improve the feature learning ability of the network. Experimental results show that the proposed SSSR framework outperforms the state-of-the-art two-stage methods. In summary, the main contributions of this work are threefold:
[0048] 1. We introduce an efficient joint spatial-spectral super-resolution network that can simultaneously solve the spatial super-resolution reconstruction and spectral prediction problems in a unified end-to-end framework. By exploiting the correlation between the spatial super-resolution and spectral super-resolution branches, the proposed method outperforms the two-stage method.
[0049] 2. In order to fully exploit the complementarity of spatial SR and spectral SR tasks, we design a new spatial-spectral feature interaction block to promote the mutual guidance between the spatial SR branch and the spectral SR branch. These two regularization branches with spatial / spectral SR heads can greatly improve the spatial-spectral feature learning ability of the proposed network.
[0050] 3. The proposed SSSR method achieves state-of-the-art performance on three public datasets. Extensive ablation experiments further verify the effectiveness of different parts of the proposed method.
[0051] In this paper, we use lowercase letters to represent scalars, bold letters to represent matrices, and curly letters to represent tensors. In particular, let stands for LR-MSI. The goal of SSSR is to Estimation of images with high spatial resolution and high spectral resolution If we transform the tensor and Reshape into their matrix form, that is: Y∈R hw×s and X∈R HW×S , the observation model of LR-MSI can be written as follows:
[0052] Y=CXR (8)
[0053] where R∈R S×s is the spectral response function of the multispectral sensor, C∈R hw×HW represents the spatial degradation operator, which is usually assumed to consist of a circular convolution operator and a downsampling matrix.
[0054] As we can see from equation (1), estimating X from Y is a serious ill-posed problem, especially when the upsampling scale is very large. Therefore, it is very difficult to reconstruct HR-HSI directly from LR-MSI. In this paper, we introduce two auxiliary tasks (i.e., SISR and SSR) to help the network better recover HR-HSI. Therefore, the SSSR problem is decomposed into three sub-problems: SISR, SSR, and spatial-spectral fusion. These can be expressed as
[0055] Y=CU,and U=XR (9)
[0056] Y=VR,and V=CX (10)
[0057] where U∈RHW×s and V∈R hw×S Represent SISR results and SSR results respectively.
[0058] The present invention proposes an embodiment of a method for simultaneous super-resolution reconstruction of space and spectrum based on multi-task interactive learning. The method for simultaneous super-resolution reconstruction of space and spectrum based on multi-task interactive learning comprises the following steps:
[0059] Step 1: Establish the SSFIN network architecture;
[0060] Step 2: Train the SSFIN network using the weight decay strategy.
[0061] Furthermore, in step one, the following steps are included:
[0062] Step 1. Reference Figure 1 As shown, three branches are established in the SSFIN network: a SISR network, a SSR network, and a spatial-spectral fusion network. The three branches generate three images respectively: HR-MSI, LR-HSI, and HR-HSI. The three images are used as the output of the SSFIN network, and LR-MSI is used as the input of the SSFIN network.
[0063] Step 1 and 2: Spatial super-resolution branch:
[0064] Spatial super-resolution branch: We first use convolutional layers to extract spatial shallow features from the LR-MSI input. Then we use N spatial residual groups (SpaRGs) to extract deep spatial features. In addition, a long skip connection is added to preserve the original low-frequency information of LR-MSI. Each spatial residual group consists of several residual spatial attention blocks (RSABs) and a short skip connection to explore the spatial relationship of the feature map. Finally, referring to Figure 3 As shown in (a), the spatial super-resolution head module consisting of two convolutional layers and an upsampling model is used to reconstruct HR-MSI. The detailed network architecture of the spatial residual group is shown in Figure 2 (a) shown.
[0065] The SISR network is optimized using a loss defined as L1:
[0066]
[0067] in is the output of the SISR network, and U is the ground-true HR-MSI;
[0068] Step 1 and 2, spectral super-resolution branch: Similar to the SISR network branch, we first use a convolutional layer to extract shallow spectral features. Then we use N spectral residual groups (SpeRGs) to extract deep spectral features. Each spectral residual group consists of several residual channel attention blocks (RCABs). The residual channel attention block applies a global average pooling layer to learn the weight of each channel, which can greatly help the network recover the spectral information of HSI. Finally, refer to Figure 3 As shown in (b), the spectral super-resolution head module consisting of a convolutional layer is used to reconstruct LR-HSI. The detailed network architecture of the residual channel attention block is shown in Figure 2 (b) as shown.
[0069] We apply L1 loss to train the SSR network:
[0070]
[0071] in is the output of the SSR network, and V is the ground-true LR-HSI.
[0072] Step 13: Spatial-spectral feature interaction module:
[0073] HSI contains rich spatial and spectral information, which are intrinsically related. Therefore, we designed spatial-spectral feature interaction to allow the SISR network and the SSR network to exchange information and guide each other. Figure 1 As shown, in the kth SSFIB:
[0074]
[0075]
[0076]
[0077] Where Conv represents the convolution operation and Concat represents the cascade operator. SpaRG , H SpeRG and H fm They represent the spatial residual group, spectral residual group and fusion module respectively. and Represent the output of the kth spatial residual group, spectral residual group and fusion module respectively. We set To fuse the features extracted from the spatial residual group, spectral residual group and the previous fusion module, we first transform and (0 < n < k) are cascaded, and then a fusion module containing two convolutional layers and two rectified linear units (ReLU) is used to integrate spatial and spectral features. Then, convolutional layers are applied to extract the unique features of the SISR network and the SSR network respectively. Finally, we add their unique features to the original features. With this feature interaction learning strategy, we can expect that the spatial branch and the spectral branch can simultaneously consider spatial context information and spectral correlation;
[0078] Step 14. Spatial-spectral fusion network:
[0079] Since the SISR network and the SSR network can help us fully extract the spatial and spectral features of LR-MSI, we use these two networks as regularization branches to model the prior information for reconstructing HR-HSI. To prevent information loss, we provide the spatial and spectral super-resolution network learning features ( and ) to the spatial-spectral fusion network, rather than their super-resolution results ( and ). In addition, is also input into the spatial-spectral fusion network. We first use convolutional layers to integrate the features from the three branches. Then, an upsampling model is used to make these features have the same spatial scale as the HR-HSI. Then, G spatial-spectral residual groups (SSRGs) are applied to reconstruct the target HR-HSI. We use a dual residual attention block (RDAB) to connect spatial attention and channel attention. In the spatial-spectral residual group, we apply the dual residual attention block to simultaneously extract more useful spatial and spectral features. After extracting the spatial and spectral features at different levels , we use convolutional layers to integrate the features extracted by the network at different stages. This is because deep neural networks extract features with different information at different depths. To reduce the model parameters, we share the parameters of each spatial-spectral residual group. Ablation analysis experiments prove that this operation can not only reduce the number of network parameters, but also improve the network performance. After G spatial-spectral residual groups, a convolutional layer is used to reconstruct the output target. The detailed network architecture of the spatial-spectral residual group is shown in Figure 2 (c).
[0080] In the present invention, we use the L1 loss to train the spatial-spectral fusion network:
[0081]
[0082] where is the output of the fusion network, and X is the ground truth HR-HSI.
[0083] Furthermore, in step 2, specifically, in order to prevent overfitting and enhance the generalization ability of the model, we regard the spatial super-resolution network and the spectral super-resolution network as two regularization branches. In our experiments, we use a weight decay strategy to train our SSFIN so that SSFIN can gradually shift attention from auxiliary tasks to the main task. The total loss function of SSFIN is:
[0084]
[0085] Where M and m represent the total number of training rounds and the current number of training rounds respectively. α and β are trade-off parameters. From the definition of the loss function, we can see that once L spa and V spe The weight decays to zero, and the learned spatial and spectral features can be used to help the joint reconstruction of HSI.
[0086] In the actual implementation, we set the number of spaRGs to N=3 in the spatial super-resolution branch. In each spaRG, we set the number of RSABs to 5. In the spectral super-resolution branch, we set the number of speRGs to N=3. In each speRG, we set the number of RCABs to 5. In the spatial-spectral fusion branch network, we set the number of SSRGs to 4, and each SSRG has 5 RDABs. We set the convolution kernel size of all convolutional layers to 3×3. α and β are set to 0.5, and M is set to 100. We train the network with the Adam optimizer for 250,000 iterations with a batch size of 8. We set each epoch to 2500 iterations. The learning rate is initialized to 0.0008 and decayed by a factor of 2 at 5, 15, 30, and 60 epochs. We use the Pytorch framework and NVIDIA Tesla V100 GPU to implement and train our network.
[0087] Next, we compare our SSFIN method with a two-stage method consisting of state-of-the-art (SOTA) single image super-resolution + spectral super-resolution or spectral super-resolution + single hyperspectral image super-resolution and a recently proposed SOTA spatial-spectral joint super-resolution method SSJSR. We use RCAN and SAN for single image super-resolution methods, and AWAN for spectral super-resolution methods. For single hyperspectral image super-resolution, we choose RCAN and SSPSR. All compared methods are retrained on the same training dataset. As for objective comparison, we use four image quality metrics to evaluate the performance of different methods, including PSNR, SSIM, SAM, and ERGAS. PSNR and SSIM are calculated on each 2D spatial image and then averaged over all bands. They evaluate the similarity between the recovered HR-HSI and the ground truth based on MSE and structural consistency, respectively. SAM calculates the average angle between the spectral vectors of the recovered HR-HSI and the ground truth. As for ERGAS, it reflects the overall quality of the recovered HR-HSI. The smaller the values of ERGAS and SAM, the better the results, while the larger the values of PSNR and SSIM, the better the results.
[0088] First, set up the experiment:
[0089] Training datasets: We evaluate our US3RN method on two publicly available hyperspectral imaging datasets, including CAVE and Pavia Centre. The CAVE dataset consists of 32 indoor images captured under controlled illumination, with a spatial size of 512×512, including 31 10nm-wide spectral bands, covering the visible spectrum from 400 to 700nm. The HR-MSI (RGB image) is obtained by multiplying all ground-truth HR-HSI bands by the same simulated spectral response function R as the Nikon D700 camera, and the LR-HSI and LR-MSI are obtained by bicubicly downsampling the corresponding HR-HSI and HR-MSI by a factor of d. We select the first 22 samples and randomly extract 64×64 overlapping blocks from them as reference HR-HSI for training. Therefore, the sizes of HR-HSI, HR-MSI, LR-HSI, and LR-MSI are 64×64×31, 64×64×3, and LR-MSI, respectively. and The remaining 10 samples in the database are used for testing.
[0090] The Pavia Centre dataset was taken by the Reflection Optical System Imaging Spectrometer (ROSIS) sensor and is an airborne HSI taken in the central area of Pavia, northern Italy in 2001. It has 102 spectral channels and the spectral range is from 430 to 860nm. Pavia Centre contains a total of 1096×1096 pixels, but some areas have no information, so we remove them and select only 1096×715 valid pixels. The upper left 1024×128 pixels are cropped as test data, and the rest of the dataset is selected as training data. We follow the same settings as SSJSR to generate HR-MSI. We randomly extract 64×64 overlapping blocks from the training data as reference HR-HSI for training. Therefore, the sizes of HR-HSI, HR-MSIs, LR-HSI, and LR-MSI used are 64×64×102, 64×64×4, and
[0091] Then compare with SOTA:
[0092] Results on CAVE dataset: The average results of PSNR, SSIM, SAM and ERGAS of all comparison methods on CAVE dataset are reported in Table 1, where bold indicates the best result and underline indicates the second best. We can easily observe that the proposed SSFIN method significantly outperforms other algorithms in all objective evaluation indicators at all upsampling scales. For example, the average PSNR value of our method is 0.55dB higher than the second best method at ×2 scale, 0.35dB higher at ×4 scale, and 0.46dB higher at ×8 scale. Excluding our method, no method can lead other comparison methods in all objective evaluation indicators. This shows that the comparison methods only focus on restoring certain aspects of the image and cannot fully restore the spatial and spectral information of the image at the same time.
[0093]
[0094]
[0095] Table 1 Quantitative comparison of the four PQIS of the contrast methods on the 10 test images of the CAVE dataset
[0096] In order to compare the visual reconstruction results of different methods, we apply the spectral response function R to the reconstructed HR-HSI to generate RGB images of three test images. Figure 4 It can be seen that the synthetic image obtained by SSFIN is closest to the ground truth, while the results of other methods usually contain obvious erroneous structures. Obviously, the proposed method performs best in recovering the details of the original HSI and achieves the smallest reconstruction error.
[0097] Results on the Pavia Centre dataset: Table 2 lists the average performance of all the comparison methods in terms of PSNR, SSIM, SAM, and ERGAS of the test images. From this table, we can see that our SSFIN outperforms other comparison methods when the upsampling factor is d=2 and d=4. When the upsampling factor is d=8, AWAN+SSPSR is slightly better than our method. However, when we show the visual results restored by the comparison methods, such as Figure 5 As shown, we can observe that although the proposed method does not achieve the best objective results, our subjective results are indeed better than the comparison methods.
[0098]
[0099]
[0100] Table 2 Quantitative comparison of the four PQIS methods on the Pavia Centre dataset
[0101] In order to prove the credibility of the spectral reconstruction results, Figure 6 In , we plot the spectral vectors reconstructed by six contrasting methods at four randomly selected locations. The spectral vectors restored by the proposed SSFIN are closer to the ground truth, which means that our method can preserve the spectral information well.
[0102] Then, we conduct ablation analysis:
[0103] We further investigate the performance of our model through ablation studies and summarize them in Table 3.
[0104]
[0105] Table 3 Effectiveness analysis of feature interaction learning and regularization branches
[0106] Role of feature interaction learning: To verify the effectiveness of the proposed SSFIB, we use normal residual connections (i.e., spatial residual connections and spectral residual connections) to replace SSFIB. In this case, there is no interaction between spatial and spectral SR. For fairness, we make the network parameters of the two structures as close as possible. Table 3 shows the average results of 10 test images on spatial × 2 super-resolution. By comparing the fifth and sixth rows, we can observe that when we incorporate SSFIB, the performance of the proposed SSFIN is improved in all four image quality indicators, for example, the PSNR is +0.34dB. As for other objective indicators, the improvement is also considerable. This shows that the spatial super-resolution branch and the spectral super-resolution branch can indeed guide each other and improve the feature learning ability.
[0107] Effectiveness of Regularization Branches: To verify the effectiveness of the two regularization branches, we report the performance of SSFIN with different regularization branches. By comparing the second, third, fifth, and sixth rows in Table 4, we can observe that when both α and β are set to 0, the PSNR value (×2) on CAVE is relatively low. Using a single spatial regularization branch or a spectral regularization branch can appropriately improve the network performance. When both branches are used, the network performance is greatly improved. This shows that the spatial super-resolution task and the spectral super-resolution task are related and can truly guide each other to improve the network performance. In addition, the performance of all methods is higher than the baseline (first row), indicating that all structures in the network are useful.
[0108] Effectiveness of parameter sharing: Table 4 shows the performance of the proposed SSFIN with and without parameter sharing. We can see that sharing the parameters of SSRGs can reduce the parameters of the model and improve the performance of the network.
[0109]
[0110] Table 4 Effectiveness analysis of shared parameters
[0111] Finally, the model complexity and running time analysis are performed:
[0112] We show the model size, test time and performance of different methods on CAVE(×2) in Table 5. It is worth noting that due to the different spatial and channel sizes of the images input to the network, the amount of computation changes when the order of RCAN and AWAN is exchanged, but the number of network parameters does not change. In addition, for the comparison method AWAN+RCAN, in order to extract more features, we increase the number of convolution kernels in the convolution layer of RCAN from 64 to 128. It can be seen from the table that SSJSR has a small number of parameters, but its performance is the worst. Although our SSFIN achieves the best performance, its number of parameters is less than that of the two-stage method. Although SSJSR has a small number of parameters, it uses 3D convolutions and its FLOPs are in the same order of magnitude as other methods. Our method has the least FLOPs. Since AWAN and SAN contain non-local layers, the required physical memory increases dramatically as the spatial size of the input image increases, and larger memory is required for testing. Due to our hardware resource limitations, we only report its CPU time. In summary, SSFIN makes a better trade-off between model size and performance.
[0113]
[0114] Table 5 Comparison of parameters, computation, running time and PSNR performance on CAVE(×2)
[0115] In this paper, we present a deep spatial-spectral feature interaction network (SSFIN) for spatial-spectral super-resolution to simultaneously upsample the spatial and spectral resolutions of LR-MSI. We introduce two auxiliary tasks (i.e., spatial super-resolution and spectral super-resolution) to help the network better recover HR-HSI. In particular, we design a spatial-spectral feature interaction block (SSFIB) so that spatial super-resolution and spectral super-resolution can guide each other. These two tasks can provide prior knowledge for the final HR-HSI reconstruction. In addition, we use a weight decay strategy to train SSFIN, so that SSFIN can gradually shift its attention from the auxiliary tasks to the main task. Evaluations on three public HSI datasets show that the proposed SSFIN achieves state-of-the-art performance in terms of quantitative results and visual quality.
Claims
1. A method for simultaneous spatial-spectral super-resolution reconstruction based on multi-task interactive learning, characterized in that: The method for simultaneous spatial-spectral super-resolution reconstruction based on multi-task interactive learning comprises the following steps: Step 1: Establish the SSFIN network architecture; Step 2: Use weight decay strategy to train SSFIN network; In step one, the following steps are included: Step 11: Establish three branches in the SSFIN network: SISR network, SSR network and spatial spectral fusion network. The three branches generate three images respectively: HR-MSI, LR-HSI and HR-HSI. The three images are used as the output of the SSFIN network, and LR-MSI is used as the input of the SSFIN network. Step 1 and 2: Spatial super-resolution branch: First, convolutional layers are used to extract shallow spatial features from LR-MSI input. Then, N spatial residual groups SpaRGs are used to extract deep spatial features. A long skip connection is added to retain the original low-frequency information of LR-MSI. Each spatial residual group consists of several residual spatial attention blocks RSAB and a short skip connection to explore the spatial relationship of feature maps. Finally, a spatial super-resolution head module consisting of two convolutional layers and an upsampling model is used to reconstruct HR-MSI. The SISR network is optimized using a loss defined as L1: in is the output of the SISR network, and U is the ground-true HR-MSI; Step 13: Spectral super-resolution branch: First, a convolutional layer is used to extract shallow spectral features, and then N spectral residual groups SpeRGs are used to extract deep spectral features. Each spectral residual group consists of several residual channel attention blocks RCAB. The residual channel attention block applies a global average pooling layer to learn the weight of each channel. Finally, the spectral super-resolution head module composed of a convolutional layer is used to reconstruct LR-HSI. Apply L1 loss to train the SSR network: in is the output of the SSR network, V is the ground-true LR-HSI; Step 14: Spatial-spectral feature interaction module: In the kth SSFIB: Where Conv represents the convolution operation, Concat represents the cascade operator, and H SpaRG , H SpeRG and H fm Represent the spatial residual group, spectral residual group and fusion module respectively. and Represent the output of the kth spatial residual group, spectral residual group and fusion module respectively, Set To fuse the features extracted from the spatial residual group, spectral residual group, and previous fusion module, first and are concatenated, where 0 < n < k. Then, a fusion module containing two convolutional layers and two rectified linear units ReLU is used to integrate the spatial and spectral features. Next, convolutional layers are applied to extract the unique features of the SISR network and SSR network respectively. Finally, their unique features are added to the original features; Step 15: Spatial-spectral fusion network: First, a convolutional layer is used to integrate the features from the three branches. Then, an upsampling model is used to make these features have the same spatial scale as HR-HSI. Then, G space-spectral residual groups SSRGs are applied to reconstruct the target HR-HSI. The dual residual attention block RDAB is used to connect spatial attention and channel attention. In the space-spectral residual group, the dual residual attention block is applied to extract more useful spatial and spectral features at the same time. After that, a convolutional layer is used to integrate the features extracted by the network at different stages. This is because the deep neural network extracts features with different information at different depths and shares the parameters of each empty spectrum residual group. After G empty spectrum residual groups, a convolutional layer is used to reconstruct the output target. Use L1 loss to train the spatial-spectral fusion network: in is the output of the fusion network, and X is the ground-truth HR-HSI.
2. The method for simultaneous spatial-spectral super-resolution reconstruction based on multi-task interactive learning according to claim 1, characterized in that: In step 2, specifically, the spatial super-resolution network and the spectral super-resolution network are regarded as two regularized branches, and the weight decay strategy is used to train SSFIN. The total loss function of SSFIN is: Where M and m represent the total number of training rounds and the current number of training rounds, respectively. α and β are trade-off parameters. From the definition of the loss function, we know that once L spq and L spe The weight of is decayed to zero, that is, the learned spatial and spectral features are used to help the joint reconstruction of HSI.
3. The method for simultaneous spatial-spectral super-resolution reconstruction based on multi-task interactive learning according to claim 2, characterized in that: During the training process, specifically, in the spatial super-resolution branch, the number of spaRGs is set to N=3, and in each spaRG, the number of RSABs is set to 5. In the spectral super-resolution branch, the number of speRGs is set to N=3, and in each speRG, the number of RCABs is set to 5. In the spatial-spectral fusion branch network, the number of SSRGs is set to 4, and each SSRG has 5 RDABs. The convolution kernel sizes of all convolutional layers are set to 3×3, α and β are set to 0.5, and M is set to 100. The network is trained using the Adam optimizer for 250,000 iterations with a batch size of 8. Each epoch is set to 2500 iterations. The learning rate is initialized to 0.0008 and decayed by a factor of 2 when reaching 5, 15, 30, and 60 epochs. The network is implemented and trained using the Pytorch framework and NVIDIA Tesla V100 GPU.
Citation Information
Patent Citations
Hyperspectral spatial-spectral joint feature extraction method based on transfer learning
CN112580670A
Hyperspectral and panchromatic image fusion method based on CNN and Laplace pyramid
CN112669248A
A hyperspectral image super-resolution realization method and system
CN112767243A
Hyperspectral image super-resolution reconstruction method and device and electronic equipment
CN113139902A
Systems and Methods for Multi-Spectral Image Fusion Using Unrolled Projected Gradient Descent and Convolutinoal Neural Network
US20200302249A1
