A step-by-step spectral super-resolution method based on spectral back-projection residual

Abstract

The present invention discloses a step-by-step spectral super-resolution method based on spectral back-projection residual, and the steps include: step 1, pre-processing the training data to obtain a new training set; step 2, constructing a spectral reconstruction module, and outputting a feature image I1; step 3, constructing a spectral back-projection residual block, and outputting a feature image I2; step 4, performing step-by-step spectral reconstruction of the second stage and the third stage in the spectral super-resolution network; step 5, optimizing the spectral super-resolution network, using the average relative absolute error to calculate the total loss function during the spectral reconstruction network training, and optimizing the proposed spectral reconstruction network by continuously minimizing the total loss. The method of the present invention makes full use of the loss of the back-projected RGB and the original input RGB to refine the reconstructed hyperspectral image, and the accuracy is significantly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of hyperspectral image processing and relates to a step-by-step spectral super-resolution method based on spectral back-projection residual. Background Art

[0002] Hyperspectral imaging can not only obtain spatial information in the scene, but also capture rich spectral information from hundreds or thousands of continuous spectra. Its spectral characteristics have been proven to be effective in many fields, such as image segmentation, medical diagnosis, environmental detection, target tracking, etc. At present, the methods for obtaining hyperspectral images can be roughly divided into three categories: 1) Fusion-based methods, which fuse high spatial resolution RGB images and low spatial resolution hyperspectral images to produce high-quality hyperspectral images; 2) Single image super-resolution, which can directly improve the spatial resolution of hyperspectral images by spatial super-resolution methods; 3) Spectral super-resolution, which can generate hyperspectral images by improving the spectral resolution of a given RGB image, also known as spectral reconstruction. Although all three methods can obtain high-quality hyperspectral images, the first two methods require dedicated hyperspectral sensors and expensive imaging costs, while the third method can easily capture RGB images only through traditional RGB cameras. This imaging method can greatly reduce the cost of capturing hyperspectral images. Therefore, the method of generating hyperspectral images through spectral super-resolution of RGB images has attracted widespread attention and research.

[0003] Existing spectral super-resolution methods can be roughly divided into two categories, based on traditional methods and deep learning methods. The direction based on traditional methods mainly focuses on using the prior information of potential hyperspectral images to learn the three-to-many mapping relationship from RGB images to hyperspectral images through sparse recovery, low-rank tensor recovery, and some shallow mapping models. Since the prior information of hyperspectral images is often a shallow structure designed manually or heuristically, these methods have not been well promoted in practical applications.

[0004] In recent years, with the rapid development of deep learning, spectral super-resolution methods based on convolutional neural networks (CNNs) have been widely used. Galliani et al. first introduced deep learning into the spectral super-resolution task based on Densenet and designed 56 convolutional layers to restore hyperspectral images. Xiong et al. upsampled the spectral dimension of RGB images and then used a global residual learning strategy to reconstruct hyperspectral images. Li et al. designed a hybrid 2D-3D deep residual attention network to make full use of spatial-spectral context information to improve the accuracy of spectral super-resolution. Although these CNN-based methods have made up for the shortcomings of traditional methods to a certain extent, most of them focus on the nonlinear expression ability of the network and use CNN to learn the complex mapping relationship from RGB to hyperspectral images at one time. They do not fully consider that mapping from 3 channels of RGB to 31 channels of hyperspectral images is a serious ill-posed problem, which limits the accuracy of spectral super-resolution algorithms to restore hyperspectral images. Summary of the invention

[0005] The purpose of the present invention is to provide a step-by-step spectral super-resolution method based on spectral back-projection residual, which solves the problem of insufficient accuracy in existing spectral super-resolution methods, in which the hyperspectral image is restored by one-time learning of the ill-posed complex mapping relationship between RGB and the hyperspectral image.

[0006] The technical solution adopted by the present invention is a step-by-step spectral super-resolution method based on spectral back-projection residual, which is specifically implemented according to the following steps:

[0007] Step 1: Preprocess the training data.

[0008] The dataset used is NTIRE 2020. The training images in this dataset consist of 450 pairs of RGB images and HSI images with a spatial resolution of 512*482. The paired RGB and HSI images are randomly cropped to 256*256 to obtain a new training set of 450 pairs of RGB images with a size of 256*256*3 and HSI images with a size of 256*256*31.

[0009] Step 2: Build a spectrum reconstruction module.

[0010] The input of the spectrum reconstruction module is the original RGB image with a size of 256*256*3, and the output of the spectrum reconstruction module is a feature image I1 with a size of 256*256*8;

[0011] Step 3: Construct the spectral back-projection residual block.

[0012] The input of the spectral back-projection residual block is the feature image I1 obtained in step 2, which has a size of 256*256*8; the output of the spectral back-projection residual block is the feature image I2, which also has a size of 256*256*8;

[0013] Step 4, performing step-by-step spectrum reconstruction in the second and third stages of the spectrum super-resolution network;

[0014] Step 5: Optimize the spectral super-resolution network.

[0015] The total loss function of the spectral reconstruction network training is calculated using the mean relative absolute error, and the total loss L is continuously minimized. MRAE To optimize the proposed spectral reconstruction network, we have it.

[0016] The beneficial effect of the present invention is that the number of spectral channels can be gradually increased in three stages through the proposed step-by-step spectral super-resolution method based on spectral back-projection residual, thereby effectively realizing end-to-end mapping from RGB image to hyperspectral image. In addition, the present invention can back-project the reconstructed hyperspectral image into an RGB image through the spectral back-projection residual block, and make full use of the loss of the back-projected RGB and the original input RGB to refine the reconstructed hyperspectral image, and the accuracy is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of the overall method of the present invention;

[0018] Figure 2 is a structural flow chart of a spectrum reconstruction module constructed by the method of the present invention;

[0019] Figure 3 It is a structural flow chart of the spectral back-projection residual block constructed by the method of the present invention;

[0020] Figure 4 The method of the present invention is based on Figure 2 Structural flowchart of the channel-wise spatial attention block in [5]. DETAILED DESCRIPTION

[0021] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Reference Figure 1The method of the present invention is based on a step-by-step spectral super-resolution network of spectral back-projection residual, hereinafter referred to as spectral super-resolution network, which consists of three stages of spectral reconstruction modules and spectral back-projection residual blocks. The spectral image is reconstructed by the spectral reconstruction module, and then the spectral back-projection residual block is used to back-project the reconstructed spectral image into an RGB image to refine the reconstructed image; wherein the spectral reconstruction module of the first stage (Stage 1) directly takes the original RGB image as input, uses a double residual structure and a channel space attention block to generate an 8-channel spectral image, and then the spectral back-projection residual block back-projects the 8-channel spectral image into an RGB image. The image loss between the back-projected RGB and the original input RGB is upsampled and formed with the 8-channel spectral image to form residual learning and added to the input of the spectral reconstruction module of the second stage (Stage2); the spectral reconstruction modules and spectral back-projection residual blocks of the second stage (Stage2) and the third stage (Stage3) are the same as the structural principles of the first stage, the difference is that the number of channels restored in the three stages is inconsistent; the first stage is restored to 8 channels, the second stage is restored to 18 channels, and the third stage is restored to 31 channels. Finally, through the gradual reconstruction of the three stages, the output channel number is 31. Hyperspectral image.

[0023] The method of the present invention is implemented by using the above-mentioned spectrum super-resolution network according to the following steps:

[0024] Step 1: Preprocess the training data.

[0025] The dataset used in this step is NTIRE 2020. The training images in this dataset consist of 450 pairs of RGB images and HSI images (hyperspectral label images) with a spatial resolution of 512*482. In order to reduce the complexity and computational complexity, the paired RGB and HSI images are randomly cropped to 256*256, and a new training set of 450 pairs of RGB images with a size of 256*256*3 and HSI images with a size of 256*256*31 is obtained.

[0026] Step 2: Build a spectrum reconstruction module.

[0027] The input of the spectral reconstruction module is the original RGB image with a size of 256*256*3, and the output of the spectral reconstruction module is a feature image I1 with a size of 256*256*8; the main function of the spectral reconstruction module is to reconstruct the input original RGB image into an 8-channel spectral image to obtain the feature image I1;

[0028] Reference Figure 2,The structural process of the spectral reconstruction module is as follows: original RGB image (Input image) as input image → Conv 3*3 layer → PReLU layer → Conv 3*3 layer → CSAB layer → residual structure layer → PReLU layer → Conv 3*3 layer → PReLU layer → Conv 3*3 layer → CSAB layer → residual structure layer → Conv 3*3 layer → output feature image I1 (Output feature);

[0029] Among them, each Conv 3*3 layer is a convolution operation, the convolution kernel size is 3*3, the convolution step is 1, the padding value is 0, and the number of feature maps is 8; three PReLu layers are used as activation functions to introduce more nonlinearity and accelerate convergence; the structural flow of each CSAB layer (full name channel space attention block) is as follows Figure 4 As shown in the figure, it mainly includes channel attention block (ChannelAttention) and spatial attention block (SpatialAttention). The channel attention block compresses the features of the spatial dimension through the average pooling (GlobalPooling) operation, and then obtains 8 scalars between 0 and 1 through Conv 1*1-ReLU-Conv 1*1-Sigmoid. These scalars representing the correlation between feature channels are used as the weight of the channel and the weight is multiplied by each feature channel before average pooling. The obtained feature information can better fit the complex correlation between channels; the spatial attention block compresses the features of the channel dimension through the maximum pooling (Max Pooling) operation, and then uses the spatial relationship of the features through the Conv 3*3-ReLU-Conv 3*3-Sigmoid operation to generate a spatial attention map. The spatial attention map will encode the emphasized or suppressed areas, so as to better utilize the spatial information; the dual residual structure forms dual residual learning through long and short jump connections, which ensures that while making full use of the rich shallow feature information of the original RGB image, it can also solve the problems of gradient disappearance and explosion in the deep network.

[0030] Step 3: Construct the spectral back-projection residual block.

[0031] The input of the spectral back-projection residual block is the feature image I1 obtained in step 2, which has a size of 256*256*8; the output of the spectral back-projection residual block is the feature image I2, which also has a size of 256*256*8. The main function of the spectral back-projection residual block is to back-project the reconstructed spectral image into an RGB image to refine the reconstructed image;

[0032] Reference Figure 3, the structural process of the spectral back-projection residual block is as follows: the feature image I1 output in step 2 is used as the input feature (Input Feature) → Conv11*1 layer (generates the back-projected RGB image) → calculates the loss layer (the loss of the back-projected RGB image and the original RGB image) → Conv21*1 layer → residual structure layer → output is the feature image I2 (OutputFeature);

[0033] Among them, the Conv11*1 layer is a convolution operation, the convolution kernel size is 1*1, the convolution step size is 1, the padding value is 0, the number of feature maps is 3, and the 8-channel feature image output in step 2 is spectrally downsampled to generate a 3-channel back-projected RGB image using 1*1 convolution; the original RGB image is input to calculate its loss with the back-projected RGB image, so as to further refine the reconstructed image by minimizing the loss; the Conv21*1 layer is a convolution operation, the convolution kernel size is 1*1, the convolution step size is 1, the padding value is 0, the number of feature maps is 8, and the 1*1 convolution is used for spectral upsampling to restore the back-projected 3-channel RGB image to an 8-channel feature image; the residual structure layer is to perform a residual connection between the 8-channel feature image after spectral upsampling in this step and the 8-channel feature image output in step 2, which can not only solve the gradient disappearance problem, but also make full use of the feature information output in step 2; after residual learning, the feature image I2 is output.

[0034] Step 4: Perform the step-by-step spectrum reconstruction of the second stage (Stage 2) and the third stage (Stage 3) in the spectrum super-resolution network.

[0035] The modules and operation steps included in the second stage (Stage2) and the third stage (Stage3) are the same as those in the first stage (Stage1). The difference is that the number of spectral channels generated by the spectral reconstruction module in the second stage Stage2 is 18, and then the 18-channel spectral image is back-projected into a 3-channel RGB image in the spectral back-projection residual block; and the third stage (Stage3) generates a 31-channel hyperspectral image through the spectral reconstruction module, and then the 31-channel hyperspectral image is back-projected into a 3-channel RGB image in the spectral back-projection residual block. Through the step-by-step spectral reconstruction process in three stages, the reconstructed spectral image is repeatedly projected into an RGB image to continuously refine the reconstructed hyperspectral image.

[0036] Step 5: Optimize the spectral super-resolution network.

[0037] In this step, the mean relative absolute error (MRAE) is used to calculate the total loss function during the spectral reconstruction network training, and the total loss L is continuously minimized. MRAE To optimize the proposed spectral reconstruction network;

[0038] The total loss function is used in this step, and its expression is: L MRAE＝ L rec +L p

[0039] L rec is the hyperspectral reconstruction loss, and its expression is:

[0040] Where N is the total number of pixels, is the p-th pixel value of the hyperspectral label image, is the pth pixel value of the hyperspectral reconstructed image;

[0041] L p is the back-projection refinement loss. It is worth noting that L p =L1+L2+L3 refers to the sum of the back-projection refinement losses of the three stages (Stage1, Stage2, Stage3), and its expression is:

[0042]

[0043] Where N is the total number of pixels, is the pth pixel value of the input original RGB image, is the p-th pixel value of the back-projected RGB image.

Claims

1. A step-by-step spectral super-resolution method based on spectral back-projection residual, characterized in that: Follow these steps to implement it: Step 1: Preprocess the training data. The dataset used is NTIRE 2020. The training images in this dataset consist of 450 pairs of RGB images and HSI images with a spatial resolution of 512*482. Randomly crop the paired RGB and HSI images into 256*256, and get a new training set of 450 pairs of RGB size 256*256*3 and HSI size 256*256*31; Step 2: Build a spectrum reconstruction module. The input of the spectrum reconstruction module is the original RGB image with a size of 256*256*3, and the output of the spectrum reconstruction module is a feature image I1 with a size of 256*256*8; Step 3: Construct the spectral back-projection residual block. The input of the spectral back-projection residual block is the feature image I1 obtained in step 2, with a size of 256*256*8; the output of the spectral back-projection residual block is the feature image I2, also with a size of 256*256*8, completing the first stage of spectral reconstruction; Step 4: Perform the step-by-step spectrum reconstruction of the second and third stages in the spectrum super-resolution network. The second stage repeats steps 2 and 3, and the third stage repeats steps 2 and 3 again. The difference is: In the second stage, the number of spectral channels generated by the spectral reconstruction module is 18. Then, in the spectral back-projection residual block, the 18-channel spectral image is back-projected into a 3-channel RGB image; The third stage is to generate a 31-channel hyperspectral image through the spectral reconstruction module, and then the 31-channel hyperspectral image is back-projected into a 3-channel RGB image in the spectral back-projection residual block. Thus, the reconstructed spectral image is repeatedly projected into RGB image to continuously refine the reconstructed hyperspectral image in the three-stage step-by-step spectral reconstruction process; Step 5: Optimize the spectral super-resolution network. The total loss function of the spectral reconstruction network training is calculated using the mean relative absolute error, and the total loss L is continuously minimized. MRAE To optimize the proposed spectral reconstruction network, we have it.

2. The step-by-step spectral super-resolution method based on spectral back-projection residual according to claim 1, characterized in that: The structural process of the spectral reconstruction module is as follows: original RGB image as input image → Conv 3*3 layer → PReLU layer → Conv 3*3 layer → CSAB layer → residual structure layer → PReLU layer → Conv 3*3 layer → PReLU layer → Conv3*3 layer → CSAB layer → residual structure layer → Conv 3*3 layer → output feature image I1; Among them, each Conv 3*3 layer is a convolution operation, the convolution kernel size is 3*3, the convolution step is 1, the padding value is 0, and the number of feature maps is 8; three PReLu layers are used as activation functions; the structural process of each CSAB layer includes a channel attention block and a spatial attention block, in which the channel attention block compresses the features of the spatial dimension through the average pooling operation, and then passes through Conv 1*1-ReLU-Conv 1*1-Sigmoid to obtain 8 scalars between 0 and 1, and then uses these scalars representing the correlation between feature channels as the channel weight and multiplies the weight by each feature channel before average pooling; the spatial attention block compresses the features of the channel dimension through the maximum pooling operation, and then uses the Conv 3*3-ReLU-Conv 3*3-Sigmoid operation to use the spatial relationship of the features to generate a spatial attention map, which will encode the emphasized or suppressed areas; the dual residual structure forms dual residual learning through long and short jump connections.

3. The step-by-step spectral super-resolution method based on spectral back-projection residual according to claim 1, characterized in that: The structural process of the spectral back-projection residual block is as follows: the feature image I1 output in step 2 is used as the input feature → Conv1 1*1 layer → calculate the loss layer and the loss of the original RGB image → Conv2 1*1 layer → residual structure layer → output as feature image I2; Among them, the Conv1 1*1 layer is a convolution operation, the convolution kernel size is 1*1, the convolution step is 1, the padding value is 0, the number of feature maps is 3, and the 1*1 convolution is used for spectral downsampling to back-project the 8-channel feature image output in step 2 into a 3-channel RGB image; Conv2 1*1 layer is a convolution operation, the convolution kernel size is 1*1, the convolution step is 1, the padding value is 0, the number of feature maps is 8, and 1*1 convolution is used for spectral upsampling to restore the back-projected 3-channel RGB image to an 8-channel feature image; The residual structure layer performs a residual connection between the 8-channel feature image after spectral upsampling in this step and the 8-channel feature image output in step 2, and outputs the feature image I2 after residual learning.

4. The step-by-step spectral super-resolution method based on spectral back-projection residual according to claim 1, characterized in that: In the step 5, Using the total loss function, the expression is: L MRAE＝ L rec +L p , L rec is the hyperspectral reconstruction loss, expressed as: Where N is the total number of pixels, is the p-th pixel value of the hyperspectral label image, is the pth pixel value of the hyperspectral reconstructed image; L p is the back-projection refinement loss. It is worth noting that L p =L1+L2+L3 refers to the sum of the back-projection refinement losses in the three stages, expressed as: Where N is the total number of pixels, is the pth pixel value of the input original RGB image, is the p-th pixel value of the back-projected RGB image.

Images