Hyperspectral image super-resolution reconstruction method based on double-branch network
By designing a dual-branch feature reconstruction network and utilizing global and local reconstruction modules, information interaction and fusion between hyperspectral and multispectral images are achieved, solving the problem of information destruction in existing technologies and improving reconstruction results.
Patent Information
- Application Number
- CN202410873115.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-01
- Publication Date
- 2024-11-26
AI Technical Summary
Existing hyperspectral super-resolution reconstruction techniques destroy the inherent prior information of images when fusing hyperspectral and multispectral images, making it difficult to fully utilize their respective advantages and achieve efficient information interaction and fusion.
A bi-branch feature reconstruction network was designed, which includes a global attention module, a feature interaction module, and a local reconstruction unit. The feature fusion module integrates the bi-branch feature information and preserves the inherent prior information of hyperspectral and multispectral images.
It significantly improves the super-resolution reconstruction effect of hyperspectral images, with excellent image reconstruction results and high fidelity in texture details and spectral information.
Smart Images

Figure CN119027314A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of hyperspectral image processing, and in particular relates to a hyperspectral image super-resolution reconstruction method based on a double-branch network. Background Art
[0002] Hyperspectral images record the continuous spectral bands of the target object from visible light to near-infrared wavelengths, providing rich spectral information, which gives it unique advantages in detection and classification operations. Therefore, hyperspectral images have played an important role in many fields such as mineral exploration, environmental monitoring, and medical diagnosis. However, due to the physical limitations of spectral imagers, improving the spectral resolution of hyperspectral images often comes at the expense of spatial resolution, limiting its practical application potential. In order to solve this problem, hyperspectral super-resolution reconstruction technology has emerged. Among them, a mainstream technology is to generate a hyperspectral image with high spatial resolution by fusing a hyperspectral image with low spatial resolution with a multispectral image with high spatial resolution in the same scene. Existing hyperspectral image super-resolution technologies of this type can be roughly divided into two categories: methods based on variational optimization and methods based on deep learning.
[0003] Variational optimization-based methods usually formulate the fusion process of hyperspectral images and multispectral images as a constrained optimization problem with specific priors, and obtain the required hyperspectral images by establishing an optimization model and iteratively solving it. Although such methods can achieve good results, specific prior information is required before establishing the optimization model. This information is usually scene-related and needs to be fine-tuned to adapt to different application scenarios.
[0004] Methods based on deep learning usually construct end-to-end deep neural networks, which effectively learn the potential relationship between input images and output images through their powerful feature extraction capabilities, thereby achieving excellent reconstruction results and are favored by a large number of researchers. However, most existing methods directly stack the upsampled hyperspectral image with the multispectral image, and then input them into the designed deep neural network to reconstruct the image. This method destroys the inherent spectral information prior of the hyperspectral image and the inherent spatial information prior of the multispectral image. Therefore, researchers proposed a method to input the two images into the deep neural network in parallel and fuse the image information through a two-stream network. The technical difficulty of this method lies in how to make full use of the respective advantages of hyperspectral images and multispectral images, and efficiently realize the information interaction and fusion between the two-stream networks. Summary of the invention
[0005] In order to overcome the shortcomings of the prior art, the present invention proposes a hyperspectral image super-resolution reconstruction method based on a dual-branch network. The method designs a dual-branch feature reconstruction network including a global attention module, a feature interaction module and a local reconstruction unit. Among them, the global attention module analyzes the image characteristics of each branch from the overall perspective of the image for reconstruction. In the process of dual-branch feature reconstruction, the feature interaction module realizes information interaction between the two branches, so that each branch can learn the useful information of the other branch. The local reconstruction unit analyzes the internal characteristics of the image from the local perspective of the image for reconstruction. In addition, a feature fusion module is designed, which completes the integration of the dual-branch feature information by weighted summing the dual-branch features. This method gives full play to the respective characteristics of hyperspectral images and multispectral images, and significantly improves the effect of hyperspectral image super-resolution reconstruction technology.
[0006] In order to achieve the above object, the technical solution adopted by the present invention comprises the following steps:
[0007] Step 1: Generate training data and test data based on the existing hyperspectral image dataset.
[0008] Step 2: Perform feature alignment on the training data.
[0009] Step 3: The feature-aligned data is input into multiple sets of serially connected dual-branch feature reconstruction networks for feature reconstruction.
[0010] Step 4: The dual-branch feature information after feature reconstruction is input into the feature fusion module for integration, and then added to the hyperspectral image after upsampling in step 2 to obtain the reconstructed image.
[0011] Step 5: The error between the reconstructed image and the true image in the training set is calculated using the L1 loss function, and the network parameters are updated using the gradient descent algorithm.
[0012] Step 6: Repeat steps 2 to 5 until the model parameters reach the optimal state. And use the test data to test the model.
[0013] The data set described in step 1 includes hyperspectral image data sets such as CAVE, Harvard and Chikusei. 80% of the hyperspectral image data sets are randomly selected as training sets, and the rest are used as test sets. Due to the scarcity of hyperspectral image data, the training set data needs to be enhanced by random cropping, random flipping and random rotation. The hyperspectral images in the data set are regarded as true value images, and the hyperspectral images are blurred by applying Gaussian blur kernels, and then downsampled using the bicubic interpolation algorithm to generate low-resolution images.
[0014] The feature alignment method described in step 2 is: upsample the input low-resolution hyperspectral image through the bicubic interpolation algorithm to keep it consistent with the resolution of the multispectral image. Then, they are convolved with a convolution kernel size of 1 to keep the number of feature channels of the two consistent.
[0015] The number of dual-branch feature reconstruction networks described in step 3 is determined according to the actual task requirements. Each feature reconstruction network includes a global attention module, a feature interaction module, and a local reconstruction unit. The global attention module and the feature interaction module use jump connections to form short-range residual learning for easy network training.
[0016] The input of the global attention module is the hyperspectral image features and multispectral image features after feature alignment in step 2. The core of this module is the attention mechanism, which reconstructs the hyperspectral image features and multispectral image features in the spatial dimension and channel dimension respectively according to the respective characteristics of the two images. In order to reduce the computational complexity of the model, the hyperspectral image feature branch adopts a linear aggregation attention mechanism, and the multispectral image feature branch adopts a self-attention mechanism. Before the attention calculation, the Q vectors in the two branches are simply interacted through convolution, so that the Q vectors of both sides can learn the guidance information of the other branch that is beneficial to their own image reconstruction, so that the attention calculation results are more in line with the reconstruction task requirements.
[0017] The input of the feature interaction module is the dual-branch image features processed by the global attention module. The dual-branch image features calculated by the attention mechanism are spliced, and the interaction of the dual-branch features is realized through two sets of convolutions with a kernel size of 1, so that each branch can learn the effective information of the other branch, laying the foundation for subsequent feature fusion. After the interaction is completed, the module splits the fused information back to the original dual-branch form, so that the hyperspectral image features and multispectral image features can continue to be reconstructed separately in the subsequent processing process.
[0018] The local reconstruction unit is composed of multiple local reconstruction modules, the number of which depends on the actual task requirements. The input of this module is the image feature information of a single branch after the output of the feature interaction module. By performing channel scaling and using convolution kernels of different sizes to deeply mine the local information and detail features of the image, the model's ability to reconstruct image texture is improved.
[0019] The feature fusion module mentioned in step 4 takes as input the dual-branch feature information output by the dual-branch feature reconstruction network. This module designs a simple weight network, treats the reconstructed dual-branch features equally, assigns corresponding weights to each branch, and then completes the fusion by weighted summation of the dual branches, thereby efficiently integrating all image feature information. The fused feature information also needs to be added to the upsampled hyperspectral image in step 2 to form global residual learning, and finally obtain the reconstructed image.
[0020] Among them, the optimizer selected by the gradient descent algorithm in step 5 is selected according to the actual task requirements.
[0021] The number of repetitions in step 6 depends on actual task requirements.
[0022] Based on this, the present invention proposes a hyperspectral image super-resolution reconstruction method based on a dual-branch network, which improves the deficiencies in the prior art. The method adopts a dual-branch structure to retain the inherent prior information of hyperspectral images and multispectral images. It can reconstruct from both the global and local perspectives of the image, and realizes the efficient integration of dual-branch feature information through a feature fusion module. The method can achieve excellent reconstruction effects with a relatively low number of parameters. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a schematic diagram of the network structure of a hyperspectral image super-resolution reconstruction method based on a dual-branch network proposed in the present invention.
[0024] Figure 2 It is a schematic diagram of the network structure of the global attention module proposed in the present invention.
[0025] Figure 3 It is a schematic diagram of the network structure of the feature interaction module proposed in the present invention.
[0026] Figure 4 It is a schematic diagram of the network structure of the local reconstruction module proposed in the present invention.
[0027] Figure 5 It is a schematic diagram of the network structure of the feature fusion module proposed in the present invention.
[0028] Figure 6 It is a schematic diagram of the objective indicator comparison results of the embodiment of the present invention and other cutting-edge methods on the CAVE dataset. DETAILED DESCRIPTION
[0029] In order to make the technical solution of the present invention clearer, a specific embodiment is now described in detail in conjunction with the accompanying drawings. The embodiments of the present invention include but are not limited to the following contents.
[0030] like Figure 1As shown, the main body of the hyperspectral image super-resolution reconstruction method based on a dual-branch network proposed in the present invention is composed of multiple sets of dual-branch feature reconstruction networks connected in series and a feature fusion module. The number of dual-branch feature reconstruction networks depends on the actual task requirements, and in this embodiment, the number is 3. Each reconstruction network is composed of a global attention module, a feature interaction module, and a local reconstruction unit in turn. The specific implementation steps of the embodiment are as follows:
[0031] Step 1: Generate training data and test data based on the existing hyperspectral image dataset.
[0032] First, the embodiment uses the CAVE dataset to generate training data and test data. The training set contains 20 hyperspectral images and multispectral images in their corresponding scenes, and the test set contains 11 hyperspectral images and multispectral images (HRMSI) in their corresponding scenes. In order to expand the training data, data enhancement operations such as random cropping, random flipping, and random rotation are performed on the training set images to generate 3920 sets of image pairs of size 64×64 containing hyperspectral images and multispectral images. The hyperspectral images in the training set and the hyperspectral images in the test set are regarded as high-resolution hyperspectral images (HRHSI), that is, true value images.
[0033] Then, the HRHSI in the training set and the test set are processed. First, they are blurred using a Gaussian blur kernel with a standard deviation of 2, and then downsampled using a bicubic linear interpolation algorithm to generate the corresponding low-resolution hyperspectral image (LRHSI). In this way, a pair of low-resolution and high-resolution hyperspectral images is formed for subsequent experiments.
[0034] Step 2: Perform feature alignment on the training data.
[0035] Let LRHSI be y∈R C×h×w ,HRMSI is z∈R c×H×W Among them, C, h, w, c, H and W are the number of channels, length and width of LRHSI and HRMSI respectively. First, y is upsampled to the target multiple through the bicubic algorithm to obtain the blurred high-resolution hyperspectral image y U ∈R C×H×W , y U The feature alignment is completed by convolution with a convolution kernel size of 1 with z to obtain y U 0∈R F×H×W and z0∈R F×H×W , in this embodiment, the F value is 64.
[0036] Step 3: The feature-aligned data is input into multiple sets of serially connected dual-branch feature reconstruction networks for feature reconstruction.
[0037] The number of dual-branch feature reconstruction networks is determined according to actual task requirements, and three are used in the embodiment. Taking one set of dual-branch feature reconstruction networks as an example, the feature reconstruction process is as follows:
[0038] Step 3.1: Set y U 0 and z0 are input into the global attention module for global feature reconstruction.
[0039] The global attention module is as follows Figure 2 As shown, first, y U 0 and z0 are layer-normalized to help speed up the network training process and avoid gradient vanishing or gradient exploding problems during the training process. U 0 and z0 perform global attention modeling in two different ways.
[0040] y U 0 is y U The feature map of y needs to be supplemented by the feature reconstruction network. In the global attention module, the present invention U 0 performs global attention modeling in the spatial dimension. Through three mapping functions consisting of convolution kernel size 1, the layer-normalized y U 0 is projected to obtain Q∈R F×H×W ,K∈R F×H×W and V∈R F ×H×W .
[0041] Since z0 is U 0 has guiding significance in the spatial dimension, and y U 0 also has guiding significance in the z0 channel dimension. Therefore, the y U 0 and z0 are concatenated according to the channel dimension, and a convolution with a convolution kernel size of 1 is used to obtain the supplementary vector q∈R 2F×H×W . Split the supplementary vector q into q1∈R by channel F×H×W ,q2∈R F×H×W Two complementary vectors, added back to the U 0 and z0 projected two Q vectors, so that y U When calculating attention, both 0 and z0 can pay more attention to the guidance information that is beneficial to their own reconstruction.
[0042] Transform the dimensions of Q, K and V to obtain a new Q∈R HW×F ,K∈R HW×F and V∈R HW×F Then, the focused linear attention is calculated for the Q, K and V matrices. The calculation formula is defined as:
[0043]
[0044] Among them, Sim(·) is the similarity calculation function, DWC(·) is the depth convolution, x **P represents the Pth power of x, and in this embodiment, P is 3. Finally, the attention calculation result is dimensionally converted to obtain the output y U’ 0∈R F×H×W .
[0045] Similarly, z0 is the feature map of z, and its missing channel feature information needs to be supplemented by the feature reconstruction network. Therefore, in the global attention module, the present invention performs global attention modeling on z0 in the channel dimension. Through three mapping functions consisting of convolution kernel size 1 and deep convolution kernel size 5, z0 after layer normalization is projected to obtain Q∈R F×H×W ,K∈R F×H×W and V∈R F×H×W .
[0046] In which, Q is added to the supplementary vector q2 as above to obtain the new Q vector. Q, K and V are transformed into the new Q∈R F×HW ,K∈R F×HW and V∈R F×HW Matrix. Then, the self-attention calculation is performed on the Q, K and V matrices. The calculation formula is defined as:
[0047]
[0048] Finally, the attention calculation result is transformed into the output z ’ 0∈R F×H×W .
[0049] It should be noted that the global attention module uses a jump connection outside, and the output result y described above U’ 0 and z ’ 0 also needs to be different from the input y U Add 0 and z0 to get the final output y of step 3.1 U 1∈R F×H×W and z1∈R F×H×W .
[0050] Step 3.2: Set y U 1 and z1 are input into the feature information interaction module for dual-branch feature information interaction.
[0051] Feature information interaction module such as Figure 3 As shown, first, y U1 and z1 are concatenated according to the channel dimension to obtain the fusion feature f∈R 2F×H×W , and then perform layer normalization, and obtain the interaction feature f after two convolutions with a kernel size of 1 and a GeLU activation function ’ ∈R 2F×H×W . ’ Split by channel to get the output f ’ y ∈R F×H×W and f ’ z ∈R F×H×W .
[0052] It should be noted that the feature information interaction module uses a skip connection outside. The output result f ’ y and f ’ z Also need to be respectively with the input y U Add 1 and z1 to get the final output y of step 3.2 U 2∈R F×H×W and z2∈R F×H×W .
[0053] Step 3.3: Set y U 2 and z2 are respectively input into the local reconstruction units in their respective branches for local feature reconstruction.
[0054] The local reconstruction unit is composed of multiple groups of local reconstruction modules. In this embodiment, the number of local reconstruction modules is 3. Taking a local reconstruction module of the z branch as an example, the reconstruction process is as follows:
[0055] Local reconstruction module such as Figure 4 As shown in Figure 1, z2 is first layer normalized. Then, it is convolved with a convolution kernel size of 1 to reduce the number of channels by 3 times to obtain z ’ 2∈R F / 3×H×W .
[0056] z ’ 2. Through three convolution kernels of different sizes, the features z under different receptive fields are obtained. ’ 21 ∈R F / 3×H×W 、z ’ 22 ∈R F / 3×H×W and z ’ 23 ∈R F / 3×H×W . Then, z ’ 21 、z ’ 22 and z ’23 Add together to get the fusion feature z ’ 2f ∈R F / 3×H×W .
[0057] Finally, z is convolved with a kernel size of 1. ’ 2f The number of feature channels is restored, and the output feature z is obtained through a GeLU activation function and a convolution kernel size of 1. 2f ∈R F×H×W , added to the input z2 to get the final output z3.
[0058] The above content is the overall reconstruction process of the feature reconstruction network. Since the number of feature reconstruction networks in the embodiment is 3. Therefore, the dual-branch feature finally outputted in step 3 is y U 9∈R F×H×W and z9∈R F×H×W .
[0059] Step 4: The dual-branch feature information after feature reconstruction is input into the feature fusion module for integration, and then added to the hyperspectral image after upsampling in step 2 to obtain the reconstructed image.
[0060] Feature fusion module such as Figure 5 As shown, first, according to the input y U 9 and z9 generate the weights required for fusion.
[0061] y U Add 9 at a ratio of 0.75 and z9 at a ratio of 0.25 to get y U 9 weight vector. Similarly, z9 is proportional to y at 0.75. U 9 is added at a ratio of 0.25 to obtain the weight vector of z9. The purpose of this is to balance the contribution of the two input features and ensure that the main information of the weight vector comes from the input features on the same side while also retaining the information of the input features on the other side.
[0062] Then, these two weight vectors are processed through a 1×1 convolution and a GeLU activation function, and then through a second 1×1 convolution and a Sigmoid activation function to generate the weight w y ∈R F×H×W and w z ∈R F×H×W It is worth noting that the number of channels drops to a quarter of the original after the first convolutional layer, and recovers to the original number of channels after the second convolutional layer.
[0063] y U 9 and z9 are respectively subjected to a depth-separable convolution with a convolution kernel size of 3 to obtain y U91 ∈R F×H×W and z 91 ∈R F ×H×W ,y U 91 and z 91 The weight w y and w z Multiply and add to get the dual-branch fusion feature F∈R F×H×W .
[0064] F restores the number of feature channels to the original number of channels through a convolution with a convolution kernel size of 1 to obtain F1∈R C ×H×W , and then a depth-separable convolution with a convolution kernel size of 3 is performed to obtain the final output O∈R of the feature fusion module C×H×W , and y U Add together to get the reconstructed image.
[0065] Step 5: The error between the reconstructed image and the true image in the training set is calculated using the L1 loss function, and the network parameters are updated using the gradient descent algorithm.
[0066] The embodiment uses the Adam optimizer for parameter update, and its parameters β1 and β2 are 0.9 and 0.999, respectively. The learning rate of the optimizer is updated using the cosine annealing algorithm.
[0067] Step 6: Repeat steps 2 to 5 until the model parameters reach the optimal state. And use the test data to test the model.
[0068] The above embodiment is run on an Ubuntu operating system equipped with a Xeon(R) Platinum 8255C CPU and an NVIDIA GeForce RTX3090 GPU, and the model network is built using the PyTorch 1.11.0 deep learning framework. The batch size of the embodiment is 32 and the epoch is 500.
[0069] In order to prove the effectiveness of the method of this embodiment, simulation experiments were conducted on the test set generated in step 1 with the existing ResTFNet, SSRNet, MSSJFL, HSRNet, GuideNet and PSRT methods, and peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and spectral angle mapping (SAM) were selected as objective evaluation indicators. Among them, the higher the PSNR and SSIM values, the better the image reconstruction effect; the lower the SAM value, the better the image reconstruction effect.
[0070] The experimental results are as follows Figure 6As shown, the various indicators of the embodiment method on the test set are significantly better than those of other methods. This verifies that the image reconstructed by this method not only has better texture details, but also performs extremely well in terms of spectral information fidelity. These advantages also prove that the method of the present invention has broad prospects and important application value in practical applications.
Claims
1. A hyperspectral image super-resolution reconstruction method based on a dual-branch network, characterized in that: The following steps are involved: Step 1: Generate training data and test data based on the existing hyperspectral image dataset; Step 2: Align features of training data; Step 3: The data after feature alignment is input into multiple sets of series-connected dual-branch feature reconstruction networks for feature reconstruction; Step 4: The dual-branch feature information after feature reconstruction is input into the feature fusion module for integration, and then added to the hyperspectral image after upsampling in step 2 to obtain the reconstructed image; Step 5: The error between the reconstructed image and the true image in the training data is calculated using the L1 loss function, and the network parameters are updated using the gradient descent algorithm; Step 6: Repeat steps 2 to 5 until the model parameters reach the optimal state, and test the model using test data.
2. The method for super-resolution reconstruction of hyperspectral images based on a dual-branch network according to claim 1, characterized in that: The method for generating training data and test data in step 1 is as follows: 80% of the hyperspectral image data set is randomly selected as the training set, and the rest is used as the test set; the training set data is enhanced by random cropping, random flipping and random rotation, and the hyperspectral images in the data set are regarded as true value images; a Gaussian blur kernel is applied to the hyperspectral image for blurring, and then the bicubic interpolation algorithm is used to downsample it to generate a corresponding low-resolution hyperspectral image.
3. The method for super-resolution reconstruction of hyperspectral images based on a dual-branch network according to claim 1, characterized in that: The feature alignment method in step 2 is: up-sampling the input low-resolution hyperspectral image through the bicubic interpolation algorithm to make it consistent with the resolution of the multispectral image; then, they are respectively convolved with a convolution kernel size of 1 to make the number of feature channels of the two consistent.
4. The method for super-resolution reconstruction of hyperspectral images based on a dual-branch network according to claim 1, characterized in that: The dual-branch feature reconstruction network in step 3 includes: Global attention module: The input of this module is the hyperspectral image features and multispectral image features after feature alignment; its core is the attention mechanism, which reconstructs the hyperspectral image features and multispectral image features in the spatial dimension and channel dimension respectively according to the respective characteristics of the two images; the hyperspectral image feature branch adopts the linear aggregation attention mechanism, and the multispectral image feature branch adopts the self-attention mechanism; before the attention calculation, the Q vectors in the two branches are simply interacted through convolution, so that the Q vectors of both sides can learn the guidance information of the other branch that is beneficial to their own image reconstruction, so that the attention calculation results are more in line with the reconstruction task requirements; Feature interaction module: The input of this module is the dual-branch image features processed by the global attention module; the dual-branch image features calculated by the attention mechanism are spliced, and the interaction of the dual-branch features is realized through two sets of convolutions with a convolution kernel size of 1, so that each branch can learn the effective information of the other branch, laying the foundation for subsequent feature fusion; after the interaction is completed, the module splits the fused information back to the original dual-branch form, so that the hyperspectral image features and multispectral image features can continue to be reconstructed separately in the subsequent processing process; Local reconstruction unit: It is composed of multiple local reconstruction modules. The input of this module is the image features of a single branch after the output of the feature interaction module. It deeply mines the local information and detail features of the image by performing channel scaling and using convolution kernels of different sizes, thereby improving the model's ability to reconstruct image textures.
5. The method for super-resolution reconstruction of hyperspectral images based on a dual-branch network according to claim 1, characterized in that: The feature fusion module in step 4: its input is the dual-branch feature information output by the dual-branch feature reconstruction network; this module designs a simple weight network, treats the reconstructed dual-branch features equally, assigns corresponding weights to each branch, and then completes the fusion by weighted summation of the dual branches, thereby efficiently integrating all image feature information.
Citation Information
Cited By
Double-branch hyperspectral spatial super-resolution network model product, training method and application
CN119941516A
Unsupervised hyperspectral image reconstruction method and device, electronic equipment and storage medium
CN120876739A
Unsupervised hyperspectral image reconstruction method and device, electronic equipment and storage medium
CN120876739B
Infrared and visible light image fusion method based on cross focusing linear attention
CN120976036A
An infrared and visible light image fusion method based on cross-focusing linear attention
CN120976036B
