A hyperspectral image reconstruction method and device based on an AGDRN network
Through the AGDRN network and SSPA attention mechanism, the problems of capturing long-range correlation and gradient vanishing in hyperspectral image reconstruction are solved, the reconstruction accuracy and efficiency are improved, and efficient low-frequency detail preservation and feature information transmission are achieved.
Patent Information
- Application Number
- CN202211072825.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-02
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2042-09-02
AI Technical Summary
Existing technologies have difficulty capturing long-range correlations in hyperspectral image reconstruction, lack discriminative learning capabilities, gradient disappearance, and difficulty retaining low-frequency information, resulting in insufficient reconstruction accuracy.
The AGDRN network is adopted, combined with residual skip connections and the novel SSPA attention mechanism. The global attention is used to capture long-distance correlations and generate importance weights of different regions to enhance the discriminative learning ability of the network. At the same time, a spectral-spatial block attention module is designed to capture the correlation between the spectral and spatial dimensions of the feature map.
It effectively preserves low-frequency detail information, improves the accuracy and efficiency of hyperspectral image reconstruction, solves the gradient vanishing problem, and enhances the network's discriminative learning ability.
Smart Images
Figure CN115511991B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of hyperspectral computed imaging, and in particular to a hyperspectral image reconstruction method and device based on an AGDRN network. BACKGROUND
[0002] Hyperspectral imaging technology is a multi-dimensional information acquisition technology that combines computed imaging and spectral technology, and can simultaneously obtain spatial information and spectral information of a scene. Compared with ordinary color images, three-dimensional hyperspectral images have a wider band coverage and higher wavelength resolution, and thus contain more rich spectral detail information, and are widely used in the fields of remote sensing imaging, medical diagnosis and computer vision. CASSI (Coded Aperture Snapshot Spectral Imaging) is an emerging hyperspectral imaging technology, which is based on the sampling theory of compressed sensing, and can capture spatial and spectral information of an image using only a small amount of compressed measurement results, and has become one of the important development directions in this field. CASSI uses a coded aperture and a prism to modulate the spatial and spectral information of a hyperspectral image, obtains a two-dimensional compressed measurement containing most of the image information, and then uses a reconstruction technology to reconstruct a three-dimensional hyperspectral image. One of the bottlenecks of the CASSI technology is how to accurately reconstruct a three-dimensional hyperspectral image from a two-dimensional compressed measurement.
[0003] Inspired by the application of deep learning in other fields of computed imaging, researchers began to introduce CNN (Convolutional Neural Network) into CASSI reconstruction. The traditional CNN has inherent limitations in capturing long-range correlations due to the limited receptive field of its convolution kernel. In addition, when the network has a large number of layers, problems such as gradient disappearance occur, making it difficult to train the network and losing low-frequency information, which reduces the reconstruction accuracy. The current mainstream approach is to combine CNN with an attention mechanism to capture long-range correlations in the feature map during the reconstruction process, and to use a network structure containing skip connections as the skeleton of the reconstruction network, so that low-frequency information can be preserved to the deep network. Compared with general image reconstruction problems, the hyperspectral image reconstruction problem has unique data characteristics: such as a larger number of narrower wavebands, so the spectral dimension has high correlation, and at the same time, the spatial dimension information is distributed relatively sparsely in each spectral waveband because the information is dispersed to each waveband; the amount of hyperspectral image data is much larger than that of common grayscale or color images; CASSI compresses three-dimensional hyperspectral images into two-dimensional measurements, so the information compression rate is very high. Therefore, it is necessary to design a targeted and efficient attention module and network structure for fully extracting and transmitting feature information. SUMMARY
[0004] The application aims to provide a hyperspectral image reconstruction method and device based on an AGDRN network, which endows the network with the ability to transmit feature information of different frequency levels between different modules, thereby retaining low-frequency detail information and solving the gradient vanishing problem; meanwhile, the new SSPA attention mechanism efficiently explores the internal correlation of compressed measurement data, is combined with a convolution layer, captures long-distance correlation by calculating global attention, and generates importance weights of different regions adaptively, thereby enhancing the discriminative learning ability of the network.
[0005] The application aims to provide a hyperspectral image reconstruction method and device based on an AGDRN network, which endows the network with the ability to transmit feature information of different frequency levels between different modules, thereby retaining low-frequency detail information and solving the gradient vanishing problem; meanwhile, the new SSPA attention mechanism efficiently explores the internal correlation of compressed measurement data, is combined with a convolution layer, captures long-distance correlation by calculating global attention, and generates importance weights of different regions adaptively, thereby enhancing the discriminative learning ability of the network.
[0006] A hyperspectral image reconstruction method based on an AGDRN network comprises the following steps:
[0007] Obtaining compressed measurement data collected by a CASSI system;
[0008] Constructing an end-to-end reconstruction network based on an AGDRN backbone network and training the network, wherein the AGDRN backbone network comprises a feature mapping layer, a feature processing module and a feature reconstruction layer connected in sequence, the top level of the feature processing module is composed of a global skip connection across multiple feature processing groups and a convolution layer, each feature processing group is composed of a group-level skip connection across a residual attention block, multiple residual convolution blocks and a convolution layer, each residual attention block is composed of a local skip connection across an SSPA attention block, and each residual convolution block is composed of a local skip connection across multiple convolution layers;
[0009] Inputting the compressed measurement data into the end-to-end reconstruction network to obtain a reconstructed hyperspectral image.
[0010] The feature mapping layer is realized by a convolution layer, which is used to map the compressed measurement data to a feature space:
[0011] X M =conv(Y)
[0012] Wherein Y is the compressed measurement data collected by the CASSI system, conv represents a convolution operation, and X M is a feature map preliminarily mapped to the feature space.
[0013] The top level of the feature processing module is processed as follows:
[0014] X P =conv(G A (...G2(G1(X M ))...))+X M
[0015] Wherein G A() represents the A th feature processing group, X P represents the output of the feature processing module.
[0016] The processing procedure of each feature processing group is as follows:
[0017] G(F) = conv(R B (...R2(R1(Attn(F)))...))+F
[0018] wherein, conv represents a convolution operation, Attn() represents a residual attention block, R B () represents the B th residual convolution block, and F represents the input feature map of the feature processing group.
[0019] The processing procedure of each residual attention block is as follows:
[0020] Attn(F1) = SSPA(F1)+F1
[0021] wherein, F1 represents the input feature map of the residual attention block, and SSPA is a spatial-spectral block attention, which includes a spectral attention sub-module and a spatial attention sub-module connected in sequence:
[0022] SSPA(F1) = A S (A C (F1))
[0023] wherein, A S () represents the spatial attention sub-module, and A C () represents the spectral attention sub-module.
[0024] The spectral attention sub-module performs the following operation to assign different attention weights to each spectral channel of the feature map:
[0025] A C (F1) = F1⊙ C σ(MLP(Max(F1))+MLP(Avg(F1)))
[0026] wherein, Max() and Avg() represent a max-pooling layer and an average-pooling layer respectively, MLP represents a multi-layer perceptron, σ() represents a Sigmoid activation function, and ⊙ C represents channel-wise point multiplication.
[0027] The spatial attention sub-module performs the following operation to generate a spectral-channel-wise attention map:
[0028] A S (F1′) = F1′⊙ P σ(MLP(conv(F1′)))
[0029] wherein, conv represents a convolution operation, MLP represents a multi-layer perception, sigma () represents a Sigmoid activation function, and represents point multiplication in a spatial block P F1' represents the input of the spatial attention sub-module, that is, the output feature map of the spectral attention sub-module.
[0030] The processing process of each residual convolution block is as follows:
[0031] R (F2) = conv (Relu (conv (F2))) + F2
[0032] wherein, Relu() represents an activation function, and conv represents a convolution operation.
[0033] The feature reconstruction layer is opposite to the feature mapping layer, and uses a convolution layer to restore the output of the feature processing module to a reconstructed hyperspectral image:
[0034] X R = conv (X P )
[0035] wherein, X R represents a reconstructed hyperspectral image output by the end-to-end reconstruction network.
[0036] A hyperspectral image reconstruction device based on an AGDRN network comprises a memory, a processor, and a program stored in the memory, and the processor implements the method as described above when executing the program.
[0037] Compared with the prior art, the present application has the following beneficial effects:
[0038] (1) The present application is aimed at a series of problems existing in the traditional CNN model, such as difficulty in capturing long-distance correlation, lack of discriminative learning ability, gradient disappearance, and difficulty in retaining low-frequency information to the deep network, etc. AGDRN (Attention Guided Deep Residual Network) is proposed for the hyperspectral image reconstruction task of CASSI technology, which introduces multi-level residual skip connection on the one hand, so that the network has the ability to transmit feature information of different frequency levels between different modules, thereby retaining low-frequency detail information and solving the problem of gradient disappearance. On the other hand, the attention mechanism is combined with the convolution layer, the long-distance correlation is captured by calculating the global attention, and the importance weight of different regions is generated adaptively to enhance the discriminative learning ability of the network.
[0039] (2) According to the data characteristics of the hyperspectral image, the application proposes a new attention mechanism, SSPA (Spectral-Spatial Patch Attention), for jointly capturing the spectral dimension and spatial dimension correlation of the feature map and combining the spectral and spatial joint features; in the spectral dimension, the feature map is divided by channel, the image of each channel is taken as a patch, and then the correlation of different spectral channels is captured; in the spatial dimension, the feature map is divided by spatial region, all spectral channels of each small region are taken as a patch, and then the correlation between different spatial regions is captured; by combining the spectral attention and the spatial attention, the internal correlation of the compressed measurement data is more efficiently explored. BRIEF DESCRIPTION OF DRAWINGS
[0040] Figure 1 A flowchart of the method of the application;
[0041] Figure 2 A model framework diagram of the end-to-end reconstruction network of the application;
[0042] Figure 3 A structural diagram of the SSPA attention block. DETAILED DESCRIPTION
[0043] The application will be described in detail below in combination with the drawings and specific embodiments. The embodiments are implemented on the premise of the technical solution of the application, and detailed implementation modes and specific operation processes are given, but the protection scope of the application is not limited to the following embodiments.
[0044] A hyperspectral image reconstruction method based on an AGDRN network, as shown in Figure 1 , includes the following steps:
[0045] 1) Obtain the compressed measurement data collected by the CASSI system;
[0046] 2) Construct an end-to-end reconstruction network based on the AGDRN backbone network and train it;
[0047] The structure of the AGDRN backbone network is as shown in Figure 2 , including a feature mapping layer, a feature processing module and a feature reconstruction layer connected in sequence.
[0048] ① Feature mapping layer
[0049] The feature mapping layer is implemented by a convolution layer, for mapping the compressed measurement data to a feature space:
[0050] X M = conv(Y)
[0051] where Y is the compressed measurement data collected by the CASSI system, conv denotes the convolution operation, X M is the feature map preliminarily mapped to the feature space.
[0052] ② Feature processing module
[0053] The feature processing module is the main part of the AGDRN backbone network, which uses three different levels of skip connections to build a modular structure.
[0054] The top level of the feature processing module is composed of a global skip connection across multiple feature processing groups and a convolution layer, wherein a convolution layer is connected after the last feature processing group for global feature fusion.
[0055] The top level processing process of the feature processing module is:
[0056] X P = conv(G A (...G2(G1(X M ))...))+X M
[0057] where G A () represents the A-th feature processing group, X P represents the output of the feature processing module.
[0058] Each specific feature processing group belongs to the middle level of the feature processing module, and each feature processing group is composed of a group-level skip connection across a residual attention block, multiple residual convolution blocks and a convolution layer, and a convolution layer is also connected after the last residual convolution block for intra-group feature fusion.
[0059] The processing process of each feature processing group is:
[0060] G(F) = conv(R B (...R2(R1(Attn(F)))...))+F
[0061] where conv represents the convolution operation, Attn() represents the residual attention block, R B () represents the B-th residual convolution block, and F represents the input feature map of the feature processing group. The residual attention block is placed before the residual convolution block to adaptively generate attention weights for different regions of the feature map during training, and to guide the subsequent multiple convolution modules to process the feature information.
[0062] Each residual attention block is composed of a local skip connection across an SSPA attention block, and its processing process is:
[0063] Attn(F1) = SSPA(F1) + F1
[0064] wherein, F1 represents the input feature map of the residual attention block, SSPA is a spatial-spectral patch attention, the structure of which is as shown in Figure 3 (1), which includes a spectral attention sub-module and a spatial attention sub-module connected in turn:
[0065] SSPA(F1) = A S (A C (F1))
[0066] wherein, A S () represents the spatial attention sub-module, and A C () represents the spectral attention sub-module.
[0067] The spectral attention sub-module is as shown in Figure 3 (2), for the input feature map, the 3-dimensional feature map is divided into patches according to the spectral channel, and the image of each channel is taken as a patch, and then a one-dimensional information is extracted from each patch as a feature descriptor using a pooling layer. In this embodiment, a max pooling layer and an average pooling layer are used simultaneously to extract respectively to capture the features within the channel more fully, and two groups of feature descriptors are obtained. Subsequently, the two groups of feature descriptors are sent into a multi-layer perception (MLP) so that the network learns the mutual dependence between the spectra. The structure of the multi-layer perception is as shown in Figure 3 (4). After the two groups of results are added, they are sent into a Sigmoid activation function for normalization, thereby generating a spectral channel-by-spectral channel attention map. Finally, the attention map is point multiplied with the input feature map to give different attention weights to each spectral channel. The execution process is represented as:
[0068] A C (F1) = F1⊙ C σ(MLP(Max(F1))+MLP(Avg(F1)))
[0069] wherein, Max() and Avg() represent the max pooling layer and the average pooling layer respectively, MLP represents the multi-layer perception, σ() represents the Sigmoid activation function, and ⊙ C represents the channel-by-channel point multiplication.
[0070] The spatial attention sub-module is as shown in Figure 3(3) as shown. Since hyperspectral images are sparse in spatial dimension, generating pixel-wise attention map is very computationally expensive and hard to capture local correlation information. Therefore, for the input feature map, we first divide the feature map into patches according to spatial regions. For each patch composed of all channel images in a spatial region, we use a convolution layer to extract two-dimensional feature information from each patch, and then flatten it into a one-dimensional vector as a feature descriptor. Subsequently, we also use an MLP to capture the mutual dependence between different spatial regions of the feature map, and use a Sigmoid function for normalization. The normalized result is reshaped according to the original two-dimensional distribution to obtain the attention map for each spatial region. The attention map will be point multiplied with the input feature map to give different attention weights to each spatial region. Its execution process is represented as:
[0071] A S (F1′)=F1′⊙ P σ(MLP(conv(F1′)))
[0072] where conv represents convolution operation, MLP represents multi-layer perceptron, σ() represents Sigmoid activation function, and ⊙ P represents point multiplication by spatial block, and F1’ represents the input of the spatial attention sub-module, i.e. the output feature map of the spectral attention sub-module.
[0073] Each residual convolution block is composed of a local skip connection across multiple convolution layers, and its processing process is:
[0074] R(F2)=conv(Relu(conv(F2)))+F2
[0075] where Relu() represents the activation function, and conv represents the convolution operation, and F2 represents the input of the residual convolution block.
[0076] By using skip connection to build three levels of modular structure, the feature processing module allows the network to learn how to pass different frequency component feature information between different levels.
[0077] ③ Feature reconstruction layer
[0078] The feature reconstruction layer is opposite to the feature mapping layer, which uses a convolution layer to restore the output of the feature processing module to the reconstructed hyperspectral image:
[0079] X R =conv(X P )
[0080] where X R represents the reconstructed hyperspectral image output by the end-to-end reconstruction network.
[0081] 3) input the compressed measurement data into the end-to-end reconstruction network to obtain a reconstructed hyperspectral image.
[0082] The preferred embodiments of the present application have been described in detail above. It should be understood that modifications and variations to the preferred embodiments could be made by those skilled in the art in light of the teachings above without departing from the spirit of the present application. It is therefore contemplated that the application claim protection for such changes and modifications to the preferred embodiments made by those skilled in the art having the benefit of the teachings above.
Claims
1. A method for hyperspectral image reconstruction based on AGDRN network, characterized in that, The method comprises the following steps: obtaining compressed measurement data collected by a CASSI system; constructing and training an end-to-end reconstruction network based on an AGDRN backbone network, the AGDRN backbone network comprising a feature mapping layer, a feature processing module and a feature reconstruction layer connected in sequence, a top-level hierarchy of the feature processing module being composed of a global skip connection across multiple feature processing groups and a convolutional layer, each feature processing group being composed of a group-level skip connection across a residual attention block, multiple residual convolution blocks and a convolutional layer, each residual attention block being composed of a local skip connection across an SSPA attention block, and each residual convolution block being composed of a local skip connection across multiple convolutional layers; wherein the processing procedure of each residual attention block is as follows: wherein, denotes the input feature map of the residual attention block, SSPA is a spatial-spectral patch attention, including a spectral attention sub-module and a spatial attention sub-module connected in turn: wherein, representing a spatial attention sub-module, representing a spectral attention sub-module; inputting the compressed measurement data into the end-to-end reconstruction network to obtain a reconstructed hyperspectral image.
2. The hyperspectral image reconstruction method based on AGDRN network according to claim 1, characterized in that, the feature mapping layer is implemented by a convolutional layer, for mapping the compressed measurement data to a feature space: wherein, is the compressed measurement data collected by the CASSI system, denotes a convolution operation, is the feature map that is initially mapped to the feature space.
3. The hyperspectral image reconstruction method based on AGDRN network according to claim 2, characterized in that, the processing procedure of the top-level hierarchy of the feature processing module is as follows: wherein, represents the A feature processing group, represents the output of the feature processing module.
4. The hyperspectral image reconstruction method based on AGDRN network according to claim 1, characterized in that, the processing procedure of each feature processing group is as follows: in, represents the convolution operation, represents the residual attention block, Representative B residual convolution blocks, Represents the input feature map of the feature processing group.
5. The hyperspectral image reconstruction method based on AGDRN network according to claim 1, characterized in that, the spectral attention sub-module performs the following operations to assign different attention weights to each spectral channel of the feature map: wherein, and represent a max-pooling layer and an average-pooling layer, respectively, denotes a multi-layer perceptron, denotes a sigmoid activation function, denotes a channel-wise pointwise multiplication.
6. The hyperspectral image reconstruction method based on AGDRN network according to claim 1, characterized in that, the spatial attention sub-module performs the following operations to generate a spectral-channel-by-spectral-channel attention map: wherein, denotes a convolution operation, denotes a multi-layer perceptron, denotes a Sigmoid activation function, denotes a spatial block-wise point-wise multiplication, denotes the input of the spatial attention sub-module, i.e. the output feature map of the spectral attention sub-module.
7. The hyperspectral image reconstruction method based on AGDRN network according to claim 1, characterized in that, the processing procedure of each residual convolution block is as follows: wherein, denotes an activation function, denotes a convolution operation, denotes an input of the residual convolution block.
8. The hyperspectral image reconstruction method based on AGDRN network according to claim 3, characterized in that, the feature reconstruction layer is opposite to the feature mapping layer, and uses a convolutional layer to restore the output of the feature processing module to a reconstructed hyperspectral image: wherein, represents a reconstructed hyperspectral image that represents an end-to-end reconstructed network output. 9.A device for hyperspectral image reconstruction based on AGDRN network, comprising a memory, a processor, and a program stored in the memory, wherein, the processor implements the method according to any one of claims 1-8 when executing the program.