Hyperspectral image reconstruction method and system based on spectral perception and structure search

Abstract

The invention discloses a hyperspectral image reconstruction method and system based on spectrum sensing and structure searching, and the method comprises the steps: firstly carrying out the preprocessing of a degraded hyperspectral image, inputting the preprocessed image into a spectrum sensing dynamic structure network, and carrying out the calculation to obtain a reconstructed hyperspectral image; the spectrum sensing dynamic structure network comprises shallow feature extraction, depth feature extraction and feature reconstruction; the shallow feature extraction comprises convolution, an activation function ReLU and a BN layer; deep feature extraction comprises a microstructure and a convolution layer of four residual error stacking; the feature reconstruction comprises convolution, an activation function ReLU and a BN layer; the microstructure comprises a convolutional layer, four child nodes, feature combination and feature addition; the nodes are directly connected through a mixed residual block; the mixed residual block comprises eight substructures and is dynamically adjusted through a noise independent structure search algorithm. The method has a better expression capability for the spectral features, can balance the weights of the spatial features and the spectral features, and achieves a better reconstruction effect.

Description

Technical Field

[0001] The present invention relates to the field of computational imaging, and in particular to a hyperspectral image reconstruction method and system based on spectral perception and structure search. Background Art

[0002] Hyperspectral images contain continuous spectral features and can represent richer scene information. However, due to the influence of imaging equipment and conditions, hyperspectral images often suffer varying degrees of degradation during the imaging process, such as noise, fog, and ghosting, which greatly limits their application. Therefore, reconstructing degraded hyperspectral images to restore their scene information has important research significance and application value.

[0003] Existing hyperspectral image reconstruction technologies often use convolutional neural networks. However, due to artificially designed structural limitations, the models are usually unable to balance the weights between spatial features and spectral features, resulting in the spectral features not being represented relatively comprehensively, thus affecting the quality of the reconstructed hyperspectral images. Summary of the Invention

[0004] The purpose of the present invention is to use an adaptive structure to balance the weights between spectral features and spatial features to fully utilize spectral information for reconstruction, and to provide a hyperspectral image reconstruction method based on spectral perception and structure search.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A hyperspectral image reconstruction method based on spectral perception and structure search comprises the following steps: Step 1, image preprocessing: obtain the degraded hyperspectral image and perform normalization operation; Step 2: construct a spectral-aware dynamic structure network and input the preprocessed degraded hyperspectral image into the network to obtain a reconstructed hyperspectral image; The spectrum perception dynamic structure network includes shallow feature extraction, deep feature extraction, and feature reconstruction; The shallow feature extraction takes degraded hyperspectral images as input and outputs shallow feature embeddings; The deep feature extraction includes multiple residual stacking structures, each residual stacking structure includes a microstructure and a convolutional layer, accepts shallow feature embedding, and outputs deep image features; The microstructure consists of a convolutional layer and four nodes. The nodes are densely connected to each other through a directed acyclic graph. The computational units between the nodes are determined by a hybrid residual block. Finally, the features of all nodes are merged and skip-connected with the convolutional layer to output a single-step deep feature. The feature reconstruction input is the deep image feature, and after performing residual connection with the input image, the reconstructed hyperspectral image output is obtained. The number of input channels is the number of deep image feature channels, and the number of output channels is the number of reconstructed hyperspectral image channels.

[0006] Furthermore, in step 1, a linear transformation is used to scale the degraded hyperspectral image to the same scale. The specific normalization method is:

[0007] in, is the normalized image, is the input degraded hyperspectral image.

[0008] Furthermore, in step 2, the shallow feature extraction includes a 1×1 convolution, a ReLU activation function and a BN layer, and the number of input channels is the number of hyperspectral image channels. c , the number of output channels is the number of shallow feature channels 8 c .

[0009] Furthermore, in step 2, the feature reconstruction includes a 1×1 convolution, a ReLU activation function, and a BN layer, which is used to reconstruct the deep image features into a hyperspectral image, wherein the number of input channels is the number of feature channels, and the number of output channels is the number of hyperspectral image channels.

[0010] Furthermore, the convolutional layer consists of a 3×3 convolution, a ReLU activation function, and a BN layer, which is used to further extract features and integrate channels of the single-step features extracted from the microstructure; the number of input and output channels of the convolutional layer is the same as that of the depth image features.

[0011] Furthermore, the hybrid residual block It contains a total of 8 convolution calculation units. During calculation, the convolution calculation unit is adaptively selected through parameters and the following formula:

[0012] in, Representative node pair The computational units between Representative Node The value of , Both represent the calculation unit number, Representatives The calculation unit for input , is a structural parameter representing the node pair The calculation unit between probability.

[0013] Furthermore, the structural parameters The architecture search algorithm, namely the noise independent search algorithm, is given, and its optimization goal is:

[0014] in, Express expectations, are the optimized model parameters, is the noise level, According to the given , are samples of the training set and the test set respectively. is the objective function, defined as:

[0015] in, The noise level is The degraded image of It is adopted As a network of structural parameters, It is a dataset The samples in are other parameters of the network.

[0016] Furthermore, the 8 convolution calculation units contained in the hybrid residual block are: 4 spatial convolution structures: 3×3 convolution , 5×5 convolution , 3×3 dilated convolution , 3×3 dilated group convolution ; 2 spectral convolution structures: 1×1 convolution , 1×1 group convolution ; 2 spatial spectral convolution structures: 3×3 separable dilated convolution , 3×3 separable dilated group convolution .

[0017] Furthermore, in step 2, the spectrum-aware dynamic structure network is a spectrum-aware dynamic structure network with optimized parameters, and the parameter optimization includes the following steps: Step S1: performing image preprocessing on the degraded hyperspectral image dataset, including degraded hyperspectral images and lossless hyperspectral images; Step S2: input the preprocessed degraded hyperspectral image and the lossless hyperspectral image into the spectrum-aware dynamic structure network; Step S3: Calculate the loss function and optimize the model parameters through gradient descent and back propagation.

[0018] The present invention also provides a hyperspectral image reconstruction system based on spectral perception and structure search, comprising the following units: Image preprocessing unit: used to obtain degraded hyperspectral images and perform normalization operations; The network processing unit is used to construct a spectral perception dynamic structure network and input the preprocessed degraded hyperspectral image into the network to obtain a reconstructed hyperspectral image; The spectrum perception dynamic structure network includes shallow feature extraction, deep feature extraction, and feature reconstruction; The shallow feature extraction takes degraded hyperspectral images as input and outputs shallow feature embeddings; The deep feature extraction includes multiple residual stacking structures, each residual stacking structure includes a microstructure and a convolutional layer, accepts shallow feature embedding, and outputs deep image features; The microstructure consists of a convolutional layer and four nodes. The nodes are densely connected to each other through a directed acyclic graph. The computational units between the nodes are determined by a hybrid residual block. Finally, the features of all nodes are merged and skip-connected with the convolutional layer to output a single-step deep feature. The feature reconstruction input is the deep image feature, and after performing residual connection with the input image, the reconstructed hyperspectral image output is obtained. The number of input channels is the number of deep image feature channels, and the number of output channels is the number of reconstructed hyperspectral image channels.

[0019] The present invention reconstructs degraded hyperspectral images using a hyperspectral image reconstruction algorithm based on spectral perception and structure search. Through noise-independent structure search, the method provided by the present invention can adaptively adjust the computational units to balance the weights between spectral and spatial features, thereby adapting to reconstruction tasks with different degradation types and degrees, and enabling better application in complex degradation scenarios. Furthermore, the present invention introduces global and local skip connections, enabling better fusion of global and local features, thereby extracting richer spectral information. Compared to existing algorithms, the method provided by the present invention can more fully characterize and apply spectral information. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The technical solution of this invention is further illustrated below using embodiments and specific implementation methods. In addition, some drawings are used in the process of illustrating the technical solution. Those skilled in the art can also derive other drawings and the intent of the present invention based on these drawings without making any creative efforts.

[0021] Figure 1 is a flow chart of a method according to an embodiment of the present invention; Figure 2 This is a diagram of the spectral perception dynamic structure network architecture provided by the present invention; Figure 3 Flowchart for optimizing parameters of the spectrum-aware dynamic structure network provided by the present invention; Figure 4 This is a visualization diagram of experimental results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to facilitate ordinary technicians in this field to understand and implement the present invention, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0023] This embodiment takes a given degraded hyperspectral image as an example to further illustrate the present invention. Figure 1 This embodiment provides a hyperspectral image reconstruction method based on spectral perception and structure search, comprising the following steps: Step 1, image preprocessing: obtain the degraded hyperspectral image and perform preprocessing operations such as normalization on the hyperspectral image; Step 2: construct a spectral perception dynamic structure network and input the preprocessed hyperspectral image into the network to obtain a reconstructed hyperspectral image; Please see Figure 2 ,The spectral perception dynamic structure network includes shallow feature extraction, deep feature extraction, and feature reconstruction; The shallow feature extraction takes as input the pre-processed degraded hyperspectral image and outputs shallow feature embedding, the number of input channels is the number of hyperspectral image channels, and the number of output channels is the number of feature channels; The deep feature extraction includes four residual stacking structures, each of which includes a microstructure and a convolutional layer. The number of input and output channels is the same as that of the shallow feature extraction, and it accepts shallow feature input and outputs deep image features. The feature reconstruction input is the deep image feature, and after performing residual connection with the input image, the reconstructed hyperspectral image output is obtained. The number of input channels is the number of deep image feature channels, and the number of output channels is the number of reconstructed hyperspectral image channels.

[0024] In one embodiment, in step 1, the degraded hyperspectral image is preprocessed, including image normalization. A linear transformation is used to scale the hyperspectral image to the same scale, thereby avoiding a series of problems such as gradient explosion in subsequent calculations. The specific normalization method is: (1) in, is the normalized image, is the input hyperspectral image.

[0025] In one embodiment, in step 2, the shallow feature extraction includes a 1×1 convolution, a ReLU activation function and a BN layer, and the number of input channels is the number of channels of the hyperspectral image.c , the number of output channels is the number of shallow feature channels 8 c The preprocessed hyperspectral image undergoes shallow feature extraction to obtain feature embedding containing texture, structure and other information. At the same time, the number of channels of the hyperspectral image is converted to the same number of channels as the subsequent deep feature extraction, which facilitates structure stacking and residual connection.

[0026] In one embodiment, in step 2, the deep feature extraction includes four residual stacking structures, each residual stacking structure consists of a microstructure and a convolutional layer, and the shallow features are subjected to deep feature extraction to obtain deep features containing top-level information such as semantics, which are used to restore a reconstructed hyperspectral image containing more information in the feature reconstruction step.

[0027] In one embodiment, in step 2, the feature reconstruction part includes a 1×1 convolution, a ReLU activation function and a BN layer, which is used to reconstruct the deep image features into a hyperspectral image, and its input channel number is the number of feature channels, and the output channel number is the number of hyperspectral image channels.

[0028] In one embodiment, the microstructure includes a convolutional layer and four nodes, and the nodes are densely residually connected to each other through a directed acyclic graph. In addition, the calculation units between the nodes are determined by a mixed residual block, that is, whether the nodes are connected is calculated by a mixed residual block. Finally, the features of all nodes are merged and jump-connected with the convolutional layer to output a single-step depth feature; the number of input channels and the number of output channels of the microstructure are the same as the depth feature.

[0029] In one embodiment, the convolutional layer consists of a 3×3 convolution, a ReLU activation function, and a BN layer, which is used to further extract features and integrate channels of single-step features extracted from the microstructure; the number of input and output channels of the convolutional layer is the same as that of the depth feature.

[0030] In one embodiment, the hybrid residual block It consists of three parts: spatial convolution, spectral convolution, and space-spectral convolution. The three parts contain a total of 8 convolution calculation units. During calculation, the convolution calculation unit is adaptively selected through parameters and the following formula to flexibly handle different hyperspectral reconstruction tasks: (2) in, Representative node pair The computational units between Representative Node The value of , Both represent the calculation unit number, Representatives The calculation unit for input , is a structural parameter representing the node pair The calculation unit between The above formula means that a probability weight is assigned to each computing unit. The computing unit with the highest probability is the one that meets the characteristics of the current task. By dynamically assigning probability weights, the model can adaptively adjust its own structure and connection mode, thereby adaptively processing different tasks.

[0031] In one embodiment, the The architecture search algorithm, namely the noise independent search algorithm, is given, and its optimization goal is: (3) (4) in, Express expectations, are the parameters after model optimization, is the noise level, According to the given , are samples of the training set and the test set respectively. is the objective function, defined as: (5) in, The noise level is The degraded image of It is adopted As a network of structural parameters, It is a dataset The samples in are other parameters of the network. Through the noise independent search algorithm, the model can adaptively learn the structural parameters according to the data set. , and without subsequent additional training, the network structure and connection method can be adaptively adjusted to complete reconstruction tasks of various degradation types.

[0032] In one embodiment, the hybrid residual block contains three parts: 4 spatial convolution structures: 3×3 convolution , 5×5 convolution , 3×3 dilated convolution , 3×3 dilated group convolution ; 2 spectral convolution structures: 1×1 convolution , 1×1 group convolution ; 2 spatial spectral convolution structures: 3×3 separable dilated convolution , 3×3 separable dilated group convolution Each relaxed residual block has eight parallel convolutional computation units for selection. Each computation unit is assigned a probability related to alpha, i.e., a weight. This allows the network structure to be automatically adjusted based on the searched alpha, achieving self-adaptation. That is, different convolutions correspond to feature extraction and fusion of different domains, achieving adaptive feature extraction.

[0033] In one embodiment, in step 2, the spectrum-aware dynamic structure network is a spectrum-aware dynamic structure network with optimized parameters, see Figure 3 , parameter optimization includes the following steps: Step S1: performing image preprocessing on the degraded hyperspectral image dataset, including the degraded hyperspectral image and the lossless hyperspectral image, in the same manner as step 1; Step S2: inputting the pre-processed degraded hyperspectral image and the lossless hyperspectral image into the spectrum-aware dynamic structure network, and searching for the optimal network structure and connection mode through the architecture search algorithm, i.e., the noise-independent search algorithm; Step S3: The preprocessed degraded hyperspectral image and the lossless hyperspectral image are input into the spectral-aware dynamic structure network after architecture search, and the loss function is calculated. The model parameters are optimized through gradient descent and back propagation.

[0034] In one embodiment, 10 test images were selected from the KAIST dataset to conduct experimental tests on the embodiments of the present invention. The experimental results are shown in Table 1 and the attached figure. Figure 4 As shown in the figure. PSNR is the peak signal-to-noise ratio, and SSIM is the structural similarity index. Higher indices indicate better reconstruction. The highest and second-highest indices are bolded and underlined, respectively.

[0035] Table 1 Quantitative experimental results of the embodiment of the present invention on the KAIST dataset

[0036] DeSCI, HSSP, The -net, TSA, and GSM-based methods are respectively referenced from the literature: Liu Y, Yuan X, Suo J, et al. Rank minimization for snapshot compressive imaging[J]. IEEE transactions on pattern analysis and machine intelligence, 2018, 41(12): 2990-3006; Wang L, Sun C, Fu Y, et al. Hyperspectral image reconstruction using a deep spatial-spectral prior[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 8032-8041; Miao X, Yuan X, Pu Y, et al. l-net: Reconstruct hyperspectral images from a snapshot measurement[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 4059-4069; Meng Z, Ma J, Yuan X. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention[C] / / European conference on computer vision. Cham: Springer International Publishing, 2020: 187-204; Huang T, Dong W, Yuan X, et al. Deep gaussian scale mixture prior for spectral compressive imaging[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2021: 16216-16225.

[0037] In one implementation, the embodiment of the present invention can achieve the best results on almost all ten test images, demonstrating the superior performance of the embodiment of the present invention.

[0038] In one embodiment, see Figure 4 , provides a visualization result diagram of an embodiment of the present invention. STASC is the method proposed in an embodiment of the present invention. The reconstructed texture is the clearest and has the best visual effect.

[0039] The embodiment of the present invention further provides a hyperspectral image reconstruction system based on spectral perception and structure search, comprising the following units: Image preprocessing unit: obtains hyperspectral images and performs preprocessing operations such as normalization on the hyperspectral images; Spectral perception dynamic structure network unit, and input the preprocessed hyperspectral image into the network to obtain the reconstructed hyperspectral image; The spectrum perception dynamic structure network includes shallow feature extraction, deep feature extraction, and feature reconstruction; The shallow feature extraction, the input is the degraded hyperspectral image, the output is the shallow feature embedding, the number of input channels is the number of hyperspectral image channels, and the number of output channels is the number of feature channels; The deep feature extraction includes four residual stacking structures, each of which includes a microstructure and a convolutional layer. The number of input and output channels is the same as that of the shallow feature extraction, and it accepts shallow feature input and outputs deep image features. The feature reconstruction input is the deep image feature, and after performing residual connection with the input image, the reconstructed hyperspectral image output is obtained. The number of input channels is the number of deep image feature channels, and the number of output channels is the number of reconstructed hyperspectral image channels.

[0040] The specific implementation method of each unit is the same as that of each step and will not be described in detail in the present invention.

[0041] An embodiment of the present invention also provides a hyperspectral image reconstruction device based on spectral perception and structure search, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a hyperspectral image reconstruction method based on spectral perception and structure search as described in the above scheme.

[0042] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of protection of the patent of the present invention. Under the guidance of the present invention, ordinary technicians in this field can also make substitutions and modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested by the present invention shall be based on the attached claims.

Claims

1. A hyperspectral image reconstruction method based on spectral perception and structure search, characterized in that: The following steps are involved: Step 1, image preprocessing: obtain the degraded hyperspectral image and perform normalization processing; Step 2: construct a spectral-aware dynamic structure network and input the preprocessed degraded hyperspectral image into the network to obtain a reconstructed hyperspectral image; The spectrum perception dynamic structure network includes shallow feature extraction, deep feature extraction, and feature reconstruction; The shallow feature extraction takes degraded hyperspectral images as input and outputs shallow feature embeddings; The deep feature extraction includes multiple residual stacking structures, each residual stacking structure includes a microstructure and a convolutional layer, accepts shallow feature embedding, and outputs deep image features; The microstructure consists of a convolutional layer and four nodes. The nodes are densely connected to each other through a directed acyclic graph. The computational units between the nodes are determined by a hybrid residual block. Finally, the features of all nodes are merged and skip-connected with the convolutional layer to output a single-step deep feature. The feature reconstruction input is the deep image feature, and after performing residual connection with the input image, the reconstructed hyperspectral image output is obtained. The number of input channels is the number of deep image feature channels, and the number of output channels is the number of reconstructed hyperspectral image channels.

2. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 1, characterized in that: In step 1, linear transformation is used to scale the degraded hyperspectral image to the same scale. The specific normalization method is: in, is the normalized image, is the input degraded hyperspectral image.

3. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 1, characterized in that: In step 2, the shallow feature extraction includes a 1×1 convolution, a ReLU activation function and a BN layer, and the number of input channels is the number of hyperspectral image channels. c , the number of output channels is the number of shallow feature channels 8 c .

4. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 1, characterized in that: In step 2, the feature reconstruction includes a 1×1 convolution, a ReLU activation function, and a BN layer, which is used to reconstruct the deep image features into a hyperspectral image. The number of input channels is the number of feature channels, and the number of output channels is the number of hyperspectral image channels.

5. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 1, characterized in that: The convolutional layer consists of a 3×3 convolution, a ReLU activation function, and a BN layer, which is used to further extract features and integrate channels of the single-step features extracted from the microstructure; the number of input and output channels of the convolutional layer is the same as that of the depth image features.

6. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 1, characterized in that: The hybrid residual block It contains a total of 8 convolution calculation units. During calculation, the convolution calculation unit is adaptively selected through parameters and the following formula: in, Representative node pair The computational units between Representative Node The value of , Both represent the calculation unit number, Representatives The calculation unit for input , is a structural parameter representing the node pair The calculation unit between probability.

7. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 6, characterized in that: Structural parameters The architecture search algorithm, namely the noise independent search algorithm, is given, and its optimization goal is: in, Express expectations, are the optimized model parameters, is the noise level, According to the given , are samples of the training set and the test set respectively. is the objective function, defined as: in, The noise level is The degraded image of It is adopted As a network of structural parameters, It is a dataset The samples in are other parameters of the network.

8. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 7, characterized in that: The 8 convolution calculation units contained in the hybrid residual block are: 4 spatial convolution structures: 3×3 convolution , 5×5 convolution , 3×3 dilated convolution , 3×3 dilated group convolution ; 2 spectral convolution structures: 1×1 convolution , 1×1 group convolution ; 2 spatial spectral convolution structures: 3×3 separable dilated convolution , 3×3 separable dilated group convolution .

9. The hyperspectral image reconstruction method based on spectral perception and structure search according to claim 1, characterized in that: In step 2, the spectrum-aware dynamic structure network is a spectrum-aware dynamic structure network with optimized parameters, and the parameter optimization includes the following steps: Step S1: performing image preprocessing on the degraded hyperspectral image dataset, including degraded hyperspectral images and lossless hyperspectral images; Step S2: input the preprocessed degraded hyperspectral image and the lossless hyperspectral image into the spectrum-aware dynamic structure network; Step S3: Calculate the loss function and optimize the model parameters through gradient descent and back propagation.

10. A hyperspectral image reconstruction system based on spectral perception and structure search, characterized in that: The following units are included: Image preprocessing unit: used to obtain degraded hyperspectral images and perform normalization operations; The network processing unit is used to construct a spectral perception dynamic structure network and input the preprocessed degraded hyperspectral image into the network to obtain a reconstructed hyperspectral image; The spectrum perception dynamic structure network includes shallow feature extraction, deep feature extraction, and feature reconstruction; The shallow feature extraction takes degraded hyperspectral images as input and outputs shallow feature embeddings; The deep feature extraction includes multiple residual stacking structures, each residual stacking structure includes a microstructure and a convolutional layer, accepts shallow feature embedding, and outputs deep image features; The microstructure consists of a convolutional layer and four nodes. The nodes are densely connected to each other through a directed acyclic graph. The computational units between the nodes are determined by a hybrid residual block. Finally, the features of all nodes are merged and skip-connected with the convolutional layer to output a single-step deep feature. The feature reconstruction input is the deep image feature, and after performing residual connection with the input image, the reconstructed hyperspectral image output is obtained. The number of input channels is the number of deep image feature channels, and the number of output channels is the number of reconstructed hyperspectral image channels.

Images