A hyperspectral image reconstruction method based on spectral and spatial dual-prior frequency domain enhancement

Abstract

The present application belongs to the field of hyperspectral reconstruction in coded aperture snapshot spectral imaging system, and particularly relates to a hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement, comprising: obtaining a hyperspectral image, a system measurement value of the hyperspectral image and an imaging mask according to a coded aperture snapshot spectral imaging system, inputting the system measurement value and the imaging mask into a trained hyperspectral image reconstruction model to obtain a reconstructed hyperspectral image; the hyperspectral image reconstruction model comprises an imaging mask preprocessing module, a spatial prior denoising reconstruction module and a convolution layer; the spatial prior denoising reconstruction module is adopted to recover spatial information from different angles by using spatial learning and frequency domain learning two branches, the frequency domain learning branch uses the imaging modulation mask as spatial prior to assist in recovering spatial detail features, and the hyperspectral reconstruction quality of the model is improved.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral reconstruction in coded aperture snapshot spectral imaging systems, specifically involving a hyperspectral image reconstruction method based on spatial-spectral dual-priority frequency domain enhancement. Background Technology

[0002] Unlike three-band RGB images, hyperspectral images contain hundreds of narrow and continuous spectral bands, providing rich information on object composition, material properties, and other characteristics. Hyperspectral images have been widely used in remote sensing, anomaly detection, and medical imaging. However, due to the trade-off between spatial and spectral resolution, traditional hyperspectral imagers cannot capture dynamic scenes. To overcome this problem, a coding-based system has been proposed using compressed sensing theory, including the coded aperture snapshot spectral imaging system. This system has been widely used for low-cost hyperspectral image acquisition. The coded aperture snapshot spectral imaging system compresses three-dimensional hyperspectral images into two-dimensional measurements by introducing a binary coded aperture, and then uses reconstruction algorithms to recover the desired hyperspectral image.

[0003] Reconstructing a 3D hyperspectral image from 2D measurement data is an ill-posed inverse problem. Traditional model-based methods address this optimization problem by using handcrafted priors as regularization, including sparsity priors, total variational priors, and low-rank priors. These methods focus on modeling the linear features of hyperspectral images, but their performance on real-world data is often unsatisfactory. Unlike model-based methods, state-of-the-art deep learning models possess powerful nonlinear modeling capabilities and achieve state-of-the-art reconstruction performance. Typical deep learning models include end-to-end mapping networks and deep unrolling networks. However, existing models do not fully consider the inherent characteristics of hyperspectral images and the imaging principles of coded aperture snapshot spectral imaging systems. This makes the models highly complex to achieve good nonlinear mapping capabilities, which significantly limits the reconstruction efficiency of hyperspectral images. To explore modulation mask information in coded aperture snapshot spectral imaging systems, some methods multiply the mask by the measurement value as the network input. Although this improves reconstruction quality, it destroys the spectral information stored in the measurement value, limiting the performance improvement. Furthermore, hyperspectral images store continuous spectral bands, resulting in strong correlations between adjacent spectral bands.

[0004] The existing technologies mentioned above do not effectively utilize the imaging modulation mask in the coded aperture snapshot spectral imaging system for frequency domain enhancement and recovery of spatial detail features, nor do they effectively utilize the interspectral correlation characteristics of hyperspectral spectra, resulting in limited reconstruction quality. Summary of the Invention

[0005] To address the aforementioned problems, this invention employs a hyperspectral image reconstruction method based on spatial-spectral dual-priority frequency domain enhancement, comprising:

[0006] The system measurements and imaging mask of the hyperspectral image are obtained by the coded aperture snapshot spectral imaging system. The system measurements and imaging mask are then input into a trained hyperspectral image reconstruction model to obtain the reconstructed hyperspectral image. The hyperspectral image reconstruction model includes a measurement preprocessing module, an imaging mask preprocessing module, a spatial prior noise reduction and reconstruction module, and a convolutional layer.

[0007] The training process for a hyperspectral image reconstruction model includes:

[0008] S1. Obtain the real hyperspectral image, the system measurement value of the real hyperspectral image, and the imaging mask based on the coded aperture snapshot spectral imaging system;

[0009] S2. Input the system measurement values ​​into the measurement value preprocessing module to obtain the measurement feature X;

[0010] S3. Input the imaging mask into the imaging mask preprocessing module to obtain the mask features.

[0011] S4. Combine the measurement feature X and the mask feature. Input the spatial prior noise reduction and reconstruction module to obtain the noise reduction and reconstruction features;

[0012] S5. Input the noise reduction and reconstruction features into the convolutional layer for dimensionality transformation to obtain the reconstructed hyperspectral image;

[0013] S6. Calculate the overall loss function value based on the reconstructed hyperspectral image and the real hyperspectral image, update the model parameters based on the overall loss function value, and complete the model training when the overall loss function is minimized.

[0014] The imaging mask preprocessing module processes the imaging mask by expanding it along the channel dimension to obtain mask M. * Using convolution pairs M * Feature extraction is performed to obtain mask information. Then, discrete cosine transform is applied to the mask information to obtain mask features.

[0015] The measurement preprocessing module processes the system's measurement values, including:

[0016] The initial signal H is obtained by performing inverse dispersion shift on the system measurement value. The measurement feature X is obtained by performing preliminary feature extraction on H using 3×3 convolution.

[0017] The spatial prior denoising and reconstruction module includes multiple frequency domain learning modules and multiple denoising and reconstruction modules; the spatial prior denoising and reconstruction module applies the measured features X and mask features... The processing includes:

[0018] S41. Mask features Input the first frequency domain learning module to obtain the first frequency domain learning result.

[0019] S42. Combine the measured feature X with the first frequency domain learning result. Input the first noise reduction and reconstruction module to obtain the first noise reduction and reconstruction feature X1;

[0020] S43. Apply the previous frequency domain learning results Input the current frequency domain learning module to obtain the current frequency domain learning result. The previous noise reduction and reconstruction feature X i-1 and the current frequency domain learning results Input the current noise reduction and reconstruction module to obtain the current noise reduction and reconstruction feature X. i ;

[0021] S44. Repeat step S43 until all noise reduction and reconstruction modules and frequency domain learning modules have been processed to obtain the final noise reduction and reconstruction feature X. N N represents the number of noise reduction and reconstruction modules.

[0022] The noise reduction and reconstruction module includes: a normalization layer, a spatial frequency domain modeling module, a convolutional layer, and a GFNB module; the noise reduction and reconstruction module modifies the previous noise reduction and reconstruction feature X. i-1 and the current frequency domain learning results The processing includes:

[0023] The previous noise reduction and reconstruction feature X i-1 Input the normalization layer, and combine the output of the normalization layer with the current frequency domain learning result. Input the spatial frequency domain modeling module, input the output of the spatial frequency domain modeling module into the convolutional layer, and input the output of the convolutional layer and the previous denoising and reconstruction feature X. i-1 The sum is then input into the GFNB module to obtain the current denoising and reconstruction feature X. i ; where GFNB is a gated depthwise convolutional feedforward network module.

[0024] The spatial frequency domain modeling module processes the output of the normalization layer and the frequency domain learning results, including:

[0025]

[0026] Where DCT[·] represents the Discrete Cosine Transform, IDCT[·] represents the Inverse Discrete Cosine Transform, and X i LN The output of the normalization layer is represented by SLB, which stands for Spatial Learning Module, and FLB stands for Frequency Domain Learning Module. This represents the frequency domain learned feature of the i-th FLB output. This is the output of the spatial frequency domain modeling module.

[0027] The spatial learning module includes a feature extraction module, an average pooling module, and a fusion module; spatial learning

[0028] The module processes the output of the normalization layer, including:

[0029] The output X of the normalization layer i LN Input the feature extraction module to obtain the feature extraction result; input the feature extraction result into the average pooling module to obtain the average pooling result; combine the average pooling result with X... i LN The feature extraction results are input into the fusion module to obtain the output results of the spatial learning module.

[0030] The GFNB module includes: normalization layer, convolutional layer, depthwise convolutional layer, and GELU activation layer.

[0031] The frequency domain learning module includes convolutional layers, GELU activation layers, and deep convolutional layers.

[0032] Overall loss function L total include:

[0033] L total =L Rec +λL SS

[0034] L SS =MSE(S Rec ,S GT )

[0035] Where λ represents the hyperparameter, L total Let L be the overall loss function. Rec L is the pixel-level reconstruction loss function. ss S represents the interspectral similarity loss function. Rec and S GT Let MSE[·] represent the inter-spectral similarity matrix of the reconstructed hyperspectral spectrum and the true hyperspectral spectral similarity matrix, respectively, and MSE[·] represent the mean square error function.

[0036] Beneficial effects:

[0037] 1. This invention, based on the imaging principle of the coded aperture snapshot spectral imaging system, employs a lightweight model with frequency domain enhancement, improving the hyperspectral reconstruction quality of the model. 2. This invention uses a noise reduction and reconstruction module that utilizes two branches—spatial learning and frequency domain learning—to recover spatial information from different angles. The frequency domain learning branch uses an imaging modulation mask as a spatial prior to assist in recovering spatial detail features, thus improving the hyperspectral reconstruction quality of the model. 3. Based on the spectral similarity characteristics, this invention employs a spectral similarity loss to improve the recovery of spectral information. 4. The assistance of spatial and spectral priors enables the model to reconstruct high-quality hyperspectral images with a small number of parameters, improving the reconstruction inference speed of the model while ensuring reconstruction quality. Attached Figure Description

[0038] Figure 1 A flowchart of a hyperspectral image reconstruction method based on spatial spectrum dual prior frequency domain enhancement provided in an embodiment of the present invention;

[0039] Figure 2 This is a framework diagram of a hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement provided in an embodiment of the present invention. Detailed Implementation

[0040] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] like Figure 1 , Figure 2 As shown, a hyperspectral image reconstruction method based on spatial-spectral dual-priority frequency domain enhancement includes:

[0042] S1. Obtain real hyperspectral images and their system measurements and imaging masks based on the coded aperture snapshot spectral imaging system;

[0043] S2. Input the system measurement values ​​into the measurement value preprocessing module to obtain the measurement feature X;

[0044] The initial signal H is obtained by performing inverse dispersion shift (SID) on the system measurement values. The measurement features X are obtained by performing preliminary feature extraction on H using 3×3 convolution.

[0045] S3. Input the imaging mask into the imaging mask preprocessing module to obtain the mask features.

[0046] The imaging mask preprocessing module processes the imaging mask by expanding it along the channel dimension to obtain M. * Using 1×1 convolution pairs M * Feature extraction is performed to obtain mask information. Then, a Discrete Cosine Transform (DCT) is applied to the mask information to obtain the mask features. Mask features Provide each Noise-reduced Reconstruction Module (NRM) with mask prior information for the corresponding stage to assist in reconstruction;

[0047] S4. Combine the measurement feature X and the mask feature. Input the spatial prior noise reduction and reconstruction module to obtain the noise reduction and reconstruction features;

[0048] The spatial prior denoising and reconstruction module includes multiple frequency domain learning modules and multiple denoising and reconstruction modules; the spatial prior denoising and reconstruction module applies the measured features X and mask features... The processing includes:

[0049] S41. Mask features Input the first frequency domain learning module to obtain the first frequency domain learning result.

[0050] S42. Combine the measured feature X with the first frequency domain learning result. Input the first noise reduction and reconstruction module to obtain the first noise reduction and reconstruction feature X1;

[0051] S43. Apply the previous frequency domain learning results Input the current frequency domain learning module to obtain the current frequency domain learning result. The previous noise reduction and reconstruction feature X i-1 and the current frequency domain learning results Input the current noise reduction and reconstruction module to obtain the current noise reduction and reconstruction feature X. i ;

[0052] S44. Repeat step S43 until all noise reduction and reconstruction modules and frequency domain learning modules have been processed to obtain the final noise reduction and reconstruction feature X. N N represents the number of noise reduction and reconstruction modules.

[0053] The denoising and reconstruction module includes: a normalization layer, a spatial modeling module, a convolutional layer, and a GFNB module; the denoising and reconstruction module processes the previous denoised and reconstructed feature X. i-1 and the current frequency domain learning results The processing includes:

[0054] The previous noise reduction and reconstruction feature Xi-1 Input the normalization layer to stabilize network training, and combine the output of the normalization layer with the current frequency domain learning result. The input spatial frequency domain modeling module learns spatial information from different perspectives. The output of the spatial frequency domain modeling module is input into a 1×1 convolutional layer. The output of the convolutional layer is then combined with the previous denoising and reconstructed feature X. i-1 The sum is then input into the GFNB module to fully integrate spatial and frequency domain feature information, resulting in the current denoising and reconstruction feature X. i ; where GFNB stands for Gated-dconv Feed-forward Network Block.

[0055] The spatial frequency domain modeling module processes the output of the normalization layer and the current frequency domain learning results, including:

[0056]

[0057] Where DCT[·] represents the Discrete Cosine Transform, IDCT[·] represents the Inverse Discrete Cosine Transform, and X i LN This represents the output of the normalization layer. The normalization method used is Layer Normalization (LN). SLB stands for Spatial Learning Block, and FLB stands for Frequency Learning Block. This represents the i-th frequency domain learned feature.

[0058] The spatial learning module includes a feature extraction module, an average pooling module, and a fusion module; the spatial learning module processes the output of the normalization layer, including:

[0059] The output X of the normalization layer i LN The input feature extraction module undergoes four convolutions and one GELU activation. The output of the feature extraction module is then input into the average pooling module, which performs one average pooling, two convolutions, one ReLU activation, and one sigmoid activation to obtain the average pooling result. This average pooling result is multiplied by the feature extraction result, and the multiplied result is then convolved. The convolution result is then multiplied by X. i LN Add them together to get the output of the spatial learning module.

[0060] The feature extraction module includes four convolutional layers and one GELU activation layer; the average pooling module includes an average pooling layer, two convolutional layers, one ReLU activation layer, and one sigmoid layer; the fusion layer includes convolutional layers.

[0061] The GFNB module includes: normalization layer, convolutional layer, depthwise convolutional layer, and GELU activation layer;

[0062] The input of the GFNB module is fed into a normalization layer. The normalized result is then subjected to one convolution and one depthwise convolution to obtain feature F1. The normalized result is then subjected to one convolution, one depthwise convolution, and one GELU activation to obtain feature F2. Feature F1 and feature F2 are multiplied together, and the multiplication result is then convolved to obtain the convolution result. The convolution result and the normalized result are then added together to obtain the output of the GFNB module.

[0063] The frequency domain learning module includes convolutional layers, GELU activation layers, and deep convolutional layers;

[0064] S5. Input the noise reduction and reconstruction features into the convolutional layer and perform dimensional transformation so that the dimension is consistent with the dimension of the hyperspectral image to be recovered, thereby obtaining the reconstructed hyperspectral image;

[0065] S6. Calculate the loss function value based on the reconstructed hyperspectral image and the real hyperspectral image, update the model parameters based on the loss function value, and complete the model training when the loss function converges.

[0066] The overall loss function of the model includes:

[0067] L total =L Rec +λL SS

[0068] Where λ represents the hyperparameter, L ss L represents the interspectral similarity loss. Rec This represents pixel-level reconstruction loss.

[0069] The interspectral similarity loss is:

[0070] L SS =MSE(S Rec ,S GT )

[0071] Among them, L ss S represents the interspectral similarity loss; Rec and S GT represents the spectral similarity matrix of the reconstructed hyperspectrum and the spectral similarity matrix of the true hyperspectrum, respectively; MSE[·] represents the mean square error function.

[0072] The reconstructed hyperspectral inter-spectral similarity matrix and the true hyperspectral spectral similarity matrix are obtained by calculating the cosine similarity between each channel of the hyperspectral spectrum and other channels. The cosine similarity s between channels is... ij Calculated using the following formula:

[0073]

[0074] Where i and j represent channel indices, c i and c j Represents the spectral vectors of the i-th and j-th channels. The pixel-level reconstruction loss of the model includes:

[0075] L Rec =RMSE(H Rec H GT )

[0076] Where RMSE[·] represents the root mean square error function, H Rec H GT These represent the hyperspectral image reconstructed by the model and the actual hyperspectral image, respectively.

[0077] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hyperspectral image reconstruction method based on spectral and spatial dual-prior frequency domain enhancement, characterized in that, include: The system measurements and imaging mask of the hyperspectral image are obtained from the coded aperture snapshot spectral imaging system. The system measurements and imaging mask are then input into the trained hyperspectral image reconstruction model to obtain the reconstructed hyperspectral image. The hyperspectral image reconstruction model includes a measurement preprocessing module, an imaging mask preprocessing module, a spatial prior noise reduction and reconstruction module, and a convolutional layer. The training process for a hyperspectral image reconstruction model includes: S1. Obtain the real hyperspectral image, the system measurement value of the real hyperspectral image, and the imaging mask based on the coded aperture snapshot spectral imaging system; S2, input the system measurement value into the measurement value preprocessing module to obtain measurement features ; S3, input the imaging mask into the imaging mask preprocessing module to obtain mask features ; S4, measuring the feature and the mask feature an input spatial prior denoising reconstruction module, to obtain a denoised reconstructed feature The spatial-prior denoising reconstruction module includes a plurality of frequency domain learning modules and a plurality of denoising reconstruction modules; the spatial-prior denoising reconstruction module processes the measurement feature and the mask feature The processing includes: S41. Mask features Input the first frequency domain learning module to obtain the first frequency domain learning result. ; S42, Measuring features and the first frequency domain learning results Input the first noise reduction and reconstruction module to obtain the first noise reduction and reconstruction features. ; S43. Apply the previous frequency domain learning results Input the current frequency domain learning module to obtain the current frequency domain learning result. ; Reconstruct the previous noise reduction feature and the current frequency domain learning results Input the current noise reduction and reconstruction module to obtain the current noise reduction and reconstruction features. ; The noise reduction and reconstruction module includes: a normalization layer, a spatial frequency domain modeling module, a convolutional layer, and a GFNB module; the noise reduction and reconstruction module processes the previous noise reduction and reconstruction feature. and the current frequency domain learning results The processing includes: Reconstruct the previous noise reduction feature Input the normalization layer, and combine the output of the normalization layer with the current frequency domain learning result. Input the spatial frequency domain modeling module, input the output of the spatial frequency domain modeling module into the convolutional layer, and input the output of the convolutional layer and the previous noise reduction and reconstruction features. The sum is then input into the GFNB module to obtain the current denoising and reconstruction features. ; where GFNB is a gated depthwise convolutional feedforward network module; The spatial frequency domain modeling module processes the output of the normalization layer and the frequency domain learning results, including: in, Represents the discrete cosine transform. This represents the inverse of the discrete cosine transform. The output of the normalization layer is represented by SLB, which stands for Spatial Learning Module, and FLB stands for Frequency Domain Learning Module. This represents the i-th frequency domain learned feature. The output of the spatial frequency domain modeling module; S44. Repeat step S43 until all noise reduction and reconstruction modules and frequency domain learning modules have been processed to obtain the final noise reduction and reconstruction features. , This refers to the number of noise reduction and reconstruction modules; S5. Input the noise reduction and reconstruction features into the convolutional layer for dimensionality transformation to obtain the reconstructed hyperspectral image; S6. Calculate the overall loss function value based on the reconstructed hyperspectral image and the real hyperspectral image, update the model parameters based on the overall loss function value, and complete the model training when the overall loss function is minimized.

2. The hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement according to claim 1, characterized in that, The imaging mask preprocessing module processes the imaging mask by expanding it along the channel dimension to obtain the mask. Using convolution pairs Feature extraction is performed to obtain mask information. Then, discrete cosine transform is applied to the mask information to obtain mask features. .

3. The hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement according to claim 1, characterized in that, The measurement preprocessing module processes the system's measurement values, including: The initial signal H is obtained by performing inverse dispersion shift on the system measurements. The measurement features are then extracted from H using convolution. .

4. The hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement according to claim 1, characterized in that, The spatial learning module includes a feature extraction module, an average pooling module, and a fusion module; The spatial learning module processes the output of the normalization layer, including: The output of the normalization layer Input the feature extraction module to obtain the feature extraction results; The feature extraction results are input into the average pooling module to obtain the average pooling results; the average pooling results, The feature extraction results are input into the fusion module to obtain the output results of the spatial learning module.

5. The hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement according to claim 1, characterized in that, The GFNB module includes: normalization layer, convolutional layer, depthwise convolutional layer, and GELU activation layer.

6. The hyperspectral image reconstruction method based on spatial-spectral dual-prior frequency domain enhancement according to claim 1, characterized in that, The frequency domain learning module includes convolutional layers, GELU activation layers, and deep convolutional layers.

7. The hyperspectral image reconstruction method based on spatial-spectral dual-priority frequency domain enhancement according to claim 1, characterized in that, Overall loss function include: in, Indicates hyperparameters, For the overall loss function, For pixel-level reconstruction loss function, This represents the interspectral similarity loss function. and Let represent the inter-spectral similarity matrix of the reconstructed hyperspectrum and the spectral similarity matrix of the true hyperspectrum, respectively. This represents the mean square error function.

Images