Hyperspectral image reconstruction method and device, electronic equipment and storage medium

Abstract

The invention provides a hyperspectral image reconstruction method and device, electronic equipment and a storage medium, and relates to the technical field of hyperspectral imaging, and the method comprises the steps: obtaining compression measurement data and an ideal sensing matrix corresponding to a to-be-reconstructed original hyperspectral image; the compression measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is used for representing an ideal mapping relation between the original hyperspectral image and the compression measurement data; inputting the compression measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstruction image output by the image reconstruction model; the image reconstruction model is obtained by performing iterative training based on degradation deviation between an actual sensing matrix sample and an ideal sensing matrix sample. According to the method, the difference between the ideal degradation model and the actual imaging process can be reduced, and the reconstruction precision of the image reconstruction model obtained through training is improved.

Description

Technical Field

[0001] This invention relates to the field of hyperspectral imaging technology, and in particular to a hyperspectral image reconstruction method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of hyperspectral imaging, traditional methods mainly rely on Code-Aperture Spectral Imaging (CASSI) systems. These systems compress three-dimensional hyperspectral data into two-dimensional measurements using optical coding and compressed sensing techniques, and then reconstruct the original information using reconstruction algorithms. Existing reconstruction methods can be categorized into three types: model-based methods, deep learning-based methods, and deep unfolded networks. Model-based methods (such as TwIST and GAP-TV) utilize hand-designed prior knowledge (such as sparsity and low rank) to construct optimization problems. While offering some interpretability, they rely on strong assumptions and have low computational efficiency, making them unsuitable for complex scenarios. Deep learning-based methods (such as λ-Net and TSA-Net) directly learn mapping relationships through end-to-end networks, significantly improving reconstruction speed, but lacking physical interpretability and ignoring degradation errors in actual imaging systems (such as optical distortion and noise interference). Deep unfolded networks (such as GAP-Net and DAUHST) combine the advantages of model-driven and data-driven approaches, unfolding the iterative optimization process into neural network modules, providing a better balance among existing technologies.

[0003] However, these methods still face significant challenges in practical applications. The mismatch between the static modeling of the sensor matrix and the actual degradation process is one of the main bottlenecks. CASSI systems in real-world scenarios are affected by factors such as phase distortion, spectral shift, and detector noise, while existing methods typically assume that the sensor matrix is ​​ideal and fixed, ignoring deviations in the actual degradation process. This idealized assumption leads to problems such as blurred edges and spectral distortion in the reconstruction results, severely limiting the reconstruction accuracy under complex imaging conditions. This bias is particularly pronounced in fields with high accuracy requirements, such as medical imaging and remote sensing monitoring. Summary of the Invention

[0004] This invention provides a hyperspectral image reconstruction method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies that use a fixed sensing matrix in the hyperspectral image reconstruction process, resulting in low reconstruction accuracy.

[0005] This invention provides a hyperspectral image reconstruction method, comprising the following steps.

[0006] Obtain compressed measurement data and an ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; the compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data.

[0007] The compressed measurement data and the ideal sensing matrix are input into the image reconstruction model to obtain the hyperspectral reconstructed image output by the image reconstruction model.

[0008] The image reconstruction model is obtained through iterative training based on the degradation deviation between the actual sensor matrix samples and the ideal sensor matrix samples.

[0009] According to the hyperspectral image reconstruction method provided by the present invention, the image reconstruction model includes a degradation bias learning module, a temporal-spectral fusion reconstruction module, and a cross-stage information interaction module; The image reconstruction model is trained based on the following steps: S1. Input the compressed measurement data sample and ideal sensing matrix sample corresponding to the original hyperspectral image sample, as well as the previous hyperspectral reconstructed image sample and previous degradation deviation matrix sample corresponding to the previous stage into the degradation deviation learning module to obtain the current degradation deviation matrix sample corresponding to the current stage. S2. Based on the ideal sensing matrix sample and the current degradation deviation matrix sample, determine the current actual sensing matrix sample after compensation in the current stage; S3. Input the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample; S4. Input the previous encoder feature sample from the previous stage and the current multi-scale feature sample into the cross-stage information interaction module to obtain the reconstructed fusion feature sample. S5. Input the reconstructed fusion feature sample into the temporal fusion reconstruction module to obtain the current reconstructed image sample of the current stage, and determine the current reconstructed image sample as the new previous hyperspectral reconstructed image sample; S6. Repeat steps S1 to S5 for iterative training until the iteration stops, at which point the trained image reconstruction model is obtained.

[0010] According to the hyperspectral image reconstruction method provided by the present invention, the temporal-spectral fusion reconstruction module includes a semi-quadratic splitting module and a spatiotemporal prior learning module; The step of inputting the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample includes: The previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample are input into the semi-quadratic splitting module to obtain the auxiliary variable corresponding to the current actual sensing matrix sample; The auxiliary variables and the spatiotemporal prior learning module are input to obtain the current multi-scale feature samples.

[0011] According to the hyperspectral image reconstruction method provided by the present invention, the spatiotemporal prior learning module includes an encoder and a decoder; the current multi-scale feature sample includes the encoder input features of the current stage and the state of the previous decoder layer; The step of inputting the auxiliary variables into the spatiotemporal prior learning module to obtain the current multi-scale feature samples includes: The auxiliary variables are input into the encoder to obtain the encoder input features for the current stage; The previous encoder feature sample and the current stage encoder input feature are input into the decoder to obtain the previous decoder layer state.

[0012] According to the hyperspectral image reconstruction method provided by the present invention, the step of inputting the previous encoder feature samples from the previous stage and the current multi-scale feature samples into the cross-stage information interaction module to obtain reconstructed fused feature samples includes: Based on the multi-scale attention mechanism, the attention weights corresponding to the current stage encoder input features and the previous decoder layer state in the previous encoder feature sample, the current multi-scale feature sample, and the current stage encoder input features and the previous decoder layer state are determined respectively. Based on the previous encoder feature samples, the current stage encoder input features, and the previous decoder layer state, as well as their respective attention weights, the reconstructed fusion feature samples are obtained by fusing them together.

[0013] According to the hyperspectral image reconstruction method provided by the present invention, the decoder includes a nonlocal spectral attention branch, a channel convolution branch, and a spectral-space interaction module; The step of inputting the reconstructed fusion feature samples into the temporal fusion reconstruction module to obtain the current reconstructed image samples at the current stage includes: The reconstructed fused feature samples are respectively input into the nonlocal spectral attention branch and the channel convolution branch in the decoder to obtain the global spectral self-attention feature output by the nonlocal spectral attention branch and the local spatial feature output by the channel convolution branch. The global spectral self-attention features and the local spatial features are input into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage.

[0014] According to the hyperspectral image reconstruction method provided by the present invention, the step of inputting the global spectral self-attention features and the local spatial features into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage includes: Determine the spatial attention map corresponding to the global spectral self-attention feature, and the spectral attention map corresponding to the local spatial feature; By fusing the global spectral self-attention features, the local spatial features, the spatial attention map, and the spectral attention map, the current reconstructed image sample for the current stage is obtained.

[0015] The present invention also provides a hyperspectral image reconstruction apparatus, comprising the following modules.

[0016] The acquisition module is used to acquire compressed measurement data and an ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; the compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data.

[0017] The reconstruction module is used to input the compressed measurement data and the ideal sensing matrix into the image reconstruction model to obtain the hyperspectral reconstructed image output by the image reconstruction model.

[0018] The image reconstruction model is obtained through iterative training based on the degradation deviation between the actual sensor matrix samples and the ideal sensor matrix samples.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the hyperspectral image reconstruction method as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hyperspectral image reconstruction method as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the hyperspectral image reconstruction method as described above.

[0022] The hyperspectral image reconstruction method, apparatus, electronic device, and storage medium provided by this invention, after acquiring compressed measurement data of a two-dimensional image representing the original hyperspectral image, and an ideal sensing matrix characterizing the ideal mapping relationship between the original hyperspectral image and the compressed measurement data, inputs the compressed measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstructed image output by the image reconstruction model. In this invention, the image reconstruction model learns the degradation deviation between actual sensing matrix samples and ideal sensing matrix samples, and dynamically compensates the sensing matrix based on the degradation deviation, rather than using a fixed ideal sensing matrix. This reduces the difference between the ideal degradation model and the actual imaging process, improves the reconstruction accuracy of the trained image reconstruction model, and avoids spatial detail blurring and spectral feature distortion in the hyperspectral reconstructed image. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the hyperspectral image reconstruction method provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the training process of the image reconstruction model provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of the residual learning block provided in an embodiment of the present invention.

[0027] Figure 4 This is a flowchart illustrating the time-spectrum fusion and reconstruction module provided in an embodiment of the present invention.

[0028] Figure 5 This is a comparative schematic diagram of the image reconstruction results provided in the embodiments of the present invention.

[0029] Figure 6 This is a schematic diagram of the structure of the hyperspectral image reconstruction device provided in an embodiment of the present invention.

[0030] Figure 7 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0032] To address the problem of low reconstruction accuracy caused by using a fixed sensor matrix in the hyperspectral image reconstruction process in existing technologies, embodiments of the present invention provide a hyperspectral image reconstruction method. Figure 1 This is a schematic flowchart of the hyperspectral image reconstruction method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes steps 110 and 120.

[0033] Step 110: Obtain compressed measurement data and ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; the compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data.

[0034] The original hyperspectral image is a three-dimensional data cube, represented as x Where H1 represents the height of the original hyperspectral image, W represents the width of the original hyperspectral image, and N... λ This indicates the number of wavelengths corresponding to the original hyperspectral image.

[0035] After encoding, dispersing, and integrating the original hyperspectral image using the CASSI system, the resulting two-dimensional image on the detector is the compressed measurement data.

[0036] An ideal sensing matrix is ​​used to characterize the mapping process from raw three-dimensional hyperspectral images to compressed two-dimensional measurement data under ideal hardware conditions. In the CASSI system, this ideal sensing matrix is ​​determined by an ideal physical model of the coded aperture mask and dispersive elements.

[0037] Step 120: Input the compressed measurement data and the ideal sensing matrix into the image reconstruction model to obtain the hyperspectral reconstructed image output by the image reconstruction model.

[0038] The image reconstruction model is obtained through iterative training based on the degradation deviation between the actual sensor matrix samples and the ideal sensor matrix samples.

[0039] Actual sensor matrix samples are used to describe the real mapping relationship in a real physical system under the influence of degradation bias, including ideal encoding and dispersion effects, as well as degradation bias introduced by hardware factors such as lens distortion, alignment error, stray light and sensor nonlinear response.

[0040] Degradation bias is the difference between actual sensor matrix samples and ideal sensor matrix samples, used to quantify the systematic deviation of the real imaging system relative to the ideal model.

[0041] Specifically, the image reconstruction model is a fully trained model. After constructing the initial image reconstruction model, the initial image reconstruction model learns the degradation deviation between the actual sensing matrix samples and the ideal sensing matrix samples, and dynamically corrects the deviation between the theoretical degradation process and the actual degradation process to obtain the fully trained image reconstruction model.

[0042] After determining the compressed measurement data and the ideal sensing matrix, these are input into the trained image reconstruction model. After adaptive error correction, the reconstructed hyperspectral image is obtained. Compared to existing technologies that ignore this degradation bias or attempt to directly model the complex actual degradation process, the image reconstruction model trained in this embodiment has lower reconstruction complexity and higher reconstruction accuracy.

[0043] The hyperspectral image reconstruction method provided in this invention, after acquiring compressed measurement data of a two-dimensional image representing the original hyperspectral image and an ideal sensing matrix characterizing the ideal mapping relationship between the original hyperspectral image and the compressed measurement data, inputs the compressed measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstructed image output by the image reconstruction model. In this invention, the image reconstruction model learns the degradation deviation between actual sensing matrix samples and ideal sensing matrix samples, and dynamically compensates the sensing matrix based on the degradation deviation, rather than using a fixed ideal sensing matrix. This reduces the difference between the ideal degradation model and the actual imaging process, improves the reconstruction accuracy of the trained image reconstruction model, and avoids spatial detail blurring and spectral feature distortion in the hyperspectral reconstructed image.

[0044] In one embodiment, Figure 2 This is a schematic diagram of the training process of the image reconstruction model provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the image reconstruction model includes a degradation bias learning module (i.e. Figure 2 The D shown in the figure represents the temporal spectrum fusion reconstruction module and the cross-stage information interaction module (i.e., ...). Figure 2 C is shown in the figure.

[0045] The image reconstruction model is trained based on the following steps: S1. Input the compressed measurement data sample and ideal sensing matrix sample corresponding to the original hyperspectral image sample, as well as the previous hyperspectral reconstructed image sample and previous degradation deviation matrix sample corresponding to the previous stage into the degradation deviation learning module to obtain the current degradation deviation matrix sample corresponding to the current stage. S2. Based on the ideal sensing matrix sample and the current degradation deviation matrix sample, determine the current actual sensing matrix sample after compensation in the current stage; S3. Input the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample; S4. Input the previous encoder feature sample from the previous stage and the current multi-scale feature sample into the cross-stage information interaction module to obtain the reconstructed fusion feature sample. S5. Input the reconstructed fusion feature sample into the temporal fusion reconstruction module to obtain the current reconstructed image sample of the current stage, and determine the current reconstructed image sample as the new previous hyperspectral reconstructed image sample; S6. Repeat steps S1 to S5 for iterative training until the iteration stops, at which point the trained image reconstruction model is obtained.

[0046] Specifically, in practical applications, device errors caused by phase aberration are unavoidable. This means that using a fixed sensing matrix predetermined under ideal conditions is almost insufficient to reflect the real degradation process. Therefore, in this embodiment of the invention, the reconstruction task is transformed into a multi-objective optimization problem, and an adaptive method is used to compensate for the degradation deviation between the actual sensing matrix samples and the ideal sensing matrix samples. That is, as shown in Equation (1), the problem of learning the degradation deviation matrix samples is solved jointly as a pre-training task. Equation (1) is: .

[0047] Where x represents the original hyperspectral image sample, and y represents the compressed measurement data sample. Represents the ideal sensing matrix sample. Represents a sample of the degradation bias matrix. Let J(x) denote the Frobenius norm term, J(x) denote the penalty term for x, and R(x) denote the Frobenius norm term. )express The penalty term, λ, represents the weight corresponding to J(x). R ( The weight of ) Denotes the optimization term for x. express Optimization items.

[0048] Equation (1) can be broken down into two independent but related problems corresponding to Equations (2) and (3). Equation (2) is: .

[0049] Where Y = y - x.

[0050] Equation (3) is: .

[0051] As can be seen from equations (2) and (3), the learning process of the degradation bias matrix sample and the original hyperspectral image sample is optimized alternately. Equation (2) is solved by the degradation bias learning module, and Equation (3) is solved by the temporal-spectral fusion reconstruction module.

[0052] For the (k+1)th stage (i.e., the current stage), k is an integer greater than or equal to 0 and less than or equal to K-1, where K represents the total number of iteration stages. First, the compressed measurement data sample and the ideal sensing matrix sample, as well as the previous hyperspectral reconstructed image sample and the previous degradation deviation matrix sample corresponding to the previous stage, are input into the degradation deviation learning module to obtain the current degradation deviation matrix sample corresponding to the current stage. To solve equation (2), an auxiliary variable z needs to be introduced into equation (1), so equation (1) is transformed into equation (4), which is: .

[0053] Where τ represents the penalty parameter.

[0054] Based on the update of equation (1), in this embodiment of the invention, the solution of equation (2) is divided into equation (5) and equation (6) in the degradation bias learning module using the semi-quadratic splitting method. Equation (5) is: .

[0055] in, This represents the current degradation deviation matrix sample corresponding to the (k+1)th stage (i.e., the current stage). x represents the sample of the previous degradation deviation matrix corresponding to the previous stage. k This represents the previous hyperspectral reconstructed image sample corresponding to the previous stage, z k This represents the auxiliary variable for the k-th stage (i.e., the previous stage).

[0056] Equation (5) is a typical quadratic regularized least squares problem. In the degradation bias learning module, Equation (7) can be used to obtain the current degradation bias matrix sample corresponding to the (k+1)th stage. Equation (7) is: .

[0057] Equation (6) is: .

[0058] in, This represents the auxiliary variable corresponding to the (k+1)th stage. The degradation bias learning module includes a bias learning module, which contains multiple residual learning blocks (RLBs). Figure 3 This is a schematic diagram of the structure of the residual learning block provided in an embodiment of the present invention. The structure of the residual learning block is as follows: Figure 3 As shown. Inputting the previous degradation bias matrix sample and the previous hyperspectral reconstructed image sample corresponding to the previous stage into the bias learning module, equation (6) can be solved to obtain the auxiliary variables corresponding to the k+1th stage.

[0059] Next, the sum of the ideal sensing matrix sample and the current degradation bias matrix sample is calculated. This sum is the current actual sensing matrix sample after compensation for the current stage. Represented as: .

[0060] Subsequently, existing methods exhibit significant shortcomings in feature fusion and feature propagation. Regarding spatial-spectral feature fusion, most methods employ sequential or alternating processing (such as the self-attention mechanism in TSA-Net), failing to fully exploit the unique cross-dimensional correlations of hyperspectral data. In multi-stage optimization, while deep unfolded networks improve reconstruction performance through phased processing, the lack of effective information interaction mechanisms between stages makes it difficult for deep networks to fully utilize key features extracted from shallow layers. This information attenuation phenomenon is particularly prominent in long-sequence spectral reconstruction tasks, making it difficult for existing methods to achieve globally optimal reconstruction results. Therefore, in this embodiment of the invention, the previous hyperspectral reconstructed image sample, compressed measurement data sample, and the current actual sensing matrix sample are input into the temporal-spectral fusion reconstruction module to obtain the current multi-scale feature sample. This current multi-scale feature sample and the previous encoder feature sample from the previous stage are then input into the cross-stage information interaction module to fuse the feature samples from different sources and scales, obtaining a reconstructed fused feature sample. This selectively preserves and propagates key features between iteration stages, thereby maintaining the fidelity of the reconstruction. After fusing the reconstructed fused feature samples, the cross-stage information interaction module feeds these samples back to the temporal-spectral fusion reconstruction module. Through a cross-dimensional attention mechanism, it simultaneously captures spatial-spectral dependencies, achieving efficient feature fusion between different hyperspectral image channels to reconstruct the current reconstructed image sample for the current stage. Then, based on the current reconstructed image sample and the original hyperspectral image sample, a loss function is calculated. The convergence of the image reconstruction model is then determined based on the loss function. If convergence fails, the model parameters are updated, and the above steps are repeated. If convergence occurs, training ends, resulting in a successfully trained image reconstruction model.

[0061] In one embodiment, the temporal spectrum fusion reconstruction module includes a semi-quadratic splitting module and a spatiotemporal prior learning module; The step of inputting the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample includes: The previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample are input into the semi-quadratic splitting module to obtain the auxiliary variable corresponding to the current actual sensing matrix sample; The auxiliary variables and the spatiotemporal prior learning module are input to obtain the current multi-scale feature samples.

[0062] Specifically, guided by the current actual sensing matrix samples, equation (3) can be simplified to the conventional hyperspectral reconstruction problem as shown in equation (8), which is: .

[0063] Where, x k+1 Let represent the current reconstructed image sample corresponding to the (k+1)th stage. Then, equation (8) is decomposed into a joint problem corresponding to equations (9) and (10). Equation (9) is: .

[0064] in, This represents the penalty parameter.

[0065] Equation (10) is: .

[0066] Equation (10) is solved using a semi-quadratic splitting module to obtain the auxiliary variable for the (k+1)th stage. Then, this auxiliary variable for the (k+1)th stage is input into the spatiotemporal prior learning module, and Equation (11) is used to calculate the current multi-scale feature sample that can be input into the cross-stage information interaction module for feature fusion. Equation (11) is: .

[0067] In one embodiment, the spatiotemporal prior learning module includes an encoder and a decoder; the current multi-scale feature sample includes the encoder input features of the current stage and the state of the previous decoder layer; The step of inputting the auxiliary variables into the spatiotemporal prior learning module to obtain the current multi-scale feature samples includes: The auxiliary variables are input into the encoder to obtain the encoder input features for the current stage; The previous encoder feature sample and the current stage encoder input feature are input into the decoder to obtain the previous decoder layer state.

[0068] Specifically, the spatiotemporal prior learning module includes an encoder and decoder with a U-shaped structure, consisting of five structurally similar layers. The encoder includes three Spatial-Spectral Interaction Transformers (S...). 2 IT), and the decoder includes two S 2 IT. The auxiliary variables for the current stage are first input into the first S in the encoder. 2 IT, got the first S 2 Reconstructed image samples output by IT. For the first S... 2 After downsampling the reconstructed image samples output by IT, the second S is input. 2 IT, got the second S 2 Reconstructed image samples output by IT. For the second S... 2 After downsampling the reconstructed image samples output by IT, the third S is input. 2 IT, got the third S 2 The reconstructed image samples output by IT, and three S 2 During the computation process, IT outputs the encoder input features of the current stage and the current encoder feature sample of the current stage. This current encoder feature sample participates in the feature fusion calculation of the (k+2)th stage. The encoder input features of the current stage are output to the cross-stage information interaction module. Then, the previous encoder feature sample and the current stage encoder input features are input into the decoder, and the third S in the encoder is... 2 The first S in the reconstructed image sample input to the decoder from the IT output. 2 IT, for the third S in the encoder 2 The reconstructed image samples output by IT are upsampled to obtain the first S in the decoder. 2 The reconstructed image sample output by IT and the state of the previous decoder layer.

[0069] In one embodiment, inputting the previous encoder feature samples from the previous stage and the current multi-scale feature samples into the cross-stage information interaction module to obtain reconstructed fused feature samples includes: Based on the multi-scale attention mechanism, the attention weights corresponding to the current stage encoder input features and the previous decoder layer state in the previous encoder feature sample, the current multi-scale feature sample, and the current stage encoder input features and the previous decoder layer state are determined respectively. Based on the previous encoder feature samples, the current stage encoder input features, and the previous decoder layer state, as well as their respective attention weights, the reconstructed fusion feature samples are obtained by fusing them together.

[0070] Specifically, to enhance reconstruction performance, the state of the previous decoder layer, the feature samples of the previous encoder, and the input features of the current stage encoder are input into the cross-stage information interaction module for feature fusion. To fuse these three features at different scales, Equation (12) is used to calculate the attention weights corresponding to each of the three features based on the convolutional attention mechanism. Equation (12) is as follows: .

[0071] Where H2 represents the encoder input features at the current stage, h represents the attention weights corresponding to the encoder input features at the current stage, D1 represents the state of the previous decoder layer, d represents the attention weights corresponding to the state of the previous decoder layer, F represents the previous encoder feature sample, f represents the attention weights corresponding to the previous encoder feature sample, GAP (Global Average Pooling) concatenates H2, D1, and F, and performs global average pooling on the concatenated features to obtain a vector representing global information. Conv(GAP(H2,D1,F)) convolves the vector representing global information. Gelu (Gaussian Error Linear Unit) uses the Gelu activation function for non-linear mapping. Conv(Gelu(Conv(GAP(H2,D1,F)))) maps the features to the required channel dimension through a convolutional layer, and then transforms the output features to the range of 0 to 1 through the sigmoid function to generate the final attention weight map Attention(h,d,f), which includes the attention weights corresponding to H2, D1, and F.

[0072] Then, using equation (13), the reconstructed fused feature samples are obtained by fusing the current encoder input features, the previous decoder layer state, and the previous encoder feature samples, along with their respective attention weights. Equation (13) is: .

[0073] Where, x out Indicates the reconstructed fusion feature samples, This indicates element-wise multiplication. After the reconstructed fusion feature samples are obtained by fusing in the cross-stage information interaction module, they are then sent to the temporal spectrum fusion reconstruction module, effectively reducing information loss between stages.

[0074] In one embodiment, the decoder includes a nonlocal spectral attention branch, a channel convolution branch, and a spectral-space interaction module; The step of inputting the reconstructed fusion feature samples into the temporal fusion reconstruction module to obtain the current reconstructed image samples at the current stage includes: The reconstructed fused feature samples are respectively input into the nonlocal spectral attention branch and the channel convolution branch in the decoder to obtain the global spectral self-attention feature output by the nonlocal spectral attention branch and the local spatial feature output by the channel convolution branch. The global spectral self-attention features and the local spatial features are input into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage.

[0075] Specifically, Figure 4 This is a flowchart illustrating the time-spectrum fusion and reconstruction module provided in an embodiment of the present invention, as shown below. Figure 4 As shown, each S 2 The IT module includes a non-local spectral attention branch, a channel convolution branch, and a spectral-space interaction module. The non-global spectral attention branch extracts global spectral features, the channel convolution branch extracts local spatial features, and the spectral-space interaction module fuses global spectral features and local spatial features.

[0076] With S in the encoder 2 Taking IT as an example, after receiving the reconstructed fusion feature sample sent by the cross-stage information interaction module, the reconstructed fusion feature sample is input into the nonlocal spectral attention branch and the channel convolution branch, respectively, and the two branches are executed in parallel. Specifically, the nonlocal spectral attention branch divides the reconstructed fusion feature sample into several sub-blocks, each of size L×L×N. λ Then, similar to conventional attention mechanisms, the size of these sub-blocks is readjusted to LL×N. λ The query (Q), key (K), and value (V) are embedded, and the self-attention feature corresponding to each sub-block is calculated using Equation (14). Equation (14) is: .

[0077] Where Attention(q) i ,k i ,v i ) represents sub-block x i Self-attention features, q i =x i Q, k i =x i K, v i =x i V, β represent learnable scaling parameters used to control k i and q i The size of the dot product. After determining the self-attention features corresponding to all sub-blocks, all self-attention features are concatenated to obtain the global spectral self-attention features corresponding to the reconstructed fused feature sample.

[0078] Meanwhile, the channel convolution branch uses equation (15) to extract the local spatial features corresponding to the reconstructed fusion feature sample. Specifically, it first performs a convolution operation on the reconstructed fusion feature sample to output a new feature map. Then, it normalizes the new feature map output by the convolution layer and inputs the normalized feature map into the activation function to obtain the local spatial features. Equation (15) is: F s =Gelu(Norm(Conv(x) out ))).

[0079] Among them, F s This represents local spatial features, and Norm represents normalization.

[0080] After determining the global spectral self-attention features and local spatial features, in order to adaptively and efficiently integrate spatial and spectral information, the global spectral self-attention features and local spatial features are input into the spectral-space interaction module for fusion to obtain the current reconstructed image sample after the current stage of reconstruction.

[0081] In one embodiment, such as Figure 4 As shown, the step of inputting the global spectral self-attention features and the local spatial features into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage includes: Determine the spatial attention map corresponding to the global spectral self-attention feature, and the spectral attention map corresponding to the local spatial feature; By fusing the global spectral self-attention features, the local spatial features, the spatial attention map, and the spectral attention map, the current reconstructed image sample for the current stage is obtained.

[0082] Specifically, the spectral-space interaction module includes a spectral channel attention branch (Spectral-CA) and a spatial channel attention branch (Spatial-CA). Global spectral self-attention features are input into the spatial channel attention branch (Spatial-CA) to calculate the spatial attention map of spatial information fusion, and local spatial features are input into the spectral channel attention branch (Spectral-CA) to calculate the spectral attention map. Then, using equation (16), the current reconstructed image sample for the current stage is calculated. Equation (16) is: x k+1 =conv(concate(S a ⊙F s ,S e ⊙F c )).

[0083] Where, x k+1 S represents the current reconstructed image sample at the current stage. a Represents a spatial attention map, Fs S represents local spatial features. e Represents the spectral attention map, F c This represents the global spectral self-attention feature, and concate represents the splicing operation.

[0084] It should be noted that in this embodiment of the invention, two widely used hyperspectral image datasets, CAVE and KAIST, are used to train and test the image reconstruction model, respectively. All scenes in the CAVE dataset are used to train the model, while representative scenes are selected from the KAIST dataset as the test dataset. Furthermore, in this embodiment, the Adam optimizer (β1=0.9, β2=0.999) is used to train the model. The number of training epochs is set to 300, the initial learning rate is 0.0005, and the decay rate is 0.5 every 50 epochs.

[0085] Furthermore, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are used as evaluation metrics. PSNR measures pixel-level accuracy, with higher values ​​indicating better performance. SSIM assesses perceptual structural similarity, with values ​​closer to 1 indicating better performance. Next, the image reconstruction method provided in this embodiment is compared with existing D... 2 The PL-Net method was compared, and the comparison results are shown in Table 1. As can be seen from Table 1, the PSNR and SSIM of the image reconstruction method provided in this embodiment are superior to existing D... 2 PL-Net method.

[0086] Table 1

[0087] Simultaneously, test images with wavelengths of 466nm, 568nm, and 615nm were selected from the KAIST dataset for reconstruction. The reconstructed image samples from this embodiment were then compared with real hyperspectral images (i.e., original hyperspectral image samples) and existing D... 2 Comparison of hyperspectral images reconstructed using the PL-Net method. Figure 5 This is a comparative schematic diagram of the image reconstruction results provided in the embodiments of the present invention. The comparison results are as follows: Figure 5 As shown, the reconstructed image sample generated by the embodiment of the present invention is closer to the real hyperspectral image, with richer spatial information and clearer edges.

[0088] The hyperspectral image reconstruction apparatus provided by the present invention is described below. The hyperspectral image reconstruction apparatus described below can be referred to in correspondence with the hyperspectral image reconstruction method described above.

[0089] This invention also provides a hyperspectral image reconstruction apparatus. Figure 6This is a schematic diagram of the hyperspectral image reconstruction device provided in an embodiment of the present invention, as shown below. Figure 6 As shown, the hyperspectral image reconstruction device 600 includes an acquisition module 610 and a reconstruction module 620.

[0090] The acquisition module 610 is used to acquire compressed measurement data and an ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; the compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data.

[0091] The reconstruction module 620 is used to input the compressed measurement data and the ideal sensing matrix into the image reconstruction model to obtain the hyperspectral reconstructed image output by the image reconstruction model.

[0092] The image reconstruction model is obtained through iterative training based on the degradation deviation between the actual sensor matrix samples and the ideal sensor matrix samples.

[0093] The hyperspectral image reconstruction apparatus provided in this invention, after acquiring compressed measurement data of a two-dimensional image representing the original hyperspectral image and an ideal sensing matrix characterizing the ideal mapping relationship between the original hyperspectral image and the compressed measurement data, inputs the compressed measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstructed image output by the image reconstruction model. In this invention, the image reconstruction model learns the degradation deviation between actual sensing matrix samples and ideal sensing matrix samples, and dynamically compensates the sensing matrix based on the degradation deviation, rather than using a fixed ideal sensing matrix. This reduces the difference between the ideal degradation model and the actual imaging process, improves the reconstruction accuracy of the trained image reconstruction model, and avoids spatial detail blurring and spectral feature distortion in the hyperspectral reconstructed image.

[0094] Optionally, the image reconstruction model includes a degradation bias learning module, a temporal-spectral fusion reconstruction module, and a cross-stage information interaction module.

[0095] Optionally, the hyperspectral image reconstruction apparatus 600 further includes a training module, which is specifically used for: S1. Input the compressed measurement data sample and ideal sensing matrix sample corresponding to the original hyperspectral image sample, as well as the previous hyperspectral reconstructed image sample and previous degradation deviation matrix sample corresponding to the previous stage into the degradation deviation learning module to obtain the current degradation deviation matrix sample corresponding to the current stage. S2. Based on the ideal sensing matrix sample and the current degradation deviation matrix sample, determine the current actual sensing matrix sample after compensation in the current stage; S3. Input the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample; S4. Input the previous encoder feature sample from the previous stage and the current multi-scale feature sample into the cross-stage information interaction module to obtain the reconstructed fusion feature sample. S5. Input the reconstructed fusion feature sample into the temporal fusion reconstruction module to obtain the current reconstructed image sample of the current stage, and determine the current reconstructed image sample as the new previous hyperspectral reconstructed image sample; S6. Repeat steps S1 to S5 for iterative training until the iteration stops, at which point the trained image reconstruction model is obtained.

[0096] Optionally, the temporal spectrum fusion reconstruction module includes a semi-quadratic splitting module and a spatiotemporal prior learning module.

[0097] Optionally, this training module is specifically used for: The previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample are input into the semi-quadratic splitting module to obtain the auxiliary variable corresponding to the current actual sensing matrix sample; The auxiliary variables and the spatiotemporal prior learning module are input to obtain the current multi-scale feature samples.

[0098] Optionally, the spatiotemporal prior learning module includes an encoder and a decoder; the current multi-scale feature sample includes the encoder input features of the current stage and the state of the previous decoder layer.

[0099] Optionally, this training module is specifically used for: The auxiliary variables are input into the encoder to obtain the encoder input features for the current stage; The previous encoder feature sample and the current stage encoder input feature are input into the decoder to obtain the previous decoder layer state.

[0100] Optionally, this training module is specifically used for: Based on the multi-scale attention mechanism, the attention weights corresponding to the current stage encoder input features and the previous decoder layer state in the previous encoder feature sample, the current multi-scale feature sample, and the current stage encoder input features and the previous decoder layer state are determined respectively. Based on the previous encoder feature samples, the current stage encoder input features, and the previous decoder layer state, as well as their respective attention weights, the reconstructed fusion feature samples are obtained by fusing them together.

[0101] Optionally, the decoder includes a nonlocal spectral attention branch, a channel convolution branch, and a spectral-space interaction module.

[0102] Optionally, this training module is specifically used for: The reconstructed fused feature samples are respectively input into the nonlocal spectral attention branch and the channel convolution branch in the decoder to obtain the global spectral self-attention feature output by the nonlocal spectral attention branch and the local spatial feature output by the channel convolution branch. The global spectral self-attention features and the local spatial features are input into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage.

[0103] Optionally, this training module is specifically used for: Determine the spatial attention map corresponding to the global spectral self-attention feature, and the spectral attention map corresponding to the local spatial feature; By fusing the global spectral self-attention features, the local spatial features, the spatial attention map, and the spectral attention map, the current reconstructed image sample for the current stage is obtained.

[0104] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a hyperspectral image reconstruction method. This method includes: acquiring compressed measurement data and an ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; the compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data; inputting the compressed measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstructed image output by the image reconstruction model; the image reconstruction model is obtained through iterative training based on the degradation deviation between actual sensing matrix samples and ideal sensing matrix samples.

[0105] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the hyperspectral image reconstruction method provided by the above methods. The method includes: acquiring compressed measurement data and an ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; the compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data; inputting the compressed measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstructed image output by the image reconstruction model; the image reconstruction model is obtained by iterative training based on the degradation deviation between the actual sensing matrix samples and the ideal sensing matrix samples.

[0107] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the hyperspectral image reconstruction method provided by the above methods. The method includes: acquiring compressed measurement data and an ideal sensing matrix corresponding to an original hyperspectral image to be reconstructed; the compressed measurement data being a two-dimensional image of the original hyperspectral image; the ideal sensing matrix being used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data; inputting the compressed measurement data and the ideal sensing matrix into an image reconstruction model to obtain a hyperspectral reconstructed image output by the image reconstruction model; the image reconstruction model being obtained through iterative training based on the degradation deviation between actual sensing matrix samples and ideal sensing matrix samples.

[0108] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hyperspectral image reconstruction method, characterized in that, include: Obtain compressed measurement data and ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; The compressed measurement data is a two-dimensional image of the original hyperspectral image; The ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data; The compressed measurement data and the ideal sensing matrix are input into the image reconstruction model to obtain the hyperspectral reconstructed image output by the image reconstruction model. The image reconstruction model is obtained through iterative training based on the degradation deviation between the actual sensor matrix samples and the ideal sensor matrix samples.

2. The hyperspectral image reconstruction method according to claim 1, characterized in that, The image reconstruction model includes a degradation bias learning module, a temporal-spectral fusion reconstruction module, and a cross-stage information interaction module. The image reconstruction model is trained based on the following steps: S1. Input the compressed measurement data sample and ideal sensing matrix sample corresponding to the original hyperspectral image sample, as well as the previous hyperspectral reconstructed image sample and previous degradation deviation matrix sample corresponding to the previous stage into the degradation deviation learning module to obtain the current degradation deviation matrix sample corresponding to the current stage. S2. Based on the ideal sensing matrix sample and the current degradation deviation matrix sample, determine the current actual sensing matrix sample after compensation in the current stage; S3. Input the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample; S4. Input the previous encoder feature sample from the previous stage and the current multi-scale feature sample into the cross-stage information interaction module to obtain the reconstructed fusion feature sample. S5. Input the reconstructed fusion feature sample into the temporal fusion reconstruction module to obtain the current reconstructed image sample of the current stage, and determine the current reconstructed image sample as the new previous hyperspectral reconstructed image sample; S6. Repeat steps S1 to S5 for iterative training until the iteration stops, at which point the trained image reconstruction model is obtained.

3. The hyperspectral image reconstruction method according to claim 2, characterized in that, The temporal spectrum fusion and reconstruction module includes a semi-quadratic splitting module and a spatiotemporal prior learning module; The step of inputting the previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample into the time-spectral fusion reconstruction module to obtain the current multi-scale feature sample includes: The previous hyperspectral reconstructed image sample, the compressed measurement data sample, and the current actual sensing matrix sample are input into the semi-quadratic splitting module to obtain the auxiliary variable corresponding to the current actual sensing matrix sample; The auxiliary variables and the spatiotemporal prior learning module are input to obtain the current multi-scale feature samples.

4. The hyperspectral image reconstruction method according to claim 3, characterized in that, The spatiotemporal prior learning module includes an encoder and a decoder; the current multi-scale feature sample includes the encoder input features of the current stage and the state of the previous decoder layer. The step of inputting the auxiliary variables into the spatiotemporal prior learning module to obtain the current multi-scale feature samples includes: The auxiliary variables are input into the encoder to obtain the encoder input features for the current stage; The previous encoder feature sample and the current stage encoder input feature are input into the decoder to obtain the previous decoder layer state.

5. The hyperspectral image reconstruction method according to claim 2, characterized in that, The step of inputting the previous encoder feature samples from the previous stage and the current multi-scale feature samples into the cross-stage information interaction module to obtain reconstructed fused feature samples includes: Based on the multi-scale attention mechanism, the attention weights corresponding to the current stage encoder input features and the previous decoder layer state in the previous encoder feature sample, the current multi-scale feature sample, and the current stage encoder input features and the previous decoder layer state are determined respectively. Based on the previous encoder feature samples, the current stage encoder input features, and the previous decoder layer state, as well as their respective attention weights, the reconstructed fusion feature samples are obtained by fusing them together.

6. The hyperspectral image reconstruction method according to claim 4, characterized in that, The decoder includes a nonlocal spectral attention branch, a channel convolution branch, and a spectral-space interaction module; The step of inputting the reconstructed fusion feature samples into the temporal fusion reconstruction module to obtain the current reconstructed image samples at the current stage includes: The reconstructed fused feature samples are respectively input into the nonlocal spectral attention branch and the channel convolution branch in the decoder to obtain the global spectral self-attention feature output by the nonlocal spectral attention branch and the local spatial feature output by the channel convolution branch. The global spectral self-attention features and the local spatial features are input into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage.

7. The hyperspectral image reconstruction method according to claim 6, characterized in that, The step of inputting the global spectral self-attention features and the local spatial features into the spectral-space interaction module to obtain the current reconstructed image sample at the current stage includes: Determine the spatial attention map corresponding to the global spectral self-attention feature, and the spectral attention map corresponding to the local spatial feature; By fusing the global spectral self-attention features, the local spatial features, the spatial attention map, and the spectral attention map, the current reconstructed image sample for the current stage is obtained.

8. A hyperspectral image reconstruction apparatus, characterized in that, include: The acquisition module is used to acquire compressed measurement data and ideal sensing matrix corresponding to the original hyperspectral image to be reconstructed; The compressed measurement data is a two-dimensional image of the original hyperspectral image; the ideal sensing matrix is ​​used to characterize the ideal mapping relationship between the original hyperspectral image and the compressed measurement data; The reconstruction module is used to input the compressed measurement data and the ideal sensing matrix into the image reconstruction model to obtain the hyperspectral reconstructed image output by the image reconstruction model. The image reconstruction model is obtained through iterative training based on the degradation deviation between the actual sensor matrix samples and the ideal sensor matrix samples.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the hyperspectral image reconstruction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hyperspectral image reconstruction method as described in any one of claims 1 to 7.

Images