A spectral reconstruction method based on double-branch heterogeneous feature fusion
By constructing a local-global feature fusion deep neural network model and a composite loss function, the problems of complexity and slow reconstruction speed of existing spectral imaging technologies are solved, achieving efficient, fast, and robust spectral reconstruction results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-05
- Publication Date
- 2026-04-07
AI Technical Summary
Existing spectral imaging technologies suffer from system complexity, high cost, and long imaging time. Furthermore, deep learning models ignore the potential correlation between measurement signals when processing block-compressed measurements, leading to inconsistent reconstruction results or decreased robustness.
A spectral reconstruction method based on dual-branch heterogeneous feature fusion is designed. By constructing a local-global feature fusion deep neural network model, including a local feature extraction module, a global context extraction module, a feature fusion module and a decoder, and designing a composite loss function, the model is trained end-to-end to achieve rapid spectral reconstruction.
It significantly improves the fidelity of reconstruction results, suppresses artifacts and noise, has real-time processing capabilities, enhances the robustness and generalization ability of the model, and achieves efficient and fast spectral reconstruction.
Smart Images
Figure CN121074274B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computational imaging and artificial intelligence, and in particular to a spectral reconstruction method based on the fusion of bi-branch heterogeneous features. Background Technology
[0002] Spectral imaging technology can simultaneously acquire two-dimensional spatial information and one-dimensional spectral information of a target, forming a three-dimensional data cube, making it a powerful tool for fine identification and analysis of matter. However, traditional scanning hyperspectral imagers are limited by their imaging principles, resulting in problems such as system complexity, high cost, and long imaging time, making it difficult to capture dynamic scenes.
[0003] To address this issue, snapshot-based computational spectral imaging technology based on compressed sensing has emerged. This technology, by designing a specific optical encoding front end, compresses three-dimensional hyperspectral data into a single two-dimensional mosaic measurement image in a single snapshot, significantly improving imaging speed. Its core task is to accurately solve an underdetermined linear inverse problem from this two-dimensional compressed measurement value through computational reconstruction algorithms to recover the original hyperspectral data.
[0004] Existing reconstruction algorithms are mainly divided into two categories:
[0005] 1. Traditional algorithms based on iterative optimization: These methods utilize prior knowledge such as the sparsity and low rank of hyperspectral images to construct an optimization objective function with a regularization term, and then solve it through iterative algorithms. Although these algorithms have strong theoretical interpretability, they are usually computationally intensive, with the reconstruction process taking minutes or even hours. Furthermore, the selection of regularization parameters heavily relies on experience, making it difficult to meet the needs of real-time applications.
[0006] 2. Deep Learning-Based Algorithms: In recent years, deep neural networks (DNNs) have been widely used to establish end-to-end mappings from compressed measurements to raw hyperspectral images due to their powerful nonlinear fitting capabilities. These methods implicitly incorporate the reconstruction process into the network's learning, resulting in extremely fast reconstruction speeds once training is complete. However, most existing models tend to process the measurement signals at each spatial location independently when handling block-based compressed measurements, ignoring the potential correlations between measurement signals from different spatial locations. For example, global factors such as lighting conditions and atmospheric conditions across the entire scene uniformly affect all measurements. Processing each measurement signal in isolation leads to a lack of awareness of the overall scene information, potentially introducing inconsistent artifacts in the reconstruction results or reducing robustness in low signal-to-noise ratio situations.
[0007] Therefore, designing a novel deep learning network architecture that can efficiently reconstruct images while explicitly modeling and utilizing global contextual information from measurement data to constrain and guide local spectral reconstruction is a key challenge for improving the fidelity and robustness of hyperspectral image reconstruction. In summary, designing a spectral reconstruction method based on dual-branch heterogeneous feature fusion is essential. Summary of the Invention
[0008] To overcome the shortcomings of the prior art, the purpose of this invention is to provide a spectral reconstruction method based on the fusion of dual-branch heterogeneous features.
[0009] To achieve the above objectives, the present invention provides the following solution:
[0010] This invention provides a spectral reconstruction method based on the fusion of dual-branch heterogeneous features, comprising:
[0011] Step 1: The snapshot compressed spectral imaging system performs optical sampling on the target scene to obtain a two-dimensional compressed measurement image vector;
[0012] Step 2: Construct a local-global feature fusion deep neural network model, which includes a local feature extraction module, a global context extraction module, a feature fusion module, and a decoder;
[0013] Step 3: Design a composite loss function, which includes reconstruction loss, physical prior loss, and feature regularization loss;
[0014] Step 4: Perform end-to-end training on the local-global feature fusion deep neural network model;
[0015] Step 5: Deploy the trained model to achieve rapid spectral reconstruction.
[0016] Preferably, in step 2, the local feature extraction module is a shared encoder, which processes each independent two-dimensional compressed measurement image vector based on the shared encoder to extract the local spectral features at the corresponding location.
[0017] Preferably, in step 2, the global context extraction module is a global encoder, which is used to receive a set of all two-dimensional compressed measurement image vectors, use them as input, and extract global context feature vectors.
[0018] Preferably, in step 2, the feature fusion module splices and fuses the local spectral features and the global context feature vector, wherein the global context feature vector is expanded to the same number as the local spectral features through a broadcast mechanism.
[0019] Preferably, in step 2, the decoder uses one or more subsequent layers to decode the fused features and outputs a reconstructed hyperspectral curve.
[0020] Preferably, in step 3, the reconstruction loss includes L1 loss, L2 loss and spectral angle matching loss, which are used to calculate the difference between the reconstructed spectrum output by the decoder and the true spectrum.
[0021] Preferably, in step 3, the physical prior loss includes spectral smoothness loss and non-negativity loss. The spectral smoothness loss is used to penalize the first or second derivative of the reconstructed spectrum, and the non-negativity loss is used to penalize negative values appearing in the reconstructed spectrum.
[0022] Preferably, in step 4, the local-global feature fusion deep neural network model is trained end-to-end, specifically as follows:
[0023] Construct a training dataset, which consists of multiple sets of measurement vectors – data pairs of real spectra;
[0024] Based on the training dataset, the local-global feature fusion deep neural network model is optimized end-to-end using the backpropagation algorithm to minimize the composite loss function until the model converges.
[0025] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0026] This invention provides a spectral reconstruction method based on dual-branch heterogeneous feature fusion. The method includes optical sampling of a target scene using a snapshot-type compressed spectral imaging system to obtain a two-dimensional compressed measurement image vector; constructing a local-global feature fusion deep neural network model, which includes a local feature extraction module, a global context extraction module, a feature fusion module, and a decoder; designing a composite loss function, including reconstruction loss, physical prior loss, and feature regularization loss; performing end-to-end training on the local-global feature fusion deep neural network model; and deploying the trained model to achieve rapid spectral reconstruction. This invention has the following significant advantages:
[0027] 1. High fidelity and high precision: By extracting and fusing local detail features and global context features in parallel, this invention can use the overall information of the scene (such as illumination consistency) to correct and constrain local spectral recovery. This complementarity and verification of dual-path information significantly improves the fidelity of the reconstruction results and effectively suppresses artifacts and noise.
[0028] 2. High efficiency: Compared with traditional iterative algorithms, this invention uses deep neural networks for reconstruction. The inference process only requires one forward calculation, and the reconstruction speed is extremely fast (millisecond level), with the potential for real-time processing.
[0029] 3. Strong robustness and generalization ability: This invention explicitly guides the network to learn decoupled and more essential signal representations through regularization methods such as feature orthogonal loss, rather than merely fitting the data appearance. This enhances the model's resistance to input noise and its generalization ability to unseen scenarios.
[0030] 4. End-to-end adaptive learning: The entire model is end-to-end differentiable. It automatically learns the optimal mapping relationship from the measured value to the spectrum through the gradient descent algorithm, avoiding the complex model design and tedious manual parameter adjustment process in traditional methods. Attached Figure Description
[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0032] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0033] Figure 2 This is a schematic diagram of the structure of a deep neural network model for local-global feature fusion.
[0034] Figure 3 This is a schematic diagram of the composite loss function structure;
[0035] Figure 4 This is a schematic diagram of the model training and inference process. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] The purpose of this invention is to provide a spectral reconstruction method based on dual-branch heterogeneous feature fusion, which combines the fast reasoning capability of deep neural networks with a novel parallel feature extraction and fusion architecture. Through a carefully designed network model and a composite loss function, local and global information is effectively integrated and constrained in the feature space, thereby achieving fast and high-precision reconstruction of hyperspectral images.
[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0039] like Figure 1 As shown, this invention provides a spectral reconstruction method based on the fusion of dual-branch heterogeneous features, comprising:
[0040] Step 1: Optical sampling of the target scene is performed based on a snapshot-type compressed spectral imaging system to obtain a two-dimensional compressed measurement image vector;
[0041] Step 2: Construct a local-global feature fusion deep neural network model, which includes a local feature extraction module, a global context extraction module, a feature fusion module, and a decoder;
[0042] Step 3: Design a composite loss function, which includes reconstruction loss, physical prior loss, and feature regularization loss;
[0043] Step 4: Perform end-to-end training on the local-global feature fusion deep neural network model;
[0044] Step 5: Deploy the trained model to achieve rapid spectral reconstruction.
[0045] In step 1, the snapshot compressed spectral imaging system optically samples the target scene to obtain a two-dimensional compressed measurement image vector, specifically:
[0046] A snapshot-type compressed spectral imaging system is used to optically sample the target scene and obtain a frame of two-dimensional compressed measurement image. This image can be regarded as being tiled by multiple spatially independent, smaller measurement sub-blocks (e.g., multiple 4x4 pixel blocks). These sub-blocks are then organized into a batch of independent measurement vectors.
[0047] In step 2, the local feature extraction module is a shared encoder. Based on the shared encoder, each independent two-dimensional compressed measurement image vector is processed to extract the local spectral features at the corresponding location. This path focuses on learning the direct mapping relationship from a single measurement to its corresponding spectrum.
[0048] In step 2, the global context extraction module is a global encoder, which is used to receive the set of all two-dimensional compressed measurement image vectors, use them as input, and extract a shared global context feature vector that can characterize the entire measurement scene.
[0049] In step 2, the feature fusion module concatenates and fuses the local spectral features and the global context feature vector, wherein the global context feature vector is expanded to the same number as the local spectral features through a broadcast mechanism.
[0050] In step 2, the decoder uses one or more subsequent layers (such as fully connected layers and one-dimensional convolutional layers) to decode the fused features and output the reconstructed hyperspectral curve.
[0051] In step 3, the reconstruction loss includes L1 loss (mean absolute error), L2 loss (mean square error) and spectral angle matching (SAM) loss to ensure that the reconstructed spectrum is consistent with the real spectrum in terms of value and shape.
[0052] In step 3, the physical prior loss includes spectral smoothness loss and non-negativity loss. The spectral smoothness loss is used to penalize the first or second derivative of the reconstructed spectrum to ensure the physical smoothness of the spectral curve. The non-negativity loss is used to penalize negative values that appear in the reconstructed spectrum, because the spectral intensity in the physical world should not be negative.
[0053] In step 3, the feature regularization loss calculates the orthogonality loss between local spectral features and global context features. This loss encourages the two paths to learn complementary and non-redundant information, making the model more efficient and robust.
[0054] In step 4, the local-global feature fusion deep neural network model is trained end-to-end, specifically as follows:
[0055] Construct a training dataset, which consists of multiple sets of measurement vectors – data pairs of real spectra;
[0056] Based on the training dataset, the local-global feature fusion deep neural network model is optimized end-to-end using the backpropagation algorithm to minimize the composite loss function until the model converges.
[0057] In step 5, the trained model is deployed to achieve rapid spectral reconstruction, specifically as follows:
[0058] Once the trained model is deployed, for new compressed measurements, only one forward propagation through the model is needed to directly obtain the final hyperspectral reconstruction result, achieving efficient reconstruction.
[0059] This invention provides an embodiment to elaborate on the above method, specifically as follows:
[0060] Step 1: The snapshot-type compressed spectral imaging system optically samples the target scene to obtain a two-dimensional compressed measurement image vector, specifically:
[0061] The physical process of acquiring and preprocessing compressed measurements in a snapshot-type compressed spectral imaging system can be abstracted into a linear model:
[0062] (1)
[0063] in, It is the vectorized raw spectral signal. It is a measured value ( ), Here, n is the sensing matrix and n is the system noise. In this embodiment, the input is a large two-dimensional image composed of multiple independent measurement blocks. Through an unfold operation, all independent measurement vectors can be efficiently extracted from the input image at once, forming a three-dimensional tensor containing the batch size, the number of measurement vectors, and the dimension of the measurement vectors.
[0064] Step 2: Construct a local-global feature fusion deep neural network model, specifically as follows:
[0065] like Figure 2 As shown, this embodiment constructs a deep learning model based on a multilayer perceptron (MLP), and the core architecture of this model is as follows:
[0066] 1. Local Spectral Encoder: This is a shared encoding module that performs an independent nonlinear transformation on each measurement vector, mapping it to high-dimensional local spectral features, focusing on mining the intrinsic correlation between a single measurement signal and its corresponding spectrum.
[0067] 2. Global Context Encoder: This module integrates all measurement vectors within a batch (e.g., flattening and concatenating them) and takes them as a single input. Through a deep non-linear network, it learns and outputs a global feature vector that represents the overall contextual information of the current batch of data.
[0068] 3. Feature Fusion Module: The global context feature vector is copied and expanded to correspond one-to-one with the local spectral features. Then, each set of local spectral features is concatenated with its corresponding global context feature along the feature dimension. The concatenated fused features are fed into a fusion network module for deep interaction and information extraction.
[0069] 4. Decoder: Receives the fused and refined features, and through a decoding module containing a fully connected layer, a non-negative activation function (such as Softplus), and a one-dimensional convolutional smoothing layer, finally generates the reconstructed spectral curve and reshapes it into a data block with a spatial-spectral structure.
[0070] Step 3: Design a composite loss function, which includes reconstruction loss, physical prior loss, and feature regularization loss, specifically:
[0071] like Figure 3 As shown, the composite loss function used in this embodiment consists of the following weighted summation:
[0072] (2)
[0073] in, , , and As weight, For spectral smoothness loss, reconstruction loss It is itself a composite item:
[0074] (3)
[0075] It ensures the reconstruction of the spectrum. With the true spectrum Both the numerical values (L1, L2) and the angles (SAM) should be as close as possible.
[0076] Spectral smoothness loss By penalizing the difference norm of the spectral curve, it encourages the generation of smooth, physically consistent spectra.
[0077] Non-negative loss Penalize negative terms in the predicted values, for example... .
[0078] Feature Orthogonal Loss for:
[0079] (4)
[0080] This calculation yields normalized local features. and global features The absolute value of the cosine similarity between the two features is minimized, which forces the two feature extraction paths to learn complementary and orthogonal information, thus improving the model's representational power.
[0081] Step 4: Perform end-to-end training on the local-global feature fusion deep neural network model, specifically as follows:
[0082] like Figure 4 As shown, during the training phase, real spectral data blocks $x$ are sampled from a large hyperspectral dataset. These are then processed using a known sensing matrix. The noise model generates simulated measurements. .Will As training pairs, input them into the network and calculate the total loss. Then, the network weights are updated using a gradient descent optimizer. A global normalization strategy can be employed during training to ensure the uniformity of data scale and physical meaning.
[0083] Step 5: Deploy the trained model to achieve rapid spectral reconstruction, specifically as follows:
[0084] The trained model is deployed by feeding new, real-world measurement images in blocks into the trained network, and the decoder output is directly obtained as the final reconstruction result. The entire process is extremely fast and requires no iteration. By multiplying by the global normalization coefficients saved during training, the inversion from normalized output to absolute physical strength can be achieved.
[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0086] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A spectral reconstruction method based on the fusion of dual-branch heterogeneous features, characterized in that, include: Step 1: The snapshot compressed spectral imaging system performs optical sampling on the target scene to obtain a two-dimensional compressed measurement image vector; Step 2: Construct a local-global feature fusion deep neural network model, which includes a local feature extraction module, a global context extraction module, a feature fusion module, and a decoder; Step 3: Design a composite loss function, which includes reconstruction loss, physical prior loss, and feature regularization loss. The reconstruction loss includes L1 loss, L2 loss, and spectral angle matching loss, used to calculate the difference between the reconstructed spectrum output by the decoder and the true spectrum. The physical prior loss includes spectral smoothness loss and non-negativity loss. The spectral smoothness loss is used to penalize the first or second derivative of the reconstructed spectrum, and the non-negativity loss is used to penalize negative values appearing in the reconstructed spectrum. The feature regularization loss calculates the orthogonality loss between local spectral features and global context features, encouraging the two paths to learn complementary and non-redundant information. The composite loss function consists of the weighted sum of the following parts: （2） in, , , and As weight, For spectral smoothness loss, reconstruction loss It is itself a composite item: （3） Ensured the reconstruction of the spectrum With the true spectrum The values (L1, L2) and angles (SAM) are close; Spectral smoothness loss By penalizing the difference norm of the spectral curve, it encourages the generation of smooth, physically consistent spectra. Non-negative loss Penalize negative values in the predicted values; Feature regularization loss for: （4） In the formula, local features are: Global features: ; Step 4: Perform end-to-end training on the local-global feature fusion deep neural network model; Step 5: Deploy the trained model to achieve rapid spectral reconstruction.
2. The method according to claim 1, characterized in that, In step 2, the local feature extraction module is a shared encoder. Based on the shared encoder, each independent two-dimensional compressed measurement image vector is processed to extract the local spectral features at the corresponding location.
3. The method according to claim 2, characterized in that, In step 2, the global context extraction module is a global encoder, which is used to receive the set of all two-dimensional compressed measurement image vectors, use them as input, and extract the global context feature vector.
4. The method according to claim 3, characterized in that, In step 2, the feature fusion module concatenates and fuses the local spectral features and the global context feature vector, wherein the global context feature vector is expanded to the same number as the local spectral features through a broadcast mechanism.
5. The method according to claim 4, characterized in that, In step 2, the decoder uses one or more subsequent layers to decode the fused features and outputs the reconstructed hyperspectral curve.
6. The method according to claim 1, characterized in that, In step 4, the local-global feature fusion deep neural network model is trained end-to-end, specifically as follows: Construct a training dataset, which consists of multiple sets of measurement vectors – data pairs of real spectra; Based on the training dataset, the local-global feature fusion deep neural network model is optimized end-to-end using the backpropagation algorithm to minimize the composite loss function until the model converges.