Snapshot spectral imaging system and method based on spatial spectrum hybrid coding

Abstract

The invention provides a snapshot type spectral imaging system and method based on spatial spectrum hybrid coding. The system comprises a spatial spectrum hybrid coding imaging module and a decoding reconstruction module. Wherein the spatial spectrum hybrid coding imaging module sequentially comprises a filtering module used for filtering wave bands outside a working wave band range; the spectrum hybrid modulation module is used for applying composite modulation to an incident light field through multiplexing to obtain a hybrid modulation light field; the spectral filtering array module is used for performing pixel-level spectral filtering on the hybrid modulation light field in a spatial dimension; and the gray level acquisition module is used for capturing the mixed light field coded by the spectrum filtering module to obtain a two-dimensional measurement value. And the decoding reconstruction module is used for reconstructing a spectral image from the two-dimensional measurement image. According to the invention, the spatial and spectral information is synchronously obtained in single exposure, the joint optimization of the spatial resolution and the spectral resolution is realized, and a solution is provided for the problem of balancing the spatial resolution and the spectral resolution of a snapshot spectral imaging system.

Description

Technical Field

[0001] This invention belongs to the field of spectral imaging technology, and particularly relates to a snapshot spectral imaging system and method based on spatial spectral hybrid coding. Background Technology

[0002] Hyperspectral images consist of multiple continuous spectral bands, providing a nearly continuous spectral curve for each spatial pixel. This results in a far superior descriptive capability compared to three-channel RGB images in revealing the material composition and internal state of the measured object. This characteristic makes them irreplaceable in fields such as precision agriculture, environmental remote sensing, biomedical diagnostics, and industrial inspection.

[0003] Traditional hyperspectral imaging techniques are mostly based on scanning principles, requiring point-by-point or line-by-line scanning in spatial or spectral dimensions to construct a data cube. This is not only complex and costly, but also unable to capture dynamic scenes due to the need for multiple exposures. To overcome this bottleneck, inspired by compressed sensing theory, snapshot compressed spectral imaging technology has emerged. This type of system aims to compress and record three-dimensional hyperspectral information onto a two-dimensional detector within a single exposure using an optical encoding module, and then reconstruct the original data cube using a spectral reconstruction algorithm. Snapshot compressed spectral imaging technology significantly reduces the hardware cost and complexity of the system while maintaining high temporal resolution and high data throughput.

[0004] Representative systems include coded aperture spectral imaging systems (CASSI, including monochromatic dispersion systems SD-CASSI and dual dispersion systems DD-CASSI) and imaging systems based on spectral filter arrays (such as Fabry-Perot filters and liquid crystal tunable filters). These systems can be broadly categorized into two types: the first type is dispersive systems, which aim to achieve high-precision spectral sampling using dispersive elements such as gratings and prisms. However, the dispersion process inevitably introduces spatial aliasing, leading to blurring or artifacts in the spectral image. The second type is spatial filtering systems, which achieve pixel-level spectral filtering by integrating filter arrays at the sensor front end, thus preserving complete spatial resolution. However, their spectral resolution is highly dependent on the non-correlation of the spectral response curves of each filter unit. Due to limitations in coding strategies, existing systems struggle to simultaneously achieve efficient sampling capabilities for both spatial details and continuous spectra. Summary of the Invention

[0005] Purpose of the invention: The technical problem to be solved by the present invention is that the coding strategies of existing snapshot spectral imaging systems are difficult to balance spatial and spectral resolution, and a snapshot spectral imaging system and method based on spatial spectral hybrid coding is provided.

[0006] The system includes a spatial spectral hybrid coding imaging module and a decoding and reconstruction module;

[0007] The spatial spectral hybrid coding imaging module includes a filtering module, a spectral hybrid modulation module, a spectral filtering array module, and a grayscale acquisition module, which are connected in sequence.

[0008] The filtering module is used to filter out bands outside the operating band range;

[0009] The spectral mixing modulation module is used to apply composite modulation to the incident light field through multiplexing to obtain a mixed modulation light field; the composite modulation physically constructs two parallel and incoherently superimposed modulation channels for the incident light: direct transmission and dispersion coding, to generate a modulation light field with a two-dimensional spatial and spectral coupling relationship.

[0010] The spectral filtering array module and the spectral mixing modulation module are configured in concert to perform further pixel-level spectral filtering and encoding on the mixed modulation light field with the spatial and spectral two-dimensional coupling relationship in the spatial dimension, thereby working together to achieve spatial spectral mixing encoding.

[0011] The grayscale acquisition module is used to capture the mixed light field after it has been processed by the spectral filtering array module, and obtain a single two-dimensional compressed measurement value.

[0012] The decoding and reconstruction module includes a spectral reconstruction algorithm; the spectral reconstruction algorithm reconstructs a spectral image from two-dimensional compressed measurements based on the modulation characteristics of the spectral mixing modulation module and the coding characteristics of the spectral filtering array module.

[0013] The spectral mixing modulation module uses polarization multiplexing as its multiplexing method.

[0014] The spectral mixing modulation module is implemented using a metasurface device, which is configured to apply different phase modulations to two orthogonal polarization components in the incident light.

[0015] The multiplexing methods of the spectral mixing modulation module also include spatial multiplexing and time-division multiplexing.

[0016] The spectral filtering array module adopts the form of a broadband coding array, which includes two or more spatial coding units. Each spatial coding unit has a different filtering response function that covers the working band.

[0017] The present invention also provides a snapshot-type spectral imaging method based on spatial spectral hybrid coding implemented based on the aforementioned system, comprising the following steps:

[0018] S1, the scene light passes through the filtering module to filter out bands outside the working band range;

[0019] S2, the spectral mixing modulation module applies composite modulation to the filtered light field through multiplexing to obtain a mixed modulated light field;

[0020] S3, the spectral filtering array module performs further pixel-level filtering and encoding on the hybrid modulated light field in the spatial dimension;

[0021] S4, the grayscale acquisition module uses a grayscale camera to capture the encoded mixed light field to obtain a two-dimensional snapshot compressed grayscale image based on spatial spectral mixed coding;

[0022] S5 uses a two-dimensional snapshot to compress grayscale images and then reconstructs hyperspectral data using a spectral reconstruction algorithm.

[0023] In step S2, the spectral mixing modulation module employs a composite coding strategy to apply a composite modulation effect to the incident light field, including a direct transmission channel and a deterministic dispersive coding channel. This composite modulation causes the output light field of the spectral mixing modulation module to diffract over a distance d, forming a composite point spread function on the subsequent imaging plane. ,in Spatial position of the sensor plane; point spread function Spatially, this is represented by the superposition of the intensities of a central main lobe and a dispersion spot, wherein the position of the dispersion spot shifts linearly with wavelength, and the shift amount is... It directly characterizes the dispersive properties of the system.

[0024] In step S3, the spectral filtering array module spatially encodes the mixed modulation light field of the incident spectral filtering array module using spectral filtering. Each spatial encoding unit has a pre-calibrated, distinct spectral filtering response. .

[0025] In step S4, the grayscale acquisition module uses a grayscale image sensor to acquire the mixed light field encoded by the spectral filtering array module, and uses... This represents the true hyperspectral data at different spatial locations in the scene, resulting in a two-dimensional compressed measurement. :

[0026] .

[0027] In step S5, the spectral reconstruction algorithm is based on the point spread function. Spatial dispersion characteristics and spectral filtering response The spectral coding characteristics of the two-dimensional compressed measurement values The spatial and spectral information in the data are decoupled, and the hyperspectral data is reconstructed based on the decoupled information.

[0028] The present invention has the following beneficial effects: (1) The core of the present invention lies in proposing a spatial-spectral hybrid coding method. Unlike traditional dispersive systems that sacrifice spatial resolution for spectral accuracy or spatial filtering systems that sacrifice spectral continuity for spatial integrity, the present invention achieves efficient sampling of spatial and spectral dimensions in a single exposure by synergistically utilizing two modulation paths: direct transmission and dispersive coding. Specifically, the present invention preserves complete spatial information through the non-dispersive part and improves the spectral resolution of the system through the dispersive part. This hybrid coding mechanism effectively balances spatial detail and spectral resolution from the information source, laying a physical foundation for high-quality spectral reconstruction.

[0029] (2) This invention offers high flexibility and scalability in hardware implementation. Specifically, this is reflected in two core modules: First, the spectral mixing modulation module can be implemented through various multiplexing mechanisms such as spatial division multiplexing, polarization multiplexing, and time division multiplexing, providing different performance and complexity trade-offs. Second, the spectral filtering array module also has multiple implementation methods, including Bayer filter arrays, liquid crystal tunable filters, metasurface devices, and Fabry-Perot filters. The entire system can either employ a dual-optical-path scheme with beam splitters combined with a mature color Bayer filter array to achieve extremely low manufacturing costs; or it can use cutting-edge nano-optical devices such as metasurfaces to implement the two core modules, further integrating the entire system to reduce the additional instability caused by the high complexity of the imaging system. This invention provides multiple solutions for cost and performance trade-offs in the hardware implementation of the imaging system.

[0030] (3) The imaging system constructed in this invention has good versatility at the algorithm level, and can be compatible with classic algorithms based on compressed sensing and various data-driven deep learning models for spectral reconstruction. On this basis, in order to further optimize the reconstruction quality for mixed coded data, this invention also specifically designs an optional plug-and-play dual-path reconstruction network framework. This framework introduces a data decoupling module to decouple mixed data from measurement values; and includes a feature extraction interaction and fusion module, which is connected to the network in a plug-and-play manner and is responsible for deep interaction and adaptive fusion of the decoupled dual-path features. It is worth noting that this fusion module can be replaced with neural network modules of different structures to adapt to different reconstruction quality and computational efficiency requirements. Attached Figure Description

[0031] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0032] Figure 1 This is a schematic diagram of the overall structure of the snapshot-type spectral imaging system based on spatial spectral hybrid coding of the present invention.

[0033] Figure 2 This is a schematic diagram of the optical path when the spectral mixing modulation module in the snapshot spectral imaging system based on spatial spectral mixing coding of the present invention uses spatial division multiplexing.

[0034] Figure 3 This is a schematic diagram of the imaging optical path of a snapshot-type spectral imaging system based on spatial spectral hybrid coding in an embodiment of the present invention.

[0035] Figure 4 This is an example diagram of the metasurface unit and spectral point diffusion function of the spectral mixing modulation module in an embodiment of the present invention.

[0036] Figure 5 This is an example diagram of the spectral response curve of the Fabry-Perot filter array used in the spectral filter array module in this embodiment of the invention.

[0037] Figure 6 This is a schematic diagram of the plug-and-play spectral reconstruction network framework in an embodiment of the present invention.

[0038] Figure 7 This is a synthesized RGB image of the reconstructed spectral image from a two-dimensional measurement map in an embodiment of the present invention. Detailed Implementation

[0039] like Figure 1 As shown, this embodiment provides a snapshot-type spectral imaging system based on spatial spectral hybrid coding, comprising a spatial spectral hybrid coding imaging module and a decoding and reconstruction module. The spatial spectral hybrid coding imaging module includes four sequentially connected modules: a filtering module, a spectral hybrid modulation module, a spectral filtering array module, and a grayscale acquisition module. The decoding and reconstruction module contains a spectral reconstruction algorithm, and its input section is connected to the spectral hybrid modulation module, the spectral filtering array module, and the grayscale acquisition module.

[0040] The following describes the use of, for example Figure 1 The system shown implements a snapshot-based spectral imaging method based on spatial spectral hybrid coding. The entire imaging optical path is as follows: Figure 3 As shown, the four modules in the spatial spectral mixing and coding imaging module are sequentially connected through objective lenses and lenses to form a complete imaging path. Specifically, this imaging method includes the following steps:

[0041] Step 1: The scene light passes through the filtering module to filter out wavelengths outside the working wavelength range. In this embodiment, a filter is used to filter the 450nm~650nm wavelength range to prevent light outside the range from interfering with the imaging quality.

[0042] Step 2: The spectral mixing modulation module applies a composite modulation effect, including a direct transmission channel and a dispersion-coded channel, to the incident light field through multiplexing, resulting in a mixed-modulated light field. This composite modulation effect can be implemented through various multiplexing methods, including spatial multiplexing, such as... Figure 2 As shown, a beam splitter is used to split the input light into two beams. One beam is not modulated, while the other beam is dispersed by a dispersive element. Finally, the two beams are merged by the beam splitter, and the image plane is merged into the same plane. Polarization multiplexing technology is used to achieve the above two different modulation effects of no modulation and dispersion processing for light with two different polarization states of 0° and 90° through polarization-sensitive elements such as metasurface materials. Alternatively, a time-division multiplexing strategy can be used to quickly switch between the above two different modulation modes in time with the help of a spatial light modulator, etc.

[0043] In this embodiment, given the modulatory capabilities of metasurface materials in the subwavelength domain, it can achieve the function of modulating light with different polarization states in different ways. The spectral mixing modulation module is implemented through polarization multiplexing using metasurface devices. Specifically, by designing the phase gradient of the metasurface array under 0° and 90° polarization states, and using the finite-difference time-domain (FDTD) method, selecting metasurface units of corresponding size and rotation angle, the emission angle of light can be controlled, thereby achieving the effect of no modulation of 0° polarized light and dispersion of 90° polarized light within the working wavelength range. Metasurface units such as... Figure 4 As shown in Figure (a), a silicon dioxide (SiO2) substrate is used, and the metasurface material is polarization-sensitive titanium dioxide (TiO2). The output light field of this module propagates through diffraction over a distance d, forming a composite point spread function on the sensor plane, denoted as . . Figure 4 Figure (b) shows an example of the spectral point spread function of this module in the operating wavelength range of 450–650 nm. To more clearly illustrate the modulation effect, Figure 4 The two modulation states are displayed in two separate rows. The first row shows the result without modulation in the 0° polarization state, and the second row shows the result of dispersive modulation in the 90° polarization state. Correspondingly, the overall... This is the superposition of the upper and lower rows, spatially represented as the superposition of the intensities of a central main lobe and a dispersion spot that shifts linearly with wavelength. The spatial position of the sensor plane, its offset This directly characterizes the system's dispersive properties. It should be noted that the implementation of this spectral mixing modulation module is not unique; strategies such as spatial division multiplexing and time division multiplexing can also be used to achieve the composite modulation effect as embodiments of this module.

[0044] Step 3: The spectral filtering array module performs pixel-level spectral filtering encoding on the mixed light field modulated by the spectral mixing modulation module in the spatial dimension. This module adopts a broadband encoding array, including multiple spatial encoding units, each with a different filtering response function covering a wide spectral range. This broadband encoding array can be implemented through various micro-nano optical or material technologies, including but not limited to: Bayer filter arrays, liquid crystal tunable filters, metacellular arrays composed of metasurface materials, Fabry-Perot resonator arrays, quantum dot spectral conversion layers, organic dye patterned coatings, and customized optical coated filter arrays.

[0045] As a preferred embodiment, in this embodiment, the spectral filtering array module can be implemented using a Fabry-Perot resonator array. For example, a coding mask scheme with spatial and spectral randomness can be disclosed in the literature Motoki Yako et al. Video-rate hyperspectral camera based on a cmoscompatible random array of fabry–perot filters. Nature Photonics, 17(3):218–223, 2023. This scheme is fabricated using standard micro-nano fabrication technology. After depositing a bottom reflective layer and a cavity layer sequentially on a substrate, various randomly distributed cavity thicknesses are formed through photolithography and etching. Subsequently, a top reflective layer is deposited and a portion of it is selectively etched to form a broadband coding array with different spectral responses. In this invention, the key point is that, in order to achieve efficient spatial spectral mixing coding, the spectral filtering array is configured in conjunction with the aforementioned spectral mixing modulation module proposed in this invention: the output image plane of the spectral mixing modulation module and the input plane of the spectral filtering array are image-matched through a relay lens to ensure that the modulated light field can be completely received by the corresponding filtering unit, thereby working together to achieve spatial spectral mixing coding. The array comprises multiple spatial coding units, each with a distinct spectral filter response, denoted as […]. ,in The spatial position corresponding to the sensor plane. Figure 5 The diagram shows some representative spectral response curves (i.e., transmittance curves) of the array. It is worth noting that the system of this invention possesses hyperspectral imaging capabilities regardless of the type of filter array used, such as Bayer filter arrays, liquid crystal tunable filters, metasurface devices, and Fabry-Perot filters. However, filter arrays with different spectral encoding capabilities can affect the system's spectral resolution and spectral reconstruction quality.

[0046] Step 4: The grayscale acquisition module uses a grayscale image sensor to acquire the mixed light field encoded by the spectral filtering array module. This represents the true hyperspectral data at different spatial locations in the scene, resulting in a two-dimensional compressed measurement. :

[0047] ,

[0048] Step 5: The spectral reconstruction module reconstructs three-dimensional hyperspectral data based on the two-dimensional measurement image obtained by the grayscale acquisition module using spatial spectral hybrid coding, and simultaneously using the point spread function of the spectral hybrid modulation module and the spectral response curve of the spectral filtering array module.

[0049] This embodiment employs a deep learning-based spectral reconstruction algorithm, and the reconstruction network structure adopts the method proposed in this invention. Figure 6 The plug-and-play network framework shown: The reconstructed network will use the calibrated point spread function. Spectral response curve and two-dimensional measurement images As input, the first two are used as prior information for encoding to establish an end-to-end mapping between the two-dimensional measurement image and the real hyperspectral data, thereby reconstructing the hyperspectral data.

[0050] First, the network is divided into two paths to decouple the mixed data by copying and inverse dispersive processing of the input two-dimensional measurement image. Then, the various feature extraction, interaction, and fusion modules perform deep interaction and adaptive fusion of the decoupled dual-path features through the spatial dispersion characteristics of the joint point spread function and the spectral coding characteristics of the spectral response curve.

[0051]

[0052] Where H1 and H2 are the dual-path features of the input module, PSF is the point spread function, T is the spectral response curve, and G is the data processing procedure of the module. Finally, the dual-path features are fused into one as the output of the reconstruction network.

[0053] To incorporate imaging model constraints into the spectral reconstruction task, this embodiment uses a loss function (Loss) consisting of two parts:

[0054] ,

[0055] Where S represents the actual hyperspectral data. To reconstruct hyperspectral data, F represents the network model, and Y represents the two-dimensional measurement image. The first term of the Loss is the loss between the reconstructed hyperspectral data and the real hyperspectral data, and the second term is the loss between the simulated measurement image obtained by passing the reconstructed hyperspectral image through the imaging process and the real measurement image. The sum of these two losses can better constrain the learning process of the reconstruction network.

[0056] In the specific reconstruction process, the feature extraction, interaction, and fusion module in the network adopts ResCNN with residual fully convolutional layers.

[0057] The reconstructed network was trained on a parallel computing device, a GeForce-RTX 4090, using the Adam optimizer with hyperparameters set to [value missing]. The training cycle is 300 rounds. The initial learning rate is fixed at 4e. -4 And adopts a cosine annealing learning rate scheduling strategy ( The reconstruction result of this embodiment is as follows: Figure 7 As shown, (a) is a two-dimensional measurement image of the scene, and (b) is an RGB image synthesized from the reconstructed spectral image. It can be seen that the reconstructed image can clearly restore the blurred texture structure on the measurement image, and the imaging system and method proposed in this invention have high imaging quality.

[0058] This invention balances the spatial and spectral resolution capabilities of snapshot spectral imaging systems and provides multiple system implementation methods to meet different needs such as low cost and high integration. It can reconstruct high-quality hyperspectral images through reconstruction algorithms, providing a solution to the spatial-spectral resolution trade-off problem of snapshot spectral imaging systems.

[0059] This invention provides a snapshot-type spectral imaging system and method based on spatial spectral hybrid coding. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A snapshot-type spectral imaging system based on spatial spectral hybrid coding, characterized in that, It includes a spatial spectral hybrid coding imaging module and a decoding and reconstruction module; The spatial spectral hybrid coding imaging module includes a filtering module, a spectral hybrid modulation module, a spectral filtering array module, and a grayscale acquisition module, which are connected in sequence. The filtering module is used to filter out bands outside the operating band range; The spectral mixing modulation module is used to apply composite modulation to the incident light field through multiplexing to obtain a mixed modulation light field; the composite modulation physically constructs two parallel and incoherently superimposed modulation channels for the incident light: direct transmission and dispersion coding, to generate a modulation light field with a two-dimensional spatial and spectral coupling relationship. The spectral filtering array module and the spectral mixing modulation module are configured in concert to perform further pixel-level spectral filtering and encoding on the mixed modulation light field with the spatial and spectral two-dimensional coupling relationship in the spatial dimension, thereby working together to achieve spatial spectral mixing encoding. The grayscale acquisition module is used to capture the mixed light field after it has been processed by the spectral filtering array module, and obtain a single two-dimensional compressed measurement value. The decoding and reconstruction module includes a spectral reconstruction algorithm; the spectral reconstruction algorithm reconstructs a spectral image from two-dimensional compressed measurements based on the modulation characteristics of the spectral mixing modulation module and the coding characteristics of the spectral filtering array module.

2. The system according to claim 1, characterized in that, The spectral mixing modulation module uses polarization multiplexing as its multiplexing method.

3. The system according to claim 2, characterized in that, The spectral mixing modulation module is implemented using a metasurface device, which is configured to apply different phase modulations to two orthogonal polarization components in the incident light.

4. The system according to claim 3, characterized in that, The multiplexing methods of the spectral mixing modulation module also include spatial multiplexing and time-division multiplexing.

5. The system according to claim 4, characterized in that, The spectral filtering array module adopts the form of a broadband coding array, which includes two or more spatial coding units. Each spatial coding unit has a different filtering response function that covers the working band.

6. A snapshot-type spectral imaging method based on spatial spectral hybrid coding implemented according to any one of claims 1 to 5, characterized in that, Includes the following steps: S1, the scene light passes through the filtering module to filter out bands outside the working band range; S2, the spectral mixing modulation module applies composite modulation to the filtered light field through multiplexing to obtain a mixed modulated light field; S3, the spectral filtering array module performs further pixel-level filtering and encoding on the hybrid modulated light field in the spatial dimension; S4, the grayscale acquisition module uses a grayscale camera to capture the encoded mixed light field to obtain a two-dimensional snapshot compressed grayscale image based on spatial spectral mixed coding; S5 uses a two-dimensional snapshot to compress grayscale images and then reconstructs hyperspectral data using a spectral reconstruction algorithm.

7. The method according to claim 6, characterized in that, In step S2, the spectral mixing modulation module employs a composite coding strategy to apply a composite modulation effect to the incident light field, including a direct transmission channel and a deterministic dispersive coding channel. This composite modulation causes the output light field of the spectral mixing modulation module to diffract over a distance d, forming a composite point spread function on the subsequent imaging plane. ,in Spatial position of the sensor plane; point spread function Spatially, this is represented by the superposition of the intensities of a central main lobe and a dispersion spot, wherein the position of the dispersion spot shifts linearly with wavelength, and the shift amount is... It directly characterizes the dispersive properties of the system.

8. The method according to claim 7, characterized in that, In step S3, the spectral filtering array module spatially encodes the mixed modulation light field of the incident spectral filtering array module using spectral filtering. Each spatial encoding unit has a pre-calibrated, distinct spectral filtering response. .

9. The method according to claim 8, characterized in that, In step S4, the grayscale acquisition module uses a grayscale image sensor to acquire the mixed light field encoded by the spectral filtering array module, and uses... This represents the true hyperspectral data at different spatial locations in the scene, resulting in a two-dimensional compressed measurement. : 。 10. The method according to claim 9, characterized in that, In step S5, the spectral reconstruction algorithm is based on the point spread function. Spatial dispersion characteristics and spectral filtering response The spectral coding characteristics of the two-dimensional compressed measurement values The spatial and spectral information in the data are decoupled, and the hyperspectral data is reconstructed based on the decoupled information.

Images