Integrated lensless snapshot spectral imaging system and method thereof

Abstract

The invention discloses an integrated lens-free snapshot spectral imaging system and a method thereof. The imaging system comprises a filtering window module, a spatial spectrum compression modulation module, a color acquisition module and a calculation reconstruction module, and the filtering window module is used for filtering light outside a window wave band; the spatial spectrum compression modulation module is used for performing different amplitude coding on scenes of different spectrum channels and projecting point light sources in the scenes to the color acquisition module in a multiplexing mode; the color acquisition module is used for carrying out spectrum coding on a scene to obtain a two-dimensional measurement value; and the calculation reconstruction module is used for reconstructing hyperspectral image data based on the two-dimensional measurement value. According to the method, the random color mask and the pixel-level periodic filtering are adopted for combined modulation, the hyperspectral image is reconstructed through calculation, the method has the remarkable characteristics of high integration, design flexibility and structural stability, and an integrated normal form is created for new-generation snapshot spectral imaging.

Description

Technical Field

[0001] The invention relates to a lens-free snapshot spectral imaging system and method, belonging to the technical field of spectral imaging. Background Art

[0002] Snapshot spectral imaging refers to capturing the three-dimensional spatial spectral information of a scene in a single exposure. It greatly improves the imaging efficiency, not only capturing the intensity information of each spatial position, but also recording the corresponding spectral information, making it possible to capture the spectral data of dynamic scenes in real time. Since the reflectance spectrum is an intrinsic property of matter, continuous and fine spectral images can reflect the complex and diverse composition and state of real-world materials. Therefore, snapshot spectral imaging has broad application potential in industrial production detection, smart agriculture, combustion dynamics analysis, environmental resource exploration and other fields.

[0003] With the emergence of emerging applications such as wearable devices, autonomous driving, robots, and virtual / augmented reality, snapshot spectral imaging systems are moving towards miniaturization. Early snapshot spectral imaging systems mostly relied on narrow and long relay mirrors and large spectroscopic devices, which made the imaging system very bulky, severely limiting its application in lightweight and portable scenarios. Among them, the more representative technologies are: Integral Field Spectrometry (IFS), Coded Aperture Snapshot Spectral Imaging (CASSI), Prism-mask Video Imaging Spectrometry, and Computed Tomography Imaging Spectrometry. These technologies are based on the principle of prism dispersion or grating diffraction spectrometry, and require a collimator group to obtain sufficient dispersion distance, which greatly increases the volume and complexity of the system.

[0004] In recent years, with the innovation of new materials on optical devices and the optimization of reconstruction algorithms, snapshot spectral imaging systems based on diffraction or pixel-level filtering principles have emerged, which can reduce the size of the imaging system to the size of lens focusing imaging. Among them, imaging systems based on the principle of diffraction optics are usually implemented using diffractive optical elements (DOE) or metasurface materials, and different coherent diffraction modes are used for computational reconstruction to obtain spectral data cubes; imaging systems based on the principle of pixel-level filtering usually use optimized filter arrays, combined with deep learning and other computational methods to perform wide-spectrum decoupling and restore spectral data cubes. However, these snapshot spectral imaging systems still have not gotten rid of the limitations of lens focusing imaging, and the system size is restricted by the back intercept of optical elements and cannot be further reduced.

[0005] Recently, the Spectral DiffuerCam has emerged, which uses a ground glass and a color filter array to configure the snapshot spectral imaging system in close proximity. However, it uses an 8×8 narrowband filter array for spectral encoding, and the device cost is high. Summary of the invention

[0006] Based on the above defects of the prior art, an object of the present invention is to provide a compact, integrated, low-cost lensless snapshot spectral imaging system and method thereof.

[0007] The technical solution adopted by the present invention is as follows:

[0008] An integrated lensless snapshot spectral imaging system comprises a filter window module, a spatial spectrum compression modulation module, a color acquisition module and a calculation and reconstruction module, wherein the filter window module, the spatial spectrum compression modulation module and the color acquisition module are connected in sequence, and the spatial spectrum compression modulation module and the color acquisition module are respectively connected to the calculation and reconstruction module; the filter window module is used to filter out light outside the window band; the spatial spectrum compression modulation module is used to perform different amplitude encoding on scenes of different spectral channels, and project point light sources in the scene to the color acquisition module in a multiplexed manner; the color acquisition module is used to perform spectral encoding on the scene to obtain a two-dimensional measurement value; the calculation and reconstruction module is used to reconstruct hyperspectral image data based on the two-dimensional measurement value.

[0009] The present invention also provides a method for utilizing the above-mentioned integrated lensless snapshot spectral imaging system, comprising the following steps:

[0010] S1, the light emitted by the scene passes through the filter window module, and the light outside the window band is filtered out;

[0011] S2, using the spatial spectrum compression modulation module to perform different amplitude encoding and diffraction projection on scenes of different spectral channels, and projecting point light sources in the scene to a color acquisition module in a multiplexed manner;

[0012] S3, the color acquisition module acquires a multispectral image to obtain a two-dimensional measurement value of spatial-spectral coupling;

[0013] S4, reconstructing the hyperspectral image data using a hyperspectral reconstruction algorithm based on the two-dimensional measurement value of the spatial-spectral coupling.

[0014] The main technologies of the present invention include the following three points:

[0015] (1) Optical devices for spatial spectrum compression modulation

[0016] Common spatial spectrum modulation and coding devices include: gratings, prisms, DOE, metasurface devices, etc. Among them, optical devices such as gratings and prisms usually require sufficient dispersion distance, which makes the system narrow, long and bulky; although optical devices such as DOE and metasurface devices have strong light field control capabilities, they require precise processing technology, which brings high costs. The color mask proposed in the present invention has a compressed spatial modulation method and rich spectral coding by cleverly combining wide-spectrum filtering with diffraction optics; at the same time, the production accuracy can be flexibly adjusted according to imaging requirements. For scenes with low imaging requirements, the color mask is produced using a process with low processing accuracy, which has extremely low production costs.

[0017] (2) Integrated spectral encoding paradigm

[0018] In the past, spectral encoding paradigms were generally divided into the following categories: coded aperture represented by CASSI, computed tomography represented by CTIS, diffraction decoupling represented by DOE spectral imaging, and wide spectrum decoupling represented by pixel-level filter array hyperspectral imaging. However, none of these paradigms can reduce the scale of snapshot spectral imaging to the sensor level. The integrated spectral encoding paradigm proposed in the present invention, in which the color mask and the multispectral image sensor are closely configured, can achieve sensor-level integration, while providing great design freedom for the design of the color mask, leaving a wide space for imagination.

[0019] (3) Filter array realizes multi-spectral channel acquisition

[0020] The projection after the spatial spectrum compression modulation module has the characteristics of highly coupled degradation, because each element in the spatial spectrum three-dimensional data undergoes multiplexed diffraction projection and is linearly superimposed in space and spectrum. Therefore, further periodic filtering and encoding are required on the sensor surface to separate the diffraction projections at different spatial positions into different spectral coding channels to provide richer spectral coding.

[0021] Therefore, the solution of the present invention has the following advantages:

[0022] (1) Compared with the existing snapshot spectral imaging scheme, the present invention adopts a color mask combined with the principle of wide-spectrum filtering and diffraction optics, which reduces the manufacturing cost of the device, has the characteristics of lightweight portability, and improves the compactness and integration of the system. At the same time, it reserves more degrees of freedom for the design of the color mask, providing a broad space for the optimization design of the imaging system.

[0023] (2) The present invention adopts a method of jointly controlling color mask diffraction projection and periodic wide-spectrum filter array, which improves the spectral sampling rate of the system, reduces the complexity of the system and the underdetermination of the reconstruction problem, and helps to improve the reconstruction accuracy of hyperspectral data.

[0024] (3) The present invention is highly integrated, which reduces the additional instability and uncertainty factors caused by the complexity of the imaging system, and is stable, robust and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the overall structure of the integrated lensless snapshot spectral imaging system of the present invention;

[0026] Figure 2 (a) Schematic diagram of imaging optical path and (b) Schematic diagram of color mask of the integrated lensless snapshot spectral imaging system of the present invention;

[0027] Figure 3 (a) a schematic diagram of color mask parameters and (b) an example diagram of filter response of the present invention;

[0028] Figure 4 is the PSF actually calibrated for different spectral channels in the embodiment of the present invention;

[0029] Figure 5 The composite RGB image is a composite of (a) the measurement values ​​collected and (b) the reconstructed spectral image according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0031] like Figure 1As shown, the integrated lensless snapshot spectral imaging system provided in this embodiment includes the following modules: a filter window module, a spatial spectrum compression modulation module, a color acquisition module and a calculation reconstruction module, wherein the filter window module, the spatial spectrum compression modulation module and the color acquisition module are connected in sequence, and the spatial spectrum compression modulation module and the color acquisition module are respectively connected to the calculation reconstruction module.

[0032] The filter window module selects the band range of hyperspectral imaging, and the light emitted by the scene passes through the filter window module to filter out the light outside the window band. In this embodiment, the filter window module uses a blocking filter F1 to select the light in the 450nm to 650nm band to pass through, so as to prevent the light in other bands from bringing crosstalk and affecting the reconstruction quality.

[0033] The spatial spectrum compression modulation module projects the point light source in the scene to the color acquisition module in a multiplexed manner through filter coding and diffraction projection. In this embodiment, the spatial spectrum compression modulation module uses a color mask CM, which is a circular device with a diameter of 2 cm. It is prepared by inkjet printing on a fully transparent film and has the characteristics of compact structure and simple processing. The color mask CM has spectral filter units with different filter responses distributed at different spatial positions, such as Figure 2 (b) in the figure, the transmittance function is denoted as M λ (x M ,y M ), where λ is the wavelength band, (x M ,y M ) is the spatial position of the color mask plane. Figure 3 The distribution of different spectral filter units in the local area of ​​the color mask and the corresponding spectral filter response curves are shown, where as shown in Figure (a), the local area of ​​the color mask is composed of spectral filter units of different colors, which are abbreviated as R (red), G (green), B (blue), C (cyan), M (magenta) and Y (yellow), and the corresponding spectral filter response curves are shown in Figure (b). These spectral filter units are arranged in a randomly distributed manner to form a color mask. It should be noted that the design form of the color mask is not unique, as long as it has the characteristic that spectral filter units with different filter responses are distributed in different spatial positions of the described device, it can be used as a color mask. For light sources of different spectral channels, the color mask has different amplitude modulation methods, resulting in different coherent diffraction patterns, such as Figure 4 The color mask encodes the scenes of different spectral channels with different amplitudes, and after a diffraction propagation of a distance d, it produces point spread functions of different coherent diffraction patterns, recorded as PSF. λ (x0,y0), where (x0,y0) is the spatial position of the sensor plane.

[0034] The color acquisition module is a multispectral image acquisition sensor. This embodiment uses a consumer-grade RGB color sensor, and the sensor is integrated with a Bayer filter array arranged in the form of RGGB. The filter array can be in the form of a 2×2 Bayer filter, or various wide-spectrum filter forms such as 3×3, 4×4, etc.; a grayscale sensor can also be used, that is, a 1×1 filter form, each pixel has the same spectral response, and there is only one spectral encoding channel. Regardless of the form of filter array used by the system of the present invention, it always has high-spectral imaging capabilities, but the difference in filtering form will affect the richness of spectral encoding, thereby affecting the quality of the reconstructed image. The filter array spatially downsamples the diffraction projection into multiple spectral encoding channels, and each spectral encoding channel has a different filter response form. Its filter response is recorded as L C (λ), where C is the number of spectral encoding channels. The diffraction projection passes through the filter array and is recorded by the sensor, which further spectrally encodes the scene to obtain a two-dimensional measurement value Y C (x,y), imaging principle as Figure 2 As shown in (a) in the figure. Two-dimensional measurement value Y C (x,y) can be described as:

[0035]

[0036] Among them, S λ (x0, y0) is the intensity at different spatial positions under the scene-specific spectrum, that is, the real spectral data cube. C Example measurement values ​​of (x, y) are as follows Figure 5 (a) in .

[0037] Calculate the reconstruction module, using the calibrated point spread function PSF λ (x0, y0) and sensor filter response curve L C The prior information provided by (λ) is used to reconstruct the hyperspectral data cube from the two-dimensional measurements of the spatial spectral coupling collected by the color acquisition module. This embodiment uses a computing medium to convert the point spread function PSF λ (x0, y0), L generated by the color acquisition module calibration C (λ) and the measured value Y C (x, y) is used as input, and the hyperspectral data cube reconstructed after processing is used as output. The reconstructed spectral image of this embodiment is synthesized into an RGB image. Figure 5 (b) shows that the measured value of the natural image texture structure has been completely lost. Figure 5 (a) The reconstructed image has restored the scene texture structure. In the center of the image, a photoelectric device box of Sorebo Company can be seen, with the word "LAB" and a big-eared puppy printed on the box.

[0038] The blocking filter F1 of the filter window module and the color mask CM of the spatial spectrum compression modulation module are closely arranged and placed, with no gap between them. The size of the blocking filter F1 is required to completely cover the size of the color mask CM to prevent some light from passing through the color mask without blocking filtering, causing crosstalk between spectra. The color mask and the RGB color sensor are closely arranged with a small interval of about 1 mm.

[0039] The computing medium is a program pre-set on the processor, and the processor can be a computer or an embedded processor platform, and the program executes the designed hyperspectral reconstruction algorithm. In this embodiment, the hyperspectral reconstruction algorithm is run on MATLAB software, and the total variation (TV) regularization prior is added to the three dimensions of the space spectrum for constraint, reducing the feasible domain range of the reconstruction problem, and using the alternating vector multiplier method (ADMM) for iterative solution to reconstruct the hyperspectral image data.

[0040] The present invention adopts random color mask and pixel-level periodic filtering for joint modulation, and reconstructs the hyperspectral image by calculation. It has the remarkable characteristics of high integration, design flexibility and structural stability, and has created an integration paradigm for the new generation of snapshot spectral imaging.

[0041] Obviously, the above embodiments are only some embodiments of the present invention, rather than all embodiments. The above embodiments are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention. Based on the above embodiments, all other embodiments obtained by those skilled in the art without making creative work, that is, all modifications, equivalent substitutions and improvements made within the spirit and principles of this application, fall within the protection scope required by the present invention.

Claims

1. An integrated lensless snapshot spectral imaging system, characterized in that: It includes a filter window module, a spatial spectrum compression modulation module, a color acquisition module and a calculation and reconstruction module, wherein the filter window module, the spatial spectrum compression modulation module and the color acquisition module are connected in sequence, and the spatial spectrum compression modulation module and the color acquisition module are respectively connected to the calculation and reconstruction module; The filter window module is used to filter out light outside the window band; The spatial spectrum compression modulation module is used to perform different amplitude encoding on scenes of different spectral channels, and to project point light sources in the scene to the color acquisition module in a multiplexed manner; The color acquisition module is used to perform spectral encoding on the scene to obtain a two-dimensional measurement value; The calculation and reconstruction module is used to reconstruct the hyperspectral image data based on the two-dimensional measurement values.

2. The integrated lensless snapshot spectral imaging system according to claim 1, characterized in that: The spatial spectrum compression modulation module adopts a color mask, and spectrum filtering units with different filtering responses are distributed at different spatial positions on the color mask.

3. The integrated lensless snapshot spectral imaging system according to claim 2, characterized in that: The color mask and the filter window module are arranged closely together.

4. The integrated lensless snapshot spectral imaging system according to claim 2, characterized in that: The color mask and the color acquisition module are arranged closely together with a small interval.

5. The integrated lensless snapshot spectral imaging system according to claim 1, characterized in that: The color acquisition module is a multi-spectral image acquisition sensor, and a Bayer filter array is integrated on the sensor.

6. The integrated lensless snapshot spectral imaging system according to claim 1, characterized in that: The computational reconstruction module utilizes a hyperspectral reconstruction algorithm to reconstruct a hyperspectral data cube from the two-dimensional measurements of spatial-spectral coupling acquired by the color acquisition module.

7. A method of using the integrated lensless snapshot spectral imaging system as claimed in claim 1, characterized in that: The method comprises the following steps: S1, the light emitted by the scene passes through the filter window module, and the light outside the window band is filtered out; S2, using the spatial spectrum compression modulation module to perform different amplitude encoding and diffraction projection on scenes of different spectral channels, and projecting point light sources in the scene to a color acquisition module in a multiplexed manner; S3, the color acquisition module further spectrally encodes the scene to obtain a two-dimensional measurement of spatial-spectral coupling; S4, reconstructing the hyperspectral image data using a hyperspectral reconstruction algorithm based on the two-dimensional measurement value of the spatial-spectral coupling.

8. The method according to claim 7, characterized in that In step S2, the spatial spectrum compression modulation module uses a color mask to perform different amplitude encoding on scenes of different spectral channels, and after diffraction propagation over a distance d, generates point spread functions of different coherent diffraction patterns, which are recorded as PSF λ (x0,y0), where (x0,y0) is the spatial position of the sensor plane.

9. The method according to claim 8, characterized in that In step S3, the color acquisition module is a multispectral image acquisition sensor, and a Bayer filter array is integrated on the sensor; the filter array downsamples the diffraction projection into multiple spectral encoding channels in space, and each spectral encoding channel has a different filter response form, and its filter response is recorded as L C (λ), where C is the number of spectral encoding channels; the diffraction projection passes through the filter array and is recorded by the sensor, which further spectrally encodes the scene to obtain a two-dimensional measurement value Y C (x,y): Among them, S λ (x0, y0) is the intensity at different spatial positions under the scene-specific spectrum, that is, the real spectral data cube.

10. The method according to claim 9, characterized in that In step S4, the point spread function PSF generated in step S2 is used λ (x0, y0), the filter response L obtained in step S3 C (λ) and the two-dimensional measurement value Y C (x,y), reconstruct the hyperspectral image data.

Images