Prism-echelette modulated spectral video imaging system

Abstract

The application discloses a prism-ladder mirror modulated spectral video imaging system. The system comprises a scene capturing module, an optical collimation module, a three-dimensional space-spectrum modulation module, a decoupling acquisition module, a coupling acquisition module and a fusion reconstruction module. After the scene information captured by the scene capturing module passes through the optical collimation module, the information is input into the three-dimensional space-spectrum modulation module, and after image slice dispersion, dispersion, spectrum band selection and light vector modulation, a spectral image is formed, in which the three-dimensional spectral data of the scene is decoupled and two-dimensionally expanded. Then, the data is sampled and input into the decoupling acquisition module, and the unsampled data is acquired by the coupling acquisition module. The spectral information of different scales acquired by the two acquisition modules is reconstructed into a complete space-spectrum data cube through the fusion reconstruction module. The application utilizes the reflection and transmission of the micromirror array to modulate the three-dimensional space-spectrum data, reduces the information loss during data modulation, and can significantly improve the luminous flux and reconstruction accuracy of the system.

Description

Technical Field

[0001] This invention relates to a full-flux prism-stepped mirror modulation spectral video imaging system, belonging to the field of spectral imaging. Background Technology

[0002] Inspired by the human eye's perception of the three primary colors, traditional cameras use Bell array filters to record visual information. However, the real world's illumination and reflectance spectra are diverse and complex, making it impossible to capture the intricate details of natural scene spectra using simple three-color representation methods. Spectral imaging that surpasses the limits of the human eye has become a new hot research area, offering new opportunities to solve complex computer vision problems.

[0003] Traditional spectral imaging techniques, including time-series spectral filtering and spatial point-line scanning, cannot simultaneously acquire spectral data with two-dimensional spatial resolution for dynamic objects, limiting their further application in many fundamental research fields. In recent years, spectral acquisition has evolved from static scanning to instantaneous imaging of dynamic scenes, moving towards high-speed acquisition, and is expected to be widely applied in fields such as industrial monitoring, military security, and combustion science.

[0004] With the increase in video frame rate, signal undersampling and shorter single-frame exposure times (reduced optical signal integration time) lead to a significant decrease in luminous flux, affecting the reconstruction accuracy and optical path signal-to-noise ratio of spectral images. Existing mainstream spectral video imaging technologies—Coded Aperture Snapshot Spectroscopy (CASSI), Tomographic Imaging Spectroscopy (CTIS), and Prism Mask Spectroscopic Video Imaging System (PMVIS)—struggle to balance the constraints between the temporal, spatial, and spectral dimensions. However, the low image quality caused by spatial-spectral signal modulation and low luminous flux, along with the high temporal and spatial complexity of the algorithms, prevents existing spectral video imaging technologies from being flexibly applied to various scenarios. For specific scenarios with high imaging requirements, existing systems and methods are inadequate. Therefore, there is an urgent need for an imaging system that can provide high luminous flux and low reconstruction complexity to achieve rapid, flexible, and high-quality acquisition of spectral video. Summary of the Invention

[0005] To address the shortcomings of the existing technology, this invention provides a prism-stepped mirror modulation spectral video imaging system.

[0006] The technical solution adopted in this invention is as follows:

[0007] A prism-stepped mirror modulation spectral video imaging system includes a scene capture module, an optical collimation module, a three-dimensional spatial-spectral modulation module, a decoupled acquisition module, a coupled acquisition module, and a fusion reconstruction module. Scene information captured by the scene capture module is input to the three-dimensional spatial-spectral modulation module after passing through the optical collimation module. The three-dimensional spatial-spectral modulation module performs image slicing discretization, dispersion, spectral band selection, and optical vector modulation to form a spectral image after decoupling and two-dimensional unfolding of the scene's three-dimensional spectral data. The unfolded data cube is then sampled and input to the decoupled acquisition module, while unsampled data is coupled and acquired by the coupled acquisition module. The two different scale spectral information acquired by the decoupled and coupled acquisition modules are then reconstructed into a complete spatial-spectral data cube through the fusion reconstruction module.

[0008] Furthermore, the three-dimensional space-spectrum modulation module includes an image slicing device X1, a dispersive prism L1, a bar-shaped periodic mask M, a light vector modulation prism L2, and a stepped mirror array X2; the optical collimation module causes the light vector direction to be perpendicular to the principal optical axis and incident on the image slicing device X1, and then sequentially pass through the dispersive prism L1, the bar-shaped periodic mask M, the light vector modulation prism L2, and the stepped mirror array X2.

[0009] Furthermore, the image slicing device X1 is composed of a group of strip-shaped reflective mirrors that are closely arranged in the direction of the principal optical axis and discretely arranged in the direction perpendicular to the principal optical axis, with an angle of 45° between them and the direction of the principal optical axis; the image slicing device X1 slices the collimated scene information uniformly along the direction of the principal optical axis and discretely along the direction perpendicular to the principal optical axis.

[0010] Furthermore, the slits of the strip-shaped periodic mask M are arranged at equal intervals on the mask plane, and the ratio of the slit width to the slit spacing is set to 1:1, which is used to extract the spectral bands of the scene information after dispersion.

[0011] Furthermore, the period of the strip periodic mask M is consistent with the distribution period of the strip reflective surface of the image slicing device X1, that is, the slit width + slit spacing = the projection width d of the strip reflective surface in the image slicing device X1 in the direction perpendicular to the principal optical axis + the distance K*d between adjacent strip reflective surfaces in the image slicing device X1 in the direction perpendicular to the principal optical axis, where K is the ratio of the distance between adjacent strip reflective surfaces to the width of the strip reflective surface.

[0012] Furthermore, the dispersive prism L1 and the optical vector modulation prism L2 are two right-angled triangular prisms with identical material and size, with their dispersive surfaces parallel to each other and their right-angled surfaces perpendicular to the principal optical axis. The relative position of the strip-shaped periodic mask M between the dispersive prism L1 and the optical vector modulation prism L2 is adjustable.

[0013] Furthermore, the stepped mirror array X2 is composed of rectangular micro-reflective mirrors that are discretely arranged in both the principal optical axis direction and the direction perpendicular to the principal optical axis. It is used to uniformly sample the scene spectral information that has been unfolded in two dimensions and is incident parallel to the principal optical axis. A portion of the spectral information is reflected by the mirrors and enters the decoupled acquisition module, while the spectral information that is not reflected by the mirrors enters the coupled acquisition module.

[0014] Furthermore, the rectangular micro-reflective mirror surface forms a 45° angle with both the two-dimensional space-spectral image plane in the direction of the principal optical axis and the direction perpendicular to the principal optical axis.

[0015] Furthermore, the size S of the micro-reflective mirror is equal to the projection width d of the strip-shaped reflective mirror in the image slicing device X1 in the direction perpendicular to the principal optical axis, and the ratio N between adjacent micro-reflective mirrors in the horizontal or vertical direction of the stepped mirror array X2 to the width of the micro-reflective mirror is equal to the ratio K between adjacent strip-shaped reflective mirrors in the image slicing device X1 to the width of the strip-shaped reflective mirror.

[0016] Furthermore, the decoupled acquisition module consists of a lens and a grayscale camera, while the coupled acquisition module consists of a lens and a grayscale camera with a large area array.

[0017] The main contributions of this invention include the following three points:

[0018] (1) Optical devices for three-dimensional spatial spectrum modulation: stepped mirrors

[0019] In existing spectral imaging systems, common optical path modulation devices mainly include filters, masks, image slicing devices, prisms, microlens arrays, gratings, etc. However, these devices can only transform spectral data cubes at a two-dimensional level. Therefore, when imaging systems constructed with these devices are used to image three-dimensional data, they are always inadequate and lack flexibility. Therefore, this invention introduces a stepped mirror with a periodically discrete distribution in space to sample the data cube after it has been unfolded in two dimensions, while unsampled data is coupled and acquired. Thus, the spectral imaging system captures two data streams: one stream captures high spatial resolution and high spectral resolution image data, but contains relatively little information (uniform sampling on the data cube); the other stream captures low spatial resolution and low spectral resolution data, but couples a large amount of spatial and spectral information. The complete data cube is reconstructed using these two sets of information at different scales. At the same time, these two sets of data have a simple mapping relationship on the detector matrix, which significantly reduces the spatiotemporal complexity of the reconstruction algorithm compared to CTIS and CASSI. Compared to existing technologies that use high spectral resolution and low spatial resolution data formats or low spectral resolution and high spatial resolution data formats for spectral fusion, the pair of data obtained in this application provides more spatial-spectral information, which helps to improve the accuracy of hyperspectral reconstruction.

[0020] (2) Spatial spectrum data modulation method with stepped mirror-symmetric prism group as the core

[0021] For scene light after collimation, this invention converts it using an image slicing device X1 to obtain spatially discrete scene information. At this point, it is necessary to disperse the scene information to project different frequency components of the scene light in space. However, the sampling of the stepped mirror needs to ensure the consistency of the incident light angle at different depth positions. Therefore, a symmetrical prism with consistent dispersion direction is used here to disperse and modulate the discrete scene information. Overall, the spatial-spectral data modulation method centered on the stepped mirror-symmetrical prism group obtains a set of easily reconstructable spatial-spectral data at different information scales.

[0022] (3) A strip-shaped periodic mask placed between symmetrical prism groups enables flexible selection of a certain range of spectral channels.

[0023] The strip-shaped periodic mask M placed between symmetrical prisms is equivalent to a bandpass filter. It can flexibly extract image information of a specific spectral range in the spectral domain by changing its spatial position (in the direction of the dispersion plane and perpendicular to the dispersion plane).

[0024] Therefore, the solution of the present invention has the following advantages:

[0025] (1) Compared with existing computational spectral video imaging systems that utilize mask design, the system of the present invention uses the reflection and transmission of micro-mirror arrays to perform three-dimensional space-spectrum data modulation, which reduces information loss during data modulation and can significantly improve the light throughput of the system.

[0026] (2) The spectral video imaging system designed in this invention captures a set of easily reconstructable spatial-spectral data at different scales, which can significantly reduce the spatiotemporal complexity of the back-end reconstruction algorithm and adapt to high-speed scene tasks; at the same time, the two data forms cover more spatial-spectral information, which helps to improve the accuracy of hyperspectral reconstruction.

[0027] (3) It provides flexible selection of spectral bands within a certain range. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the spectral video imaging system of the present invention;

[0029] Figure 2 This is a diagram of the optical path structure in an embodiment of the present invention;

[0030] Figure 3 This is a three-dimensional structural diagram of the image slicing device and the stepped mirror array according to an embodiment of the present invention;

[0031] Figure 4This is a geometric structure diagram of the three-dimensional spatial spectrum modulation module of the present invention;

[0032] Figure 5 This is a schematic diagram of the data format acquired by the decoupled acquisition module of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] like Figure 1 , Figure 2 As shown, the prism-stepped mirror modulation spectral video imaging system provided in this embodiment includes the following modules: a scene capture module, an optical collimation module, a three-dimensional spatial-spectral modulation module, a decoupled acquisition module, a coupled acquisition module, and a fusion reconstruction module. The scene information captured by the scene capture module is then input to the three-dimensional spatial-spectral modulation module after passing through the optical collimation module. The three-dimensional spatial-spectral modulation module performs image slicing discretization, dispersion, spectral band selection, and optical vector modulation to form a spectral image after decoupling and two-dimensional unfolding of the scene's three-dimensional spectral data. Based on this, stepped mirrors with periodic discrete distributions in space are used to sample the two-dimensionally unfolded data cube, and the sampled data is input to the decoupled acquisition module. Unsampled data is coupled and acquired by the coupled acquisition module. Finally, the two spectral information streams at different scales are reconstructed into a complete spatial-spectral data cube through the fusion reconstruction module.

[0035] The scene capture module can select an imaging lens composed of a convex lens or a lens group, or other specially customized imaging lenses, to capture scene information.

[0036] The optical collimation module optically collimates the scene information acquired by the scene capture module, ensuring that the light vector direction is perpendicular to the principal optical axis and incident on the next module. In this embodiment, the optical collimation module is a collimating lens that matches the lens.

[0037] The three-dimensional spatial spectrum modulation module can perform three-dimensional spatial spectrum modulation on spectral information incident perpendicular to the principal optical axis. Specifically, it includes the following optical components:

[0038] Image slicing device X1: It consists of a group of strip-shaped reflective mirrors that are closely arranged in the direction of the principal optical axis and discretely arranged in the direction perpendicular to the principal optical axis, with an angle of 45° between them and the direction of the principal optical axis. The image slicing device X1 can uniformly slice the collimated scene information along the direction of the principal optical axis and uniformly discretize it in the direction perpendicular to the principal optical axis, with a discretization coefficient of K (the ratio of the distance between adjacent slices to the slice width).

[0039] Dispersive prism L1: Disperses the sliced ​​scene information;

[0040] Strip periodic mask M: On the strip periodic mask, slits are evenly distributed on the mask plane, and the ratio of slit width to slit spacing is set to 1:1. By utilizing the gating characteristics of the mask in spatial position, the spectral band of the scene information after dispersion can be intercepted.

[0041] Light vector modulation prism L2: modulates the direction of the dispersed optical information light vector to be parallel to the principal optical axis.

[0042] With the combined action of the image slicing device X1, the dispersive prism L1, the bar-shaped periodic mask M, and the optical vector modulation prism L2, a two-dimensional unfolding of the spectrum incident parallel to the principal optical axis is achieved. The dispersive prism L1 and the optical vector modulation prism L2 are two right-angled triangular prisms of identical material and size, with their dispersive surfaces parallel to each other and their right-angled surfaces perpendicular to the principal optical axis. The relative position of the bar-shaped periodic mask M between the dispersive prism L1 and the optical vector modulation prism L2 is adjustable.

[0043] Stepped mirror array X2: It is composed of rectangular micro-reflective mirrors that are discretely arranged in both the principal optical axis direction and the direction perpendicular to the principal optical axis. It can uniformly sample the spectral information of the scene after two-dimensional unfolding. A part of the optical information is reflected by the mirrors and enters the decoupled acquisition module, while the spectral information that is not reflected by the mirrors enters the coupled acquisition module.

[0044] The decoupled acquisition module in this embodiment consists of a lens and a grayscale camera, which acquires and images the data sampled from the stepped mirror array, such as... Figure 5 The image shows a schematic diagram of the data format captured by the decoupled acquisition module in this embodiment. The image contains PQ pairs of numbers, where P represents the spatial location and Q represents the spectral band. Overall, the decoupled acquisition module, under the action of the three-dimensional spatial-spectral modulation module, achieves data capture in the form of uniform sampling of three-dimensional spatial-spectral data cubes.

[0045] The coupling acquisition module in this embodiment consists of a lens and a large-area grayscale camera, which couples and images the unsampled portion of the stepped mirror array. In particular, the array size (resolution) of the large-area camera in the coupling acquisition module will affect the reconstruction accuracy of the subsequent fusion reconstruction; therefore, it is advisable to use a camera with a larger array size and higher resolution for coupling acquisition.

[0046] The fusion reconstruction module couples and reconstructs a set of information at different scales captured by the decoupled acquisition module and the coupled acquisition module. It reconstructs a complete spatial-spectral data cube with high spatial resolution, high spectral resolution, and high information throughput from both low spatial resolution, high spectral resolution, and low information throughput data formats. Depending on different hardware parameters and band selections, the parameters of the fusion reconstruction algorithm need to be appropriately adjusted to adapt to the specific reconstruction task. Optionally, traditional machine learning methods such as bilateral filtering algorithms and deep learning methods can be used to fuse and reconstruct the two data formats.

[0047] like Figure 4 The diagram shown is a geometric structure of the three-dimensional spatial spectrum modulation module in this embodiment. Since the dispersive prism L1 and the optical vector modulation prism L2 are made of the same material, and assuming that the refractive index of the prism for light with wavelength λ1 is n1 and the refractive index for light with wavelength λ2 is n2, then:

[0048]

[0049] Therefore, for the dispersive prism L1 and the optical vector modulation prism L2, the incident and outgoing rays are parallel.

[0050] Let the slit width on the photomask be h. When h is the primary constraint, i.e., when the photomask can effectively select the wavelength band, we have:

[0051]

[0052] At this point, let the region imaged on the stepped mirror array X2 be H, then:

[0053]

[0054] When the slit width h is the primary constraint, it means that the system parameters can be changed by altering the relative position of the periodic strip mask M between the symmetrical prisms. As shown in the above formula, changing d1 and d3 can change H, i.e., the range of spectral bands that the system can simultaneously capture. Simultaneously, moving the periodic strip mask M along the dispersive plane direction can change the spectral range imaged on the detector. Therefore, this invention can achieve flexible selection of the system's working spectral band within a certain range by adjusting the relative position of the periodic strip mask M between the symmetrical prism group (dispersive prism L1 and optical vector modulation prism L2). The periodicity of the periodic strip mask M should be consistent with the periodicity of the distribution of the strip-shaped reflective surfaces on the image slicing device X1, i.e., slit width + slit spacing = the projected width d of the strip-shaped reflective surfaces in X1 in the direction perpendicular to the principal optical axis + the distance K*d between adjacent strip-shaped reflective surfaces in X1 in the direction perpendicular to the principal optical axis.

[0055] The stepped mirror array X2 in this embodiment is a three-dimensional device, such as... Figure 3 As shown, the projection of the stepped mirror array X2 onto the direction perpendicular to the principal optical axis is a complete reflective mirror, which is composed of several square micro-reflective mirrors that form a 45° angle with both the two-dimensional space-spectral image planes in the direction of the principal optical axis and the direction perpendicular to the principal optical axis. The size S of the micro-reflective mirrors should preferably be set to the level of tens of micrometers or tens of micrometers. In terms of distribution, square micro-reflective mirrors are evenly distributed in each row of the stepped mirror array. The distance between adjacent mirrors in different depth directions is N*S, where N is the ratio of the distance between adjacent micro-mirrors in the horizontal (or vertical) direction of the micro-reflective mirror array X2 to the width of the micro-mirror. The depth gradient difference between square mirror arrays in different depths is also S (ensuring that the projection of a single mirror onto the bottom surface of the stepped mirror array X2 is an S*S square). Optionally, the parameter settings of the stepped mirror array X2 can be consistent with the parameters of the image slicing device X1, that is: K=N, d=S.

[0056] Obviously, the above embodiments are merely some, not all, of the embodiments of the present invention. The above embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention. Based on the above embodiments, all other embodiments obtained by those skilled in the art without inventive effort, that is, all modifications, equivalent substitutions, and improvements made within the spirit and principle of this application, fall within the scope of protection claimed by the present invention.

Claims

1. A prism-stepped mirror modulation spectral video imaging system, characterized in that, The system includes a scene capture module, an optical collimation module, a 3D spatial-spectral modulation module, a decoupled acquisition module, a coupled acquisition module, and a fusion reconstruction module. Scene information captured by the scene capture module is processed by the optical collimation module and then input to the 3D spatial-spectral modulation module. The 3D spatial-spectral modulation module performs image slicing, dispersion, spectral band selection, and optical vector modulation to form a spectral image after decoupling and 2D unfolding of the scene's 3D spectral data. The unfolded data cube is then sampled and input to the decoupled acquisition module, while unsampled data is coupled and acquired by the coupled acquisition module. The fusion reconstruction module reconstructs the complete spatial-spectral data cube from the two different scales of spectral information acquired by the decoupled and coupled acquisition modules. The three-dimensional space-spectrum modulation module includes an image slicing device (X1), a dispersive prism (L1), a bar-shaped periodic mask (M), a light vector modulation prism (L2), and a stepped mirror array (X2). The optical collimation module directs the light vector perpendicular to the principal optical axis to be incident on the image slicing device (X1), and then sequentially through the dispersive prism (L1), the bar-shaped periodic mask (M), the light vector modulation prism (L2), and the stepped mirror array (X2). The image slicing device (X1) slices the collimated scene information uniformly along the principal optical axis and discretizes it uniformly along the direction perpendicular to the principal optical axis. The dispersive prism (L1) and the optical vector modulation prism (L2) are two right-angled triangular prisms with identical material and size, with their dispersive surfaces parallel to each other and their right-angle surfaces perpendicular to the principal optical axis. The relative position of the strip periodic mask (M) between the dispersive prism (L1) and the optical vector modulation prism (L2) is adjustable. The stepped mirror array (X2) is composed of rectangular micro-reflective mirrors that are discretely arranged in both the principal optical axis direction and the direction perpendicular to the principal optical axis. It is used to uniformly sample the scene spectral information that has been unfolded in two dimensions and is incident parallel to the principal optical axis. A portion of the spectral information is reflected by the micro-reflective mirrors into the decoupled acquisition module, while the spectral information that is not reflected by the micro-reflective mirrors enters the coupled acquisition module.

2. The prism-stepped mirror modulation spectral video imaging system according to claim 1, characterized in that, The image slicing device (X1) consists of a set of strip-shaped reflective mirrors arranged closely in the direction of the principal optical axis and discretely arranged in the direction perpendicular to the principal optical axis, with an angle of 45° between them and the direction of the principal optical axis.

3. The prism-stepped mirror modulation spectral video imaging system according to claim 2, characterized in that, The slits of the strip-shaped periodic mask (M) are arranged at equal intervals on the mask plane, and the ratio of slit width to slit spacing is set to 1:

1. The slits are used to extract the spectral bands of the scene information after dispersion.

4. The prism-stepped mirror modulation spectral video imaging system according to claim 3, characterized in that, The period of the strip periodic mask (M) is consistent with the distribution period of the strip reflective surfaces of the image slicing device (X1), that is, the slit width + slit spacing = the projection width d of the strip reflective surface in the image slicing device (X1) in the direction perpendicular to the principal optical axis + the distance K*d between adjacent strip reflective surfaces in the image slicing device (X1) in the direction perpendicular to the principal optical axis, where K is the ratio of the distance between adjacent strip reflective surfaces to the width of the strip reflective surface.

5. The prism-stepped mirror modulation spectral video imaging system according to claim 1, characterized in that, The rectangular micro-reflective mirror surface forms a 45° angle with both the two-dimensional space-spectral image plane in the direction of the principal optical axis and the direction perpendicular to the principal optical axis.

6. The prism-stepped mirror modulation spectral video imaging system according to claim 2, characterized in that, The size S of the micro-reflective mirror is equal to the projection width d of the strip-shaped reflective mirror in the image slicing device (X1) in the direction perpendicular to the principal optical axis, and the ratio N of the distance between adjacent micro-reflective mirrors in the horizontal or vertical direction of the stepped mirror array (X2) to the width of the micro-reflective mirror is equal to the ratio K of the distance between adjacent strip-shaped reflective mirrors in the image slicing device (X1) to the width of the strip-shaped reflective mirror.

7. The prism-stepped mirror modulation spectral video imaging system according to claim 1, characterized in that, The decoupled acquisition module consists of a lens and a grayscale camera, while the coupled acquisition module consists of a lens and a grayscale camera with a large area array.

Images