Three-dimensional spectral imaging method based on snapshot hyperspectral and pixel-level registration of fringe structured light image

Abstract

This invention relates to a three-dimensional spectral imaging method for pixel-level registration of snapshot-type hyperspectral imaging and fringe structured light. First, the image planes of the snapshot-type hyperspectral imaging module and the fringe structured light three-dimensional imaging module are calibrated, and the homography transformation matrix is ​​calculated to achieve pixel-level registration. Second, the snapshot-type hyperspectral module acquires a two-dimensional diffraction image in a single exposure, and a hyperspectral data cube is reconstructed using deep learning. Third, the fringe structured light module projects sinusoidal fringes, and a three-dimensional point cloud is reconstructed based on the phase-shifting method and the three-frequency heterodyne method. Finally, the spectral and spatial coordinates are fused according to the homography matrix to generate four-dimensional data points with three-dimensional spatial coordinates and one-dimensional spectral attributes. This method achieves synchronous, rapid, and pixel-level fusion acquisition of target spatial geometric information and spectral material information, possessing quasi-dynamic and even near-real-time four-dimensional monitoring capabilities, and can meet the high-time-sensitivity application scenarios such as burn wound triage, intraoperative neuronavigation, oil spill monitoring, and dynamic sorting.

Description

Technical Field

[0001] This invention belongs to the field of spectral imaging and three-dimensional measurement technology, specifically involving a three-dimensional spectral imaging method based on pixel-level registration of snapshot hyperspectral and striped structured light. It has quasi-dynamic and even near-real-time four-dimensional monitoring capabilities, which can meet the needs of applications with high timeliness requirements such as rapid triage of burn wounds, intraoperative three-dimensional navigation of blood vessels and nerves, monitoring of marine oil spill diffusion, and rapid identification and sorting of objects of different materials in a cluttered environment on a moving conveyor belt. Background Technology

[0002] Traditional two-dimensional spectral monitoring techniques cannot distinguish aliased reflection signals at different altitudes, leading to spectral interpretation distortion. In medical and environmental applications, relying solely on spatial geometric information or spectral features has limitations; simultaneous acquisition of the target's three-dimensional spatial morphology and spectral information is necessary for accurate identification and quantitative analysis. Existing four-dimensional spectral depth imaging schemes have failed to achieve a good balance between acquisition efficiency and multidimensional accuracy. Therefore, there is an urgent need for a four-dimensional imaging technology that can simultaneously achieve high spectral resolution, high depth accuracy, and rapid acquisition.

[0003] In the medical field, the assessment of burn severity depends on depth (first-degree, second-degree, third-degree) and total volume, rather than simply planar area. Traditional hyperspectral imaging can distinguish burn tissues at different depths (using indicators such as blood oxygen saturation and water content), but it can only present the results as two-dimensional pseudo-color images. It cannot accurately calculate the volume of necrotic tissue and is difficult to adapt to the assessment of wounds on complex curved surfaces such as joints and the face. Furthermore, in surgeries such as spinal surgery and plastic surgery, blood vessels and nerves often run on curved surfaces, in depressions, or are obscured by tissue. The projection distortion of two-dimensional hyperspectral imaging onto complex anatomical structures can lead to spatial misjudgments, affecting incision design and intraoperative safety.

[0004] In the field of marine ecology, coral reef ecosystems are among the most biodiverse ecosystems in the ocean. The health of corals is closely related to the photosynthetic pigment content of their symbiotic zooxanthellae. When corals are subjected to environmental stresses such as seawater warming, acidification, or pollution, it can lead to coral bleaching or even death. Traditional coral health assessments rely on visual observation by underwater personnel or single-point spectral sampling, making it difficult to achieve simultaneous quantitative characterization of three-dimensional morphology and pigment spatial distribution. Traditional hyperspectral imaging techniques are mostly two-dimensional imaging, unable to distinguish spectral reflectance signals superimposed at different depths or altitudes, resulting in distorted spectral interpretation. Seafloor topographic mapping and oil spill exploration face similar challenges: two-dimensional hyperspectral imaging can identify the spectral characteristics of oil slicks, but cannot accurately obtain the thickness distribution and spatial volume of oil slicks on complex sea surface or seafloor topography; while lidar can acquire topographic data, it lacks component identification capabilities. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a three-dimensional spectral imaging method based on pixel-level registration of snapshot-type hyperspectral imaging and stripe structured light, comprising: The first step is to calibrate the image planes of the snapshot hyperspectral imaging module and the striped structured light 3D imaging module, calculate the homography transformation matrix between the two image planes, and achieve pixel-level registration. The second step involves using a snapshot hyperspectral imaging module to acquire a two-dimensional diffraction image of the target in a single exposure, and then reconstructing a hyperspectral data cube using a deep learning network. The third step involves using a striped structured light 3D imaging module to project a sinusoidal stripe pattern onto the target, acquiring deformed stripe images, calculating the absolute phase based on the phase-shifting method and the three-frequency heterodyne method, and reconstructing the 3D point cloud of the target. The fourth step involves fusing the spectral vectors in the hyperspectral data cube obtained in the second step with the spatial coordinates of the three-dimensional point cloud obtained in the third step, based on the homography matrix calibrated in the first step, to generate four-dimensional data points with three-dimensional spatial coordinates and one-dimensional spectral attributes.

[0006] The method described above acquires four-dimensional data in 0.3 seconds through rapid alternation between a single three-dimensional scan and a single hyperspectral acquisition, and has quasi-dynamic or even near real-time four-dimensional monitoring capabilities.

[0007] In the third step, the three-dimensional imaging of striped structured light uses the three-frequency heterodyne method to project sinusoidal stripe patterns of different frequencies, and extracts the wrapped phase through the four-step phase shift method. After two-level heterodyne operation, the unambiguous absolute phase of the entire field of view is obtained.

[0008] In the first step, a checkerboard calibration board is used for calibration. Subpixel-precision corner coordinates are extracted from the imaging results of the two modules, and the homography transformation matrix is ​​calculated using the RANSAC algorithm.

[0009] In the third step, the mapping from absolute phase to three-dimensional coordinates adopts a lookup table method. The mapping relationship from phase to depth is pre-calculated during the calibration stage. During online reconstruction, the depth value is directly obtained by looking up the table and combined with the camera intrinsic parameters for back projection to obtain the complete three-dimensional coordinates.

[0010] The deep learning network in the second step is an end-to-end network used to reconstruct a hyperspectral data cube from a single frame of two-dimensional diffraction image.

[0011] The fourth step specifically includes: using the inverse transformation of the homography matrix to convert the pixel coordinates on the image plane of the structured light camera into hyperspectral image plane coordinates, and then extracting the corresponding spectral vectors from the hyperspectral data cube and matching them with the three-dimensional spatial points.

[0012] The method employs a system, the system comprising: The shared front optical path includes a zoom lens, a variable aperture, a filter, and a collimating lens, and guides the incident light to the hyperspectral imaging module and the 3D imaging module through a beam splitter. A snapshot-type hyperspectral imaging module, connected to the first optical path of the beam splitter, includes an imaging lens, a dispersive grating, and a first camera; The striped structured light 3D imaging module is connected to the second optical path of the beam splitter and includes a DLP projector, an imaging lens, and a grayscale camera. The DLP projector is also used to project a pure white image when the snapshot hyperspectral imaging module is working, providing broadband uniform illumination.

[0013] The spectral coverage of the hyperspectral data cube is the visible light band.

[0014] The four-dimensional data points simultaneously contain the target's spatial geometric information and spectral material information. Attached Figure Description

[0015] Figure 1 This is a step diagram of a three-dimensional spectral imaging method based on pixel-level registration of snapshot hyperspectral and striped structured light.

[0016] Figure 2 This is a diagram showing the optical path structure design for the training phase of a snapshot hyperspectral imaging system.

[0017] In the diagram: 1. Zoom lens; 2. Variable aperture; 3. Filter; 4. Collimating lens; 5. Beam splitter; 6. Liquid crystal tunable filter; 7. Focusing lens; 8. Liquid crystal tunable filter camera; 9. Two-dimensional grating; 10. Imaging lens; 11. Snapshot hyperspectral imaging module camera.

[0018] Figure 3 This is a design diagram of the optical path structure for the four-dimensional imaging stage.

[0019] In the diagram: 1. Zoom lens; 2. Variable aperture; 3. Filter; 4. Collimating lens; 5. Beam splitter; 7. Focusing lens; 9. Two-dimensional grating; 10. Imaging lens; 11. Snapshot hyperspectral imaging module camera; 12. Three-dimensional imaging camera.

[0020] Figure 4 To reconstruct the point cloud and its best-fit plane for a standard dot array flat plate.

[0021] Figure 5 Three-dimensional point cloud reconstruction and morphological analysis of coral samples.

[0022] Figure 6 To fusion the 4D imaging data of corals, the point cloud is given pseudo-color using the spectral signals of the RGB three channels. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0024] Figure 1 is a flowchart illustrating the steps of the three-dimensional spectral imaging method based on pixel-level registration of snapshot hyperspectral imaging and striped structured light according to an embodiment of the present invention. It fully presents the entire process from system calibration to four-dimensional data fusion, specifically including four steps: The first step is pixel-level registration calibration, in which a checkerboard calibration board is used to jointly calibrate the snapshot hyperspectral imaging module and the striped structured light three-dimensional imaging module, and the homography matrix between the two modules is calculated. H The second step is snapshot hyperspectral imaging, which acquires a two-dimensional diffraction image through a single exposure, and then uses a reconstruction network based on UNet3+ to perform spectral reconstruction to obtain a hyperspectral data cube (x, y, λ). The third step is fringe structured light three-dimensional imaging, which projects sinusoidal fringes through a DLP projector, acquires deformed fringe images through a three-dimensional imaging camera, and uses the three-frequency heterodyne method to solve the phase and reconstruct a three-dimensional point cloud (X, Y, Z). The fourth step is four-dimensional data fusion, which utilizes the inverse matrix of the homography matrix. H -1 The coordinates of the three-dimensional point cloud are mapped to the hyperspectral image coordinate system, and the spectral vectors corresponding to each point are assigned to the three-dimensional point cloud. Finally, four-dimensional data points (X, Y, Z, S (λ)) containing spatial coordinates and spectral information are obtained.

[0025] This embodiment first constructs a snapshot hyperspectral imaging system. Based on the principle of computational tomographic imaging (CTIS), this system uses diffraction elements to project a three-dimensional hyperspectral cube onto a two-dimensional plane, and then utilizes a reconstruction network based on UNet3+ for spectral reconstruction. As shown in Figure 2, the system's training phase optical path structure design diagram mainly includes two parts: a snapshot hyperspectral imaging module and a hyperspectral truth acquisition module. The component numbers and names in Figure 2 are as follows: zoom lens 1, variable aperture 2, filter 3, collimating lens 4, beam splitter 5, liquid crystal tunable filter 6, focusing lens 7, liquid crystal tunable filter camera 8, two-dimensional grating 9, imaging lens 10, and snapshot hyperspectral imaging module camera 11. The system uses beam splitter 5 (Figure 2) to divide the incident light into two independent optical paths. Both cameras are MV-CS200-10GM models to obtain high spatial resolution images: one path is the hyperspectral ground truth acquisition optical path, connected to a tunable liquid crystal filter (LCTF) module consisting of liquid crystal tunable filter 6, focusing lens 7, and liquid crystal tunable filter camera 8 (Figure 2), used to acquire hyperspectral ground truth data; the other path is the CTIS imaging optical path, connected to the two-dimensional grating 9 and imaging lens 10 (Figure 2). Figure 2The snapshot-type hyperspectral imaging module camera 11 is used to acquire CTIS two-dimensional diffraction image data.

[0026] After completing the network training and parameter calibration during the training phase, the hyperspectral truth acquisition module (including the liquid crystal tunable filter 6 and the liquid crystal tunable filter camera 8) in Figure 2 is removed. The shared optical path components 1-5, 7, 9-11 and the CTIS module (including the two-dimensional grating 9, the imaging lens 10, and the snapshot hyperspectral imaging module camera 11) are retained for snapshot hyperspectral imaging. At the same time, the original hyperspectral truth acquisition optical path is replaced with the striped structured light three-dimensional imaging module, and the component three-dimensional imaging camera 12 is added to form the optical path structure design diagram of the four-dimensional imaging stage shown in Figure 3.

[0027] As shown in Figure 3, the depth-hyperspectral four-dimensional snapshot imaging system constructed in this embodiment mainly includes two parts: a snapshot-type hyperspectral imaging module and a stripe structured light three-dimensional imaging module. The snapshot-type hyperspectral imaging module consists of... Figure 3 The first module consists of three components: a two-dimensional grating 9, an imaging lens 10, and a snapshot-type hyperspectral imaging module camera 11. The second module consists of two components: a focusing lens 7 and a three-dimensional imaging camera 12, as shown in Figure 3. The two modules are integrated with a shared front optical path via a beam splitter 5, as shown in Figure 3. The shared front optical path includes four components: a zoom lens 1, a variable aperture 2, a filter 3, and a collimating lens 4, as shown in Figure 3, providing shared incident light control and collimation functions for the two modules.

[0028] The structured light module consists of a DLP projector (TJ50, resolution 1280×800 pixels) and a grayscale camera (MV-CA013-21UM), with a baseline distance of 70mm and a working distance of 300mm. The fringe projection employs a four-step phase-shifting method, acquiring four sinusoidal fringe images with phase shifts of 0°, 90°, 180°, and 270° respectively, and extracting the wrapping phase of each pixel.

[0029] To resolve the ambiguity in phase unwrapping order under complex curved surfaces, a three-frequency heterodyne method is employed. The selected fringe periods are T1=15 pixels, T2=16 pixels, and T3=17 pixels. The first-order heterodyne yields T12=240 pixels and T23=272 pixels, while the second-order heterodyne results in a final equivalent period T123=2040 pixels, exceeding the projector's resolution and achieving unambiguous unwrapping across the entire field of view. After obtaining the absolute phase, a lookup table method is used to pre-calculate the complex triangulation geometry. During online reconstruction, the depth value is directly obtained from the table and combined with camera intrinsic parameters for backprojection to obtain the complete 3D coordinates. This projector is responsible for both projecting structured light fringes and providing bright-field illumination.

[0030] During system calibration, a checkerboard calibration board was used. The calibration board was placed within the field of view, and images were simultaneously acquired using a CTIS module and a structured light camera. Sub-pixel precision checkerboard corner coordinates were extracted from the two images, and the homography matrix between the two image planes was calculated using the RANSAC algorithm. H：

[0031] From the matrix H The structure shows that the diagonal elements H 11 ≈ H 22 This indicates a scaling relationship of approximately 3.1 times between the two image planes. The off-diagonal elements are close to zero, indicating almost no rotational or shearing distortion between the two image planes, verifying the conjugate characteristics of the optical path design. The average registration error of the 88 corner points is 0.23 pixels, and the maximum error is 0.50 pixels, meeting the requirements for sub-pixel level registration.

[0032] To verify the performance of snapshot hyperspectral imaging, tests were conducted using typical macroscopic scene samples. Quantitative evaluation results show that the reconstructed PSNR of this test sample is as high as 35.89 dB, SSIM is 0.9869, and SAM is only 3.43°, verifying the effectiveness of the UNet3+ reconstruction network in macroscopic snapshot hyperspectral imaging tasks.

[0033] To verify the performance of the structured light 3D imaging system, a circular array calibration board was used for calibration. Multiple sets of calibration board images were acquired under different poses, and the camera intrinsic parameters were solved using Zhang Zhengyou's calibration method. Based on the calibration results, the focal length in the camera intrinsic parameter matrix was determined. Pixels Pixels, principal point coordinates Translation vector This represents the distance from the camera's optical center to the calibration origin, i.e., the working distance during calibration. The system uses these parameters to generate a lookup table during the calibration phase, and directly looks up the table to complete the phase-to-depth conversion during online reconstruction.

[0034] As attached Figure 4 As shown, the depth reconstruction accuracy is tested using a standard circular dot array plate. The residuals are calculated after least-squares plane fitting of the reconstructed point cloud: the standard deviation between the point cloud and the fitted plane is... The maximum deviation is .

[0035] After verifying the individual performance of the hyperspectral module and the structured light module, the overall performance of the four-dimensional joint imaging system was tested. The fused four-dimensional point cloud possesses both spatial geometric information and spectral attributes, and the four-dimensional data maintains good geometric consistency and spectral continuity in all spatial directions. A single three-dimensional scan takes approximately 0.22 seconds at a frame rate of 60fps, CTIS snapshot spectral acquisition takes approximately 0.08 seconds, and a single four-dimensional data acquisition can be completed in 0.3 seconds, demonstrating quasi-dynamic and even near real-time four-dimensional monitoring capabilities.

[0036] The experiment first used a completely bleached staghorn coral skeleton specimen for four-dimensional imaging. The sample surface exhibited a typical branching structure and porous textures of tiny coral polyps. Different pigments were applied to simulate different states. The system projected three-frequency, four-step phase-shifted fringes onto the coral surface, acquired images of the deformed fringes, calculated the absolute phase, and generated a dense point cloud model. The original point cloud underwent statistical filtering for noise reduction, and the neighborhood point count was used. Standard deviation multiple The parameter settings effectively eliminate outliers caused by surface reflection noise and edge scattering. Figure 5 The results of three-dimensional point cloud reconstruction of two staghorn coral skeleton specimens are presented. The reconstruction results show that the system successfully captured the complex three-dimensional morphological features of the coral surface, including the texture of the coral skeleton and the micropore structure of the coral polyps. The point cloud is evenly and densely distributed with clear edge contours.

[0037] Based on the homography matrix obtained during the system calibration phase H The system performs pixel-level mapping and fusion of the hyperspectral data cube reconstructed by the CTIS module and the three-dimensional spatial coordinates acquired by the structured light module. Figure 6 The results of fusion of four-dimensional imaging data of coral samples were shown, and the point cloud was given pseudo-color using the spectral signals of the RGB three channels.

[0038] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0039] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A three-dimensional spectral imaging method based on pixel-level registration of snapshot-type hyperspectral imaging and stripe structured light, characterized in that, Includes the following steps: The first step is to calibrate the image planes of the snapshot hyperspectral imaging module and the striped structured light 3D imaging module, calculate the homography transformation matrix between the two image planes, and achieve pixel-level registration. The second step involves using a snapshot hyperspectral imaging module to acquire a two-dimensional diffraction image of the target in a single exposure, and then reconstructing a hyperspectral data cube using a deep learning network. The third step involves using a striped structured light 3D imaging module to project a sinusoidal stripe pattern onto the target, acquiring deformed stripe images, calculating the absolute phase based on the phase-shifting method and the three-frequency heterodyne method, and reconstructing the 3D point cloud of the target. The fourth step involves fusing the spectral vectors in the hyperspectral data cube obtained in the second step with the spatial coordinates of the three-dimensional point cloud obtained in the third step, based on the homography matrix calibrated in the first step, to generate four-dimensional data points with three-dimensional spatial coordinates and one-dimensional spectral attributes.

2. The method according to claim 1, characterized in that, Quasi-dynamic four-dimensional data acquisition is achieved through rapid alternation between single structured light three-dimensional scanning and single snapshot-type hyperspectral acquisition; the total time for a single complete four-dimensional data acquisition is 0.3 seconds.

3. The method according to claim 1, characterized in that, In the third step, the three-dimensional imaging of striped structured light adopts the three-frequency heterodyne method, projects sinusoidal stripe patterns of different frequencies, and extracts the wrapped phase through the four-step phase shift method. The unambiguous absolute phase of the entire field of view is obtained through two-level heterodyne operation. The depth accuracy of the reconstructed three-dimensional point cloud is better than 30μm.

4. The method according to claim 1, characterized in that, In the first step, a checkerboard calibration board is used for calibration. Subpixel-precision corner coordinates are extracted from the imaging results of the two modules, and the homography transformation matrix is ​​calculated using the RANSAC algorithm.

5. The method according to claim 1, characterized in that, In the third step, the mapping from absolute phase to three-dimensional coordinates adopts a lookup table method. The mapping relationship from phase to depth is pre-calculated during the calibration stage. During online reconstruction, the depth value is directly obtained by looking up the table and combined with the camera intrinsic parameters for back projection to obtain the complete three-dimensional coordinates.

6. The method according to claim 1, characterized in that, The deep learning network in the second step is an end-to-end network used to reconstruct a hyperspectral data cube from a single frame of two-dimensional diffraction image.

7. The method according to claim 1, characterized in that, The fourth step specifically includes: using the inverse transformation of the homography matrix to convert the pixel coordinates on the image plane of the structured light camera into hyperspectral image plane coordinates, and then extracting the corresponding spectral vectors from the hyperspectral data cube and matching them with the three-dimensional spatial points.

8. The method according to claim 1, characterized in that, The method employs a system, the system comprising: The shared front optical path includes a zoom lens, a variable aperture, a filter, and a collimating lens, and guides the incident light to the hyperspectral imaging module and the 3D imaging module through a beam splitter. A snapshot-type hyperspectral imaging module, connected to the first optical path of the beam splitter, includes an imaging lens, a dispersive grating, and a first camera; The striped structured light 3D imaging module is connected to the second optical path of the beam splitter and includes a DLP projector, an imaging lens, and a grayscale camera. The DLP projector is also used to project a pure white image when the snapshot hyperspectral imaging module is working, providing broadband uniform illumination.

9. The method according to claim 1, characterized in that, The spectral coverage of the hyperspectral data cube is the visible light band.

10. The method according to claim 1, characterized in that, The four-dimensional data points simultaneously contain the target's spatial geometric information and spectral material information.

Images