Single-photon multi-mode structured light HDR three-dimensional imaging method and system
By combining single-photon cameras and ordinary cameras, and utilizing digital twins and transfer learning methods, the challenge of 3D reconstruction of single-photon avalanche diode arrays in complex scenes was solved, achieving efficient high dynamic range 3D imaging.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2025-09-17
- Publication Date
- 2026-05-05
AI Technical Summary
Existing technologies struggle to achieve high-precision 3D reconstruction in low photon flux and high dynamic range scenarios. The nonlinear response and resolution limitations of single-photon avalanche diode arrays lead to fringe phase errors, and there is a lack of a collaborative system between CMOS and SPAD under single-exposure conditions.
The single-photon multimodal structured light HDR 3D imaging method is adopted. By working together with a single-photon camera and a regular camera, digital twins and transfer learning are used to solve the problems of nonlinear error and data scarcity, and high dynamic range reconstruction under a single exposure is achieved.
It breaks through the limitations of dynamic range and photon sensitivity of traditional sensors under single-exposure conditions, realizes high-precision 3D reconstruction, and solves the bottleneck of 3D measurement efficiency and accuracy in complex scenes.
Smart Images
Figure CN121982175A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of optical three-dimensional imaging and machine vision technology, and in particular to a single-photon multimodal structured light HDR three-dimensional imaging method and system. Background Technology
[0002] Structured light 3D imaging (Fringe Projection Profilometry, FPP) has been widely applied in industrial inspection, medical diagnosis, and cultural relic preservation due to its advantages such as non-contact operation, high precision, and full-field measurement. However, traditional FPP systems, based on CMOS / CCD sensors, face challenges in two main scenarios: (a) Low photon flux scenarios, such as black surfaces. Insufficient photon count leads to extremely low signal-to-noise ratio (SNR), requiring traditional sensors to extend exposure time or increase projected light intensity, sacrificing real-time performance and easily introducing motion blur; (b) High dynamic range scenarios, such as mixed metal and plastic surfaces. High-reflectivity areas are overexposed while low-reflectivity areas are underexposed, necessitating multi-exposure fusion or adaptive intensity modulation, resulting in decreased measurement efficiency.
[0003] Existing HDR 3D imaging technologies mainly rely on traditional CMOS / CCD sensors, which expose significant shortcomings in complex scenes: multi-exposure HDR methods require continuous acquisition of multiple stripe images with different exposure times, which is not only inefficient but also extremely sensitive to moving objects, easily producing motion artifacts; adaptive intensity modulation technology relies on pre-calibrating the reflectivity distribution of the measured surface, making it difficult to adapt to unknown materials, while also having high hardware complexity and long calibration cycles; polarization technology can effectively suppress overexposure in highly reflective areas, but it is accompanied by a significant decrease in overall light intensity, leading to a sharp deterioration in the signal-to-noise ratio in low-reflectivity areas; although deep learning methods have powerful data-driven capabilities, their performance is heavily dependent on large-scale training data, resulting in poor generalization and significantly increased errors in extremely low-light or high-contrast scenes.
[0004] Existing technologies have long been limited to a single CMOS / CCD imaging mode, struggling to overcome the dual challenges of insufficient low photon sensitivity and limited high dynamic range. In recent years, single-photon avalanche diode (SPAD) arrays have emerged as a promising new direction due to their single-photon-level sensitivity, but three major challenges remain: First, the binary output of SPADs introduces significant nonlinear response, leading to phase errors in the fringes; second, limited by manufacturing processes, mainstream SPAD arrays have a resolution of only 512×512, significantly lower than CMOS, resulting in large direct calibration errors; third, there are currently no publicly reported systems that utilize a dual-mode SPAD and CMOS approach to achieve HDR 3D reconstruction under single-exposure conditions. Therefore, how to coordinate the single-photon sensitivity of SPADs with the high resolution of CMOS to simultaneously overcome nonlinear errors and fuse HDR information in a single exposure remains the core bottleneck for achieving high-precision 3D reconstruction of complex scenes. Summary of the Invention
[0005] The purpose of this invention is to overcome the core bottleneck in the existing technology for achieving high-precision 3D reconstruction of complex scenes, and to provide a single-photon multimodal structured light HDR 3D imaging method and system.
[0006] To address the aforementioned technical problems, the present invention provides the following technical solution: On the one hand, a single-photon multimodal structured light HDR three-dimensional imaging method is disclosed, including the following steps: S1: Control the projector to project a sinusoidal fringe pattern with N-step phase shift onto the object under test, where N≥3. The single-photon camera acquires a binary image cube corresponding to the sinusoidal fringe pattern, while the ordinary camera simultaneously acquires the sinusoidal fringe pattern to obtain a clear fringe pattern. S2: Convert the binary image cube into stripes, calculate the corresponding wrapping phase, input the sin and cos components of the wrapping phase into the trained neural network, and output the corrected phase map; S3: Unify the coordinate systems of the single-photon camera and the ordinary camera, and obtain the mapping relationship between the phase and three-dimensional coordinates of the single-photon camera and the ordinary camera under the unified coordinate system; S4: Multimodal 3D data fusion, removing invalid phase regions from the corrected phase map and the clear fringe map, and calculating the HDR 3D coordinates of the measured object in a unified coordinate system based on the mapping relationship between the phase and the 3D coordinates through the phase-3D coordinate mapping model.
[0007] By adopting the above technical solution, the single-photon sensitivity of the single-photon camera and the high resolution of the ordinary camera are combined under single exposure, breaking through the dual limitations of low photon count and high dynamic range. The nonlinear error and data scarcity of the stripes acquired by the single-photon camera are solved by digital twin and transfer learning, thereby realizing the unified coordinate system HDR reconstruction of the multimodal structured light system.
[0008] As a preferred embodiment of the present invention, the single-photon camera in step S1 continuously acquires a binary image cube corresponding to the sinusoidal stripe pattern in 1-bit binary mode.
[0009] As a preferred embodiment of the present invention, step S2 includes: calibrating the single-photon camera and the projector, obtaining relative spatial position parameters, generating fringe information under simulated single-photon images based on the relative spatial position parameters and the simulated three-dimensional model, and simultaneously using domain randomization technology to change the relative spatial position parameters to randomly generate a large amount of simulation data, and using the fringe information under simulated single-photon images and the simulation data to train the constructed neural network.
[0010] As a preferred embodiment of the present invention, the neural network is a Res-Unet network model.
[0011] As a preferred embodiment of the present invention, the single-photon camera and the projector are calibrated using the inverse camera calibration method.
[0012] As a preferred embodiment of the present invention, the stripe information under the simulated single-photon image is ordinary structured light stripe data, and the stripe information under the simulated single-photon image is converted into single-photon stripe images of different bits using a virtual photon image generation method.
[0013] As a preferred embodiment of the present invention, step S3 includes: S31: Establish the pixel-level phase-three-dimensional coordinate mapping relationship between the ordinary camera and the projector; S32: Using standard planar plates at different depths, based on orthogonal phase features, sub-pixel level corresponding point pairs are established between the field of view of a single-photon camera and the field of view of a regular camera; S33: Obtain the subpixel-level three-dimensional coordinates of the corresponding point pairs in the field of view of the single-photon camera through the subpixel-level corresponding point pairs and the mapping relationship; S34: Establish a single-photon phase coordinate mapping lookup table from the perspective of a single-photon camera based on an auxiliary phase coordinate mapping lookup table; S35: Construct the mapping relationship of the sub-pixel level three-dimensional coordinates under the field of view of the single-photon camera based on the auxiliary phase coordinate mapping lookup table and the single-photon phase coordinate mapping lookup table; S36: Using the standard planar plate and the singular value decomposition (SVD) algorithm, solve for the rotation matrix and translation vector of the coordinate systems of the single-photon camera and the ordinary camera; S37: Unify the coordinate systems of the single-photon camera and the ordinary camera according to the rotation matrix and the translation vector.
[0014] As a preferred embodiment of the present invention, step S32 includes: the orthogonal phase obtained by the ordinary camera is used as a reference, the orthogonal phase obtained by the single-photon camera is used as the target, and the corresponding points in the field of view of the ordinary camera are defined as pixel matching loss, calculated by the following formula: ; in, Represents the pixel coordinates of a standard camera; Represents the pixel coordinates of a single-photon camera; and The orthogonal phase acquired by a regular camera is used as a reference; and The orthogonal phase acquired by the single-photon camera is used as the target.
[0015] The orthogonal phase acquired by the single-photon camera is used as the target.
[0016] As a preferred embodiment of the present invention, step S34 includes: obtaining sub-pixel 3D coordinates based on an auxiliary phase coordinate mapping lookup table, correcting the error of the obtained sub-pixel 3D coordinates through plane fitting, and establishing a single-photon phase coordinate mapping lookup table.
[0017] On the other hand, a multimodal structured light system is disclosed, including a projector, a conventional camera, a single-photon camera, and a data processing system that is communicatively connected to the conventional camera and the single-photon camera; The projector is used to project a sinusoidal fringe pattern with N phase shifts onto the object being measured, where N ≥ 3; The single-photon camera is used to acquire a binary image cube corresponding to the sinusoidal fringe pattern; The ordinary camera is used to synchronously capture the sinusoidal stripe pattern to obtain a clear stripe image; The data processing system is used to execute the single-photon multimodal structured light HDR three-dimensional imaging method described above.
[0018] Compared with existing technologies, the advantages of this invention are as follows: This invention proposes a complete chain solution for single-photon multimodal structured light HDR 3D imaging, covering the entire process of system calibration, data fusion, and 3D reconstruction in complex "low-light-high-reflectivity" scenarios. It aims to fully leverage the complementary advantages of the single-photon sensitivity of a single-photon camera (SPAD camera) and the high resolution of a conventional camera (CMOS camera) to achieve HDR 3D measurement that can be completed in a single exposure. At the calibration level, this invention can establish a high-precision phase-3D coordinate mapping across single-photon cameras and conventional cameras using only a textureless standard plane. At the reconstruction level, through digital twin pre-training-transfer learning joint denoising and orthogonal phase sub-pixel alignment, it solves the problems of nonlinear phase error, data scarcity, and multimodal HDR fusion of single-photon cameras, breaking through the dual bottlenecks of dynamic range and photon sensitivity of traditional single sensors. Attached Figure Description
[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 This is a flowchart of a single-photon multimodal structured light HDR three-dimensional imaging method according to Embodiment 1 of the present invention; Figure 2 This is a flowchart illustrating the generation of a digital twin single-photon stripe image for a single-photon multimodal structured light HDR three-dimensional imaging method as described in Embodiment 2 of the present invention. Figure 3 This is a comparative experiment of a low-reflectivity scene of the single-photon multimodal structured light HDR three-dimensional imaging method described in Embodiment 2 of the present invention; Figure 4 This is a schematic diagram of the SPAD nonlinear response and phase error of a single-photon multimodal structured light HDR three-dimensional imaging method according to Embodiment 2 of the present invention; Figure 5 This is a flowchart of the multimodal HDR three-dimensional reconstruction process of a single-photon multimodal structured light HDR three-dimensional imaging method according to Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of the network prediction results of a single-photon multimodal structured light HDR three-dimensional imaging method according to Embodiment 2 of the present invention; Figure 7 This is a flowchart illustrating the multimodal coordinate system alignment of a single-photon multimodal structured light HDR three-dimensional imaging method according to Embodiment 2 of the present invention. Figure 8This is a reconstruction effect diagram of a complex material HDR multimodal subsystem of a single-photon multimodal structured light HDR three-dimensional imaging method according to Embodiment 2 of the present invention; Figure 9 This is a fusion effect image of complex material HDR multimodal 3D point cloud of a single-photon multimodal structured light HDR 3D imaging method according to Embodiment 2 of the present invention; Figure 10 This is a diagram showing the overall architecture of the multimodal structured light system described in Embodiment 3 of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0021] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between components or an indirect connection via other components.
[0022] Example 1 A single-photon multimodal structured light HDR 3D imaging method, such as Figure 1 As shown, it includes the following steps: S1: Control the projector to project a sinusoidal fringe pattern with N-step phase shift onto the object under test, where N≥3. The single-photon camera acquires a binary image cube corresponding to the sinusoidal fringe pattern, while the ordinary camera simultaneously acquires the sinusoidal fringe pattern to obtain a clear fringe pattern. S2: Convert the binary image cube into stripes, calculate the corresponding wrapping phase, input the sin and cos components of the wrapping phase into the trained neural network, and output the corrected phase map; S3: Unify the coordinate systems of the single-photon camera and the ordinary camera, and obtain the mapping relationship between the phase and three-dimensional coordinates of the single-photon camera and the ordinary camera under the unified coordinate system; S4: Multimodal 3D data fusion, removing invalid phase regions from the corrected phase map and the clear fringe map, and calculating the HDR 3D coordinates of the measured object in a unified coordinate system based on the mapping relationship between the phase and the 3D coordinates through the phase-3D coordinate mapping model.
[0023] Example 2 This embodiment is a preferred implementation of the single-photon multimodal structured light HDR three-dimensional imaging method shown in Embodiment 1; Preferably, the single-photon camera in step S1 continuously acquires a binary image cube corresponding to the sinusoidal fringe pattern in 1-bit binary mode.
[0024] Preferably, step S2 includes: calibrating the single-photon camera and the projector, obtaining relative spatial position parameters, generating fringe information under simulated single-photon images based on the relative spatial position parameters and the simulated three-dimensional model, and simultaneously using domain randomization technology to change the relative spatial position parameters to randomly generate a large amount of simulation data, and using the fringe information under simulated single-photon images and the simulation data to train the constructed neural network.
[0025] The fringe information under the simulated single-photon image is ordinary structured light fringe data. A virtual photon image generation method is used to convert the fringe information under the simulated single-photon image into single-photon fringe images of different bit depths. This embodiment discloses... Figure 2 The single-photon simulated stripe image generation framework is shown.
[0026] To further enhance generalization, an SC-SC learning strategy is introduced. The input is a low-bit, two-channel floating-point type representing the sin and cos components of the wrapped phase, and the ground truth is a high-bit, two-channel floating-point type representing the sin and cos components of the wrapped phase. In this embodiment, the Res-Unet network is used for training. Subsequently, real data is collected using a single-photon camera-projector system, and the model data trained in simulation is used for fine-tuning.
[0027] Furthermore, the 3D model is derived from 800 models in the Thingi10K: A Dataset of 10,000 3D-PrintingModels dataset. The relative spatial parameters, 3D model, and stripe pattern obtained from the input calibration are used. In order to reduce the difference between virtual data and real data, a domain randomization strategy is adopted to randomly generate the 6D pose of the object, the color of the object, the roughness of the object, the distortion coefficient of the camera, the projection ratio of the projector, the projection intensity of the projector, and the 6D pose of the projector within a certain range to construct domain random data.
[0028] like Figure 3 The low-reflectivity scene comparison experiment shown demonstrates the image comparison between a single-photon camera and a regular camera in a low-reflectivity scene.
[0029] Furthermore, regarding the error problem of single-photon fringes, this embodiment discloses as follows: Figure 4 The error shown is due to the fact that this embodiment first calibrates the single-photon camera-projector system. Although the projector in the case has the influence of a cylindrical lens, which does not fully meet the assumptions of the pinhole model, this operation is only to roughly obtain the relative spatial parameters. The cylindrical lens in this embodiment is only used in this embodiment. In the actual process, it may be omitted depending on the situation. The calibration method adopted for the single-photon camera-projector system is the inverse camera calibration method. The absolute phase adopts the complementary Gray code method and uses 30-step phase shift.
[0030] The results of network prediction are as follows Figure 6 As shown, where, Figure 6 (a) Training directly on the real dataset; Figure 6 (b) Predictions after transferring the learning network; Figure 6 (c) Truth value.
[0031] Preferably, step S3 includes: S31: Establish the pixel-level phase-three-dimensional coordinate mapping relationship between the ordinary camera and the projector; S32: Using standard planar plates at different depths, based on orthogonal phase features, sub-pixel level corresponding point pairs are established between the field of view of a single-photon camera and the field of view of a regular camera; Preferably, a standard plane is used and placed in the common field of view of the ordinary camera-projector system and the single-photon camera-projector system, and the marker plane is positioned almost perpendicular to the optical axis of the single-photon camera in different orientations; Preferably, step S32 includes: Figure 7 As shown in (a), the orthogonal phase obtained by the ordinary camera is used as a reference, and the orthogonal phase obtained by the single-photon camera is used as the target. The corresponding points in the field of view of the ordinary camera are defined as pixel matching loss, and the calculation formula is as follows: ; in, Represents the pixel coordinates of a standard camera; Represents the pixel coordinates of a single-photon camera; and The orthogonal phase acquired by a regular camera is used as a reference; and The orthogonal phase acquired by the single-photon camera is used as the target.
[0032] After completing this step, some integer two-dimensional coordinates (anchor pixels) are obtained. Next, in order to improve accuracy, it is necessary to find the coordinates of the point with a phase difference of 0, rather than the coordinates of the point with the smallest phase difference. By utilizing the monotonicity of the phase, the sub-pixel coordinates are solved in reverse through spline interpolation. At present, the first stage of view transformation has been completed. S33: Obtain the sub-pixel level three-dimensional coordinates of the corresponding point pairs in the field of view of the single-photon camera through the sub-pixel level corresponding point pairs and the mapping relationship, that is, the phase of the single-photon camera; S34: Establish a single-photon phase coordinate mapping lookup table from the perspective of a single-photon camera based on an auxiliary phase coordinate mapping lookup table; Preferably, step S34 includes: obtaining sub-pixel 3D coordinates based on an auxiliary phase coordinate mapping lookup table, correcting the error of the obtained sub-pixel 3D coordinates through plane fitting, and establishing a single-photon phase coordinate mapping lookup table. The auxiliary phase coordinate mapping lookup table and the single-photon phase coordinate mapping lookup table are established based on the mapping relationship between phase and 3D coordinates.
[0033] Furthermore, to avoid directly calibrating low-resolution single-photon cameras and to establish a phase coordinate mapping lookup table under the single-photon camera's viewpoint, the ordinary camera-projector system is first calibrated to construct a pixel-level phase-three-dimensional coordinate mapping relationship. Taking advantage of the high resolution of ordinary cameras, this embodiment does not have high requirements for the calibration of the ordinary camera-projector system. As long as a pixel-level phase-three-dimensional coordinate mapping relationship can be constructed, it is acceptable regardless of the pixel-level calibration method used. S35: Construct the mapping relationship of the sub-pixel level three-dimensional coordinates under the field of view of the single-photon camera based on the auxiliary phase coordinate mapping lookup table and the single-photon phase coordinate mapping lookup table; Subpixel coordinates and auxiliary phase (i.e., the phase of a regular camera) to 3D mapping are used to obtain subpixel 3D coordinates. Considering the reconstruction error, a plane fitting method is used to eliminate the error. The plane equation can be expressed as: ; in , , and These are the parameters of the fitted plane. Ideal coordinates. This can be expressed as, ; After completing the above, the next step is to establish the relationship between the corrected 3D coordinates of each camera pixel and the phase map, such as... Figure 7 As shown in (b), this relationship can be calculated using the principle of cross-ratio invariance in photographic geometry, through the following equation: ; in , ,and It is a constant term for each pixel, and This represents the phase after expansion. For and Coordinates, and Establish a linear relationship between coordinates, that is: ;in , , and These are the coefficients of a linear function for each pixel. The unknown parameters in the above equations can be solved using the least squares method. At this point, the second stage of the single-photon phase coordinate mapping lookup table has been completed, and the single-photon camera-projector system has 3D reconstruction capabilities.
[0034] S36: Using the standard planar plate and the singular value decomposition (SVD) algorithm, solve for the rotation matrix and translation vector of the coordinate systems of the single-photon camera and the ordinary camera; S37: Unify the coordinate systems of the single-photon camera and the ordinary camera according to the rotation matrix and the translation vector.
[0035] Furthermore, step S4 includes: repeating the process in multiple poses (for example, five poses are used in this embodiment), as disclosed in this embodiment. Figure 7 As shown in (c), integer 3D points are calculated using single-photon and auxiliary phase-to-3D mapping. As previously mentioned, sub-pixel points have been found to achieve a phase difference of 0. Subsequently, the integer 3D points in the ordinary camera are converted into sub-pixel 3D points. By utilizing these corresponding sub-pixel 3D points, and with the aid of singular value decomposition (SVD), the rotation (R) and translation (T) relationship between the two camera coordinate systems is determined through least-squares rigid transformation. In fact, to reduce the influence of measurement errors, more spatial points can be obtained to optimize R&T (for example, this embodiment uses 140,000 corresponding points). Thus, the rigid transformation calculation of the third stage is completed. This embodiment discloses as follows... Figure 5 The illustrated 3D reconstruction process performs 3D reconstruction of the target object. In this example, a PCB board was used for 3D reconstruction, as disclosed below. Figure 8The three-dimensional reconstruction results shown in this embodiment also account for the differences in the three-dimensional reconstruction results caused by the absence of a third stage, as disclosed below. Figure 9 The three-dimensional reconstruction results are shown.
[0036] Example 3 A multimodal structured light system, such as Figure 10 As shown, it includes a projector, a conventional camera, a single-photon camera, and a data processing system that is communicatively connected to the conventional camera and the single-photon camera; The projector is used to project a sinusoidal fringe pattern with N phase shifts onto the object being measured, where N ≥ 3; The single-photon camera is used to acquire a binary image cube corresponding to the sinusoidal fringe pattern; The ordinary camera is used to synchronously capture the sinusoidal stripe pattern to obtain a clear stripe image; The data processing system is used to execute the single-photon multimodal structured light HDR three-dimensional imaging method described in either Example 1 or Example 2.
[0037] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.
[0038] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. A single-photon multimodal structured light HDR three-dimensional imaging method, characterized in that, Includes the following steps: S1: Control the projector to project a sinusoidal fringe pattern with N-step phase shift onto the object under test, where N≥3. The single-photon camera acquires a binary image cube corresponding to the sinusoidal fringe pattern, while the ordinary camera simultaneously acquires the sinusoidal fringe pattern to obtain a clear fringe pattern. S2: Convert the binary image cube into stripes, calculate the corresponding wrapping phase, input the sin and cos components of the wrapping phase into the trained neural network, and output the corrected phase map; S3: Unify the coordinate systems of the single-photon camera and the ordinary camera, and obtain the mapping relationship between the phase and three-dimensional coordinates of the single-photon camera and the ordinary camera under the unified coordinate system; S4: Multimodal 3D data fusion, removing invalid phase regions from the corrected phase map and the clear fringe map, and calculating the HDR 3D coordinates of the measured object in a unified coordinate system based on the mapping relationship between the phase and the 3D coordinates through the phase-3D coordinate mapping model.
2. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 1, characterized in that, The single-photon camera described in step S1 continuously acquires binary image cubes corresponding to the sinusoidal stripe pattern in 1-bit binary mode.
3. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 1, characterized in that, Step S2 includes: calibrating the single-photon camera and the projector, obtaining relative spatial position parameters, generating fringe information under simulated single-photon images based on the relative spatial position parameters and the simulated three-dimensional model, simultaneously changing the relative spatial position parameters using domain randomization technology to randomly generate a large amount of simulation data, and training the constructed neural network using the fringe information under simulated single-photon images and the simulation data.
4. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 3, characterized in that, The neural network is a Res-Unet network model.
5. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 3, characterized in that, The single-photon camera and the projector are calibrated using the inverse camera calibration method.
6. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 3, characterized in that, The stripe information under the simulated single-photon image is ordinary structured light stripe data. The stripe information under the simulated single-photon image is converted into single-photon stripe images of different bits using a virtual photon image generation method.
7. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 1, characterized in that, Step S3 includes: S31: Establish the pixel-level phase-three-dimensional coordinate mapping relationship between the ordinary camera and the projector; S32: Using standard planar plates at different depths, based on orthogonal phase features, sub-pixel level corresponding point pairs are established between the field of view of a single-photon camera and the field of view of a regular camera; S33: Obtain the subpixel-level three-dimensional coordinates of the corresponding point pairs in the field of view of the single-photon camera through the subpixel-level corresponding point pairs and the mapping relationship; S34: Establish a single-photon phase coordinate mapping lookup table from the perspective of a single-photon camera based on an auxiliary phase coordinate mapping lookup table; S35: Construct the mapping relationship of the sub-pixel level three-dimensional coordinates under the field of view of the single-photon camera based on the auxiliary phase coordinate mapping lookup table and the single-photon phase coordinate mapping lookup table; S36: Using the standard planar plate and the singular value decomposition (SVD) algorithm, solve for the rotation matrix and translation vector of the coordinate systems of the single-photon camera and the ordinary camera; S37: Unify the coordinate systems of the single-photon camera and the ordinary camera according to the rotation matrix and the translation vector.
8. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 7, characterized in that, Step S32 includes: the orthogonal phase acquired by the ordinary camera is used as a reference, the orthogonal phase acquired by the single-photon camera is used as the target, and the corresponding points in the field of view of the ordinary camera are defined as pixel matching loss, calculated as follows: ; in, Represents the pixel coordinates of a standard camera; Represents the pixel coordinates of a single-photon camera; and The orthogonal phase acquired by a regular camera is used as a reference; and The orthogonal phase acquired by the single-photon camera is used as the target.
9. The single-photon multimodal structured light HDR three-dimensional imaging method according to claim 7, characterized in that, Step S34 includes: obtaining sub-pixel 3D coordinates based on an auxiliary phase coordinate mapping lookup table, correcting the error of the obtained sub-pixel 3D coordinates through plane fitting, and establishing a single-photon phase coordinate mapping lookup table.
10. A multimodal structured light system, characterized in that, It includes a projector, a conventional camera, a single-photon camera, and a data processing system that is communicatively connected to the conventional camera and the single-photon camera; The projector is used to project a sinusoidal fringe pattern with N phase shifts onto the object being measured, where N ≥ 3; The single-photon camera is used to acquire a binary image cube corresponding to the sinusoidal fringe pattern; The ordinary camera is used to synchronously acquire the sinusoidal fringe pattern to obtain a clear fringe image; the data processing system is used to execute the single-photon multimodal structured light HDR three-dimensional imaging method according to any one of claims 1 to 9.