High-precision event camera-projector system calibration and 3d measurement method based on projector intrinsical parameters and per-pixel calibration

Abstract

The application relates to the technical field of single-pixel imaging, and discloses a high-precision event camera-projector system calibration and three-dimensional measurement method for projector intrinsic single-pixel imaging, the calibration stage comprises the following steps: collecting an event stream of a projection pattern by using an event camera to realize event camera calibration, adopting Hadamard single-pixel imaging technology, projecting a Hadamard base pattern to a calibration board, collecting light intensity by using a photoelectric detector, reconstructing a calibration board image to realize calibration of projector intrinsic imaging, and obtaining calibration parameters; the measurement stage comprises the following steps: decomposing a sinusoidal gray scale fringe into a binary bit fringe sequence and projecting, synchronously collecting the binary bit fringe by using an event camera, carrying out binary processing on the collected projection binary bit fringe, and weightedly synthesizing a gray scale fringe, then obtaining an absolute phase diagram, and realizing three-dimensional reconstruction in combination with the calibration parameters; and the application improves the calibration precision of the event camera-projector system.

Description

Technical Field

[0001] This invention relates to the field of single-pixel imaging technology, and more specifically to a high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local perspective. Background Technology

[0002] 3D reconstruction technology is a technique that acquires the 3D shape information of an object through non-contact or contact methods, and then uses computer analysis for application in related fields. Structured light method involves projecting light according to specific rules, capturing the light pattern after it modulates and deforms on the object's surface, and obtaining the 3D shape information of the object's surface based on the deformation information of the light pattern. This method not only has high accuracy but also low cost, simple principle, and ease of implementation, making it a current research hotspot in 3D measurement technology. Among these methods, fringe projection-based optical measurement technology is the most popular, thanks to its high accuracy, high speed, and irreplaceable advantages in terms of wide applicability. This method projects a fringe pattern with modulation information onto the object's surface, uses a camera to capture the fringe deformation information, and then calculates the object's 3D information using relevant algorithms. With these characteristics and advantages, fringe projection technology has a unique competitive edge.

[0003] However, traditional structured light systems suffer from data redundancy. Images captured by frame-based cameras contain a large amount of redundant information, especially in point and line structured light systems. This redundancy degrades the system's acquisition speed and overall efficiency. Furthermore, traditional frame cameras have low dynamic range, capturing only a limited range of brightness levels. When acquiring images of target surfaces (such as glossy paint or plastic surfaces), this often leads to overexposure or underexposure, resulting in degraded reconstruction quality or even reconstruction failure. Existing solutions typically employ multi-exposure fusion and multi-projection fusion techniques, but these methods require extensive manual adjustments to exposure settings and projection intensity to achieve satisfactory results. Event cameras, with their advantages of sparse data, low latency, low power consumption, asynchronous sensing, and high dynamic range, effectively compensate for the shortcomings of industrial cameras. Additionally, some existing research on 3D reconstruction using event cameras, such as methods based on laser point projectors, has achieved a certain scanning speed and high recovery accuracy. They can perform real-time dense target depth estimation within a relatively low range; however, these methods are limited by the low encoding efficiency and scanning rate of the projector, making it difficult to achieve higher-speed depth estimation. In other studies utilizing digital light processing (DLP) projectors for faster depth estimation, previous research on the calibration of structured light systems using event cameras employed deep learning algorithms to convert events into images and then used calibration tools for coarse extrinsic parameter calibration. However, these methods are complex and inconvenient to use, and the images generated by deep learning algorithms may not maintain sub-pixel realism. Although some studies have achieved high-precision calibration, these methods are only applicable to laser point projectors or event cameras with grayscale modes. In addition, some research has proposed a sub-pixel-level, simple, and universal calibration scheme for event camera-based structured light systems. This scheme is simple to calibrate but has lower calibration accuracy.

[0004] Structured light measurement, as a mature technology for industrial-grade 3D vision, still presents a key challenge for traditional frame camera-based systems in 3D reconstruction under complex lighting conditions, especially in high dynamic range (HDR) scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a high-precision calibration and 3D measurement method for event camera-projector systems using single-pixel imaging from the projector's own perspective. This invention overcomes the shortcomings of low 3D reconstruction quality in high-dynamic scenes by industrial cameras, while simultaneously employing single-pixel imaging technology to significantly improve the calibration accuracy of event camera-projector systems.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] A high-precision event camera-projector system calibration and 3D measurement method for projector-based single-pixel imaging from its own perspective includes a calibration stage and a measurement stage. The calibration stage includes: acquiring an event stream of a projected pattern using an event camera to calibrate the event camera; employing Hadamard single-pixel imaging technology, projecting a Hadamard substrate image onto a calibration board and acquiring light intensity using a photodetector to reconstruct the calibration board image to calibrate the projector's own-viewpoint imaging, thereby obtaining calibration parameters. The measurement stage includes: decomposing sinusoidal grayscale fringes into a binary fringe sequence and projecting it; synchronously acquiring the sequence with the event camera; binarizing the acquired projected binary fringe and weighting it to synthesize grayscale fringes to obtain an absolute phase map; and combining this with the calibration parameters to achieve 3D reconstruction.

[0008] In one embodiment, the step of using an event camera to acquire an event stream of a projected pattern for event camera calibration specifically includes: The black and white images are cyclically projected onto the calibration board to trigger events by the event camera. The event camera collects the event stream and distinguishes between positive and negative events based on the projection time. The event frames in the positive and negative event streams, which are composed of positive and negative events respectively, can be used as calibration images to calibrate the event camera.

[0009] In one embodiment, the use of Hadamard single-pixel imaging technology, which involves projecting a Hadamard substrate image onto a calibration board and collecting light intensity using a photodetector to reconstruct the calibration board image to achieve calibration of the projector's local viewing angle imaging, specifically includes: Hadamard single-pixel imaging is based on the Hadamard transform, and the Hadamard transform and inverse Hadamard transform are defined as follows: ; ; in,( , ) and( , ) represent spatial coordinates and Hadama field coordinates, respectively. ) represents the Hadama spectrum. and These represent the Hadamard transform and the inverse Hadamard transform, respectively, with M representing the horizontal resolution or the vertical resolution. It is the Hadamard matrix. Represents the reconstructed image of the object; A Hadamard base map is generated based on the Hadamard matrix. The Hadamard base map is projected onto the calibration plate. A photodetector is used to collect the light intensity. The calibration plate image is reconstructed through the inverse Hadamard transform to achieve the calibration of the projector's local viewpoint imaging.

[0010] In one embodiment, the step of decomposing the sinusoidal grayscale stripes into a binary stripe sequence and projecting it specifically includes: The 8-bit sinusoidal grayscale fringe with a grayscale range of 0-255 is decomposed into 8 binary fringe. The generated binary fringe is projected onto the target object and then synchronously acquired by the event camera to obtain the projected binary fringe.

[0011] In one embodiment, the binarization processing of the acquired projected binary stripes specifically includes: The difference matrix of complementary projected binary stripes is judged. Pixels with a value greater than 0 are set to 1, and those with a value less than 0 are set to 0, thus obtaining binary stripes.

[0012] In one embodiment, the weighted composite grayscale stripes specifically include: The grayscale stripes are synthesized by weighted summation of the obtained 8-bit binary stripes according to their corresponding bit weights. Each binary stripe is multiplied by a different power of 2, with the power exponent ranging from 0 to 7.

[0013] In one embodiment, obtaining the absolute phase map specifically includes: Based on the principle of multi-frequency heterodyne, phase unfolding is performed on the synthesized grayscale stripes of different periods to obtain an absolute phase map.

[0014] Compared with the prior art, the beneficial technical effects of the present invention are: This invention leverages the high dynamic range, asynchronous response, high temporal resolution, and low data redundancy of event cameras to effectively overcome the limitations of traditional cameras in HDR scenarios. Addressing the calibration accuracy bottleneck in event camera-projector systems caused by the lack of traditional grayscale images, the core innovation of this invention lies in: using single-pixel imaging technology to endow the projector with active imaging capabilities from its own "local perspective," enabling the projector to directly observe the calibration target and obtain high-precision calibration parameters (including projector intrinsic parameters and system structural parameters), thereby overcoming the limitations of low accuracy and poor robustness in existing calibration methods.

[0015] In the 3D measurement phase, this invention employs an encoding and decoding method perfectly adapted to the binary response characteristics of an event camera. This method decomposes standard sinusoidal grayscale fringes into a series of binary fringes. By projecting complementary binary patterns and utilizing the response differences of the event camera, an equivalent grayscale fringe containing high-precision phase information is reconstructed. Phase unfolding is then performed using the multi-frequency heterodyne principle to obtain an absolute phase map, which is finally fused with the aforementioned high-precision calibration results to achieve high-precision 3D reconstruction in HDR scenes.

[0016] In summary, this invention solves the core calibration problem of the system by using "projector-based single-pixel imaging from its own perspective" and fully leverages the HDR advantages of the event camera by combining "binary stripe encoding and decoding", thus forming a complete and high-precision HDR 3D measurement solution. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the event camera-projector calibration system of the present invention; Figure 3 This is a schematic diagram of the event camera calibration of the present invention; Figure 4 This is a schematic diagram of the projector calibration of the present invention; Figure 5 This is a schematic diagram of the method framework of the present invention. Detailed Implementation

[0018] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] like Figure 1 As shown, the present invention provides a high-precision event camera-projector system calibration and 3D measurement method for single-pixel imaging from the projector's local viewing angle, comprising: S1, Calibration Stage: The event camera is used to collect the event stream of the projected pattern to achieve event camera calibration. Hadamard single-pixel imaging technology is used to project the Hadamard substrate image onto the calibration board and collect the light intensity by the photodetector to reconstruct the calibration board image to achieve the calibration of the projector's local viewpoint imaging and obtain calibration parameters. S2, Measurement stage: The sinusoidal grayscale stripes are decomposed into a binary stripe sequence and projected. After being synchronously acquired by the event camera, the acquired projected binary stripes are binarized and weighted to synthesize grayscale stripes, thereby obtaining an absolute phase map. Combined with the calibration parameters, three-dimensional reconstruction is achieved.

[0020] The technical solution of the present invention will be described in detail below in several parts.

[0021] 1. Calibration of the event camera-projector.

[0022] (1) Event camera calibration: The calibration board is cyclically projected with black and white images (all zeros and all ones) to achieve blinking, which triggers the event camera. The event camera collects the event stream and distinguishes between positive and negative events based on the projection time (positive and negative events refer to events caused by the increase and decrease of light intensity, respectively). The event frames accumulated from the positive and negative event streams can be used as calibration images.

[0023] (2) Projector calibration: Hadamard single-pixel imaging technology is used. Hadamard Single-pixel Imaging (HSI) is a single-pixel imaging method based on the Hadamard transform and orthogonal basis transformation and spectral acquisition. Hadamard single-pixel imaging obtains the Hadamard spectrum of an object by projecting a Hadamard basis pattern, and then reconstructs the image of the object through the inverse Hadamard transform. The two-dimensional Hadamard transform and the inverse Hadamard transform are defined as follows: ; ; in,( , ) and( , ) represent spatial coordinates and Hadama field coordinates, respectively. ) represents the Hadama spectrum. and These represent the Hadamard transform and the inverse Hadamard transform, respectively, with M representing the horizontal resolution or the vertical resolution. It is a Hadamard matrix.

[0024] Operation steps: Generate a Hadamard substrate map, project the substrate map onto the calibration plate, collect light intensity using a single-pixel detector (photodetector), and recover the image through Hadamard inverse transform.

[0025] In one embodiment, the single-pixel imaging technology used for projector calibration can be changed from Hadamard single-pixel imaging to Fourier single-pixel imaging technology, using Fourier three-step phase shift and two-step phase shift to complete the modulation process of the target object.

[0026] Event camera-projector calibration as follows Figure 2 As shown, the experimental setup consists of an event camera, a calibration board, a photodetector, and a projection device.

[0027] Projection devices can use various spatial light modulators such as projectors, LCDs, and digital micromirror devices (DMDs) to modulate the projected pattern.

[0028] Due to the memory limitations of the DMD itself, only a sampling rate of 0.3 can be used to achieve single-pixel imaging. A DMD device with larger memory can be used to achieve single-pixel imaging with a higher sampling rate, thereby achieving higher calibration accuracy.

[0029] Figure 3 This represents the event frames captured by the event camera and the reprojection error, with a cumulative time of 10ms and an average reprojection error of [0.079, 0.088]. Figure 4 It uses a resolution of The Hadamard base map and reprojection error were obtained, with the sampling rate set to 0.3, and the average reprojection error of the projector was [0.029, 0.027].

[0030] 2. Generation of binary bit stripe coding patterns.

[0031] An 8-bit sinusoidal grayscale fringe with a grayscale range of 0-255 is decomposed into 8 binary fringe bits. Specifically, the generated sinusoidal grayscale fringe is divided successively by different powers of 2 and the integer part is taken. Then, the result is modulo 2 to extract the corresponding binary bit information, where the power exponent is from 0 to 7.

[0032] 3. Binarization.

[0033] The specific steps are as follows: the generated binary stripes are projected onto the target object, and then the event camera synchronously acquires the data, and the acquired modulated image (projected binary stripes) is binarized.

[0034] The principle of binarization of projected binary fringe is as follows: In the projected four-step phase-shifted binary fringe pattern sequence, the first step 8-bit fringe and the third step 8-bit fringe are complementary, and the second step 8-bit fringe and the fourth step 8-bit fringe are complementary. Since they are completely complementary binary fringe, when projecting the same area, the pixel value of the bright fringe is always larger than the pixel value of the dark fringe. The difference matrix of complementary fringe can be calculated, and the elements greater than 0 are set to 1, and the others are set to 0. Binarization is then performed based on this.

[0035] Regarding the number of projected binary fringe patterns: Based on the four-step phase-shift algorithm, four groups are generated, with eight binary fringe patterns in each group. Therefore, 32 binary fringe modulation patterns are needed to generate one periodic fringe pattern. Since the event camera can only detect dynamic objects, a completely black image needs to be added after each image to create a dynamic effect. A total of 192 images are needed for the three-frequency four-step fringe projection.

[0036] 4. Synthesis of grayscale stripes.

[0037] After obtaining the binary modulation information, a phase image with modulation information is synthesized. Specifically, the grayscale stripes corresponding to each step are synthesized by weighted summation of the obtained 8-bit binary stripes according to their bit weights. Each binary stripe is multiplied by a different power of 2, with the power exponent ranging from 0 to 7. Through this weighted reconstruction process, the scattered binary modulation information can be synthesized into grayscale stripes with complete modulation information.

[0038] 5. Acquisition of absolute phase and generation of 3D point cloud.

[0039] Following the above synthesis steps, the three-frequency, four-step fringes are synthesized separately and then converted into absolute phase using the phase expansion formula. This phase is then combined with calibration parameters to achieve 3D reconstruction. Calibration parameters include intrinsic matrix, extrinsic matrix, and distortion coefficients.

[0040] like Figure 5 As shown, the event frame of the projected stripes is acquired using an event camera, and then the acquired image is binarized. Subsequently, the 8-bit binarized stripes are weighted and synthesized into grayscale stripes. Since a three-frequency four-step phase unfolding method is used, it is necessary to acquire stripes with three different periods of T=256, 512, and 1024 to unfold into absolute phase. Finally, the three-dimensional reconstruction is achieved by combining the calibration parameters.

[0041] This invention uses an event camera instead of an industrial camera, taking advantage of the high dynamic range of the event camera, and combines single-pixel imaging technology and binary stripe coding and decoding technology to achieve 3D reconstruction under high dynamic range (HDR) conditions.

[0042] This invention applies single-pixel imaging technology to an event camera-projector system, which greatly improves the calibration accuracy of the event camera-projector system.

[0043] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0044] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0045] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0046] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0047] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for calibrating and measuring a high-precision event camera-projector system using single-pixel imaging from the projector's local viewing angle, characterized in that... The system includes a calibration phase and a measurement phase. The calibration phase includes: using an event camera to acquire the event stream of the projected pattern to calibrate the event camera; and employing Hadamard single-pixel imaging technology to project a Hadamard substrate image onto a calibration board and collect light intensity using a photodetector to reconstruct the calibration board image to calibrate the projector's local viewpoint imaging and obtain calibration parameters. The measurement phase includes: decomposing sinusoidal grayscale fringes into a binary fringe sequence and projecting it; after synchronous acquisition by the event camera, binarizing the acquired projected binary fringe and weighting and synthesizing grayscale fringes to obtain an absolute phase map; and combining the calibration parameters to achieve three-dimensional reconstruction.

2. The high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local viewing angle, as described in claim 1, is characterized in that... The event camera calibration process, which utilizes an event camera to capture the event stream of the projected pattern, specifically includes: The black and white images are cyclically projected onto the calibration board, causing the event camera to trigger events. The event camera collects the event stream and distinguishes between positive and negative events based on the projection time. The event frames accumulated by the positive and negative event streams can be used as calibration images to calibrate the event camera.

3. The high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local viewing angle according to claim 1, characterized in that, The method employs Hadamard single-pixel imaging technology, which involves projecting a Hadamard substrate image onto a calibration board and collecting light intensity using a photodetector to reconstruct the calibration board image, thereby achieving calibration of the projector's local viewing angle imaging. Specifically, this includes: Hadamard single-pixel imaging is based on the Hadamard transform, and the Hadamard transform and inverse Hadamard transform are defined as follows: ； ； in,( , ) and( , ) represent spatial coordinates and Hadama field coordinates, respectively. ) represents the Hadama spectrum. and These represent the Hadamard transform and the inverse Hadamard transform, respectively, with M representing the horizontal resolution or the vertical resolution. It is the Hadamard matrix. Represents the reconstructed image of the object; A Hadamard base map is generated based on the Hadamard matrix. The Hadamard base map is projected onto the calibration plate. A photodetector is used to collect the light intensity. The calibration plate image is reconstructed through the inverse Hadamard transform to achieve the calibration of the projector's local viewpoint imaging.

4. The high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local viewing angle, as described in claim 1, is characterized in that... The process of decomposing sinusoidal grayscale stripes into a binary stripe sequence and projecting it specifically includes: The 8-bit sinusoidal grayscale fringe with a grayscale range of 0-255 is decomposed into 8 binary fringe. The generated binary fringe is projected onto the target object and then synchronously acquired by the event camera to obtain the projected binary fringe.

5. The high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local viewing angle according to claim 1, characterized in that, The binarization process for the acquired projected binary stripes specifically includes: The difference matrix of complementary projected binary stripes is judged. Pixels with a value greater than 0 are set to 1, and those with a value less than 0 are set to 0, thus obtaining binary stripes.

6. The high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local viewing angle according to claim 1, characterized in that, The weighted composite grayscale stripes specifically include: The grayscale stripes are synthesized by weighted summation of the obtained 8-bit binary stripes according to their corresponding bit weights. Each binary stripe is multiplied by a different power of 2, with the power exponent ranging from 0 to 7.

7. The high-precision event camera-projector system calibration and three-dimensional measurement method for single-pixel imaging from the projector's local viewing angle according to claim 1, characterized in that, The acquisition of the absolute phase map specifically includes: Based on the principle of multi-frequency heterodyne, phase unfolding is performed on the synthesized grayscale stripes of different periods to obtain an absolute phase map.

Images