3D imaging method and system based on hybrid event camera

Abstract

The present disclosure provides a 3D imaging method and system based on a hybrid event camera, the method comprising: projecting a first set of coded structured light patterns onto an object to be measured; collecting an event stream of the structured light patterns on the surface of the object to be measured; determining a first phase of the structured light stripes corresponding to each effective pixel on the imaging plane in the structured light pattern on the surface of the object to be measured based on the event stream; projecting a second set of coded structured light patterns onto the object to be measured; collecting a first image of the structured light pattern on the surface of the object to be measured; determining a second phase of each effective pixel on the imaging plane in the corresponding structured light stripes based on brightness information of the first image; determining a third phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object to be measured based on the first phase and the second phase; and performing three-dimensional reconstruction of the object to be measured based on the third phase and calibration parameters of the hybrid event camera. The 3D imaging method and system based on a hybrid event camera provided by the present disclosure can achieve high-speed, high-precision, and high-dynamic-range 3D imaging.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, to a 3D imaging method and system based on a hybrid event camera. Background Art

[0002] Compared with 2D vision, 3D vision systems can obtain accurate three-dimensional information of the target, bringing unprecedented efficiency and accuracy to the detection field, thus opening up a new path for the development of other industries.

[0003] Among 3D vision solutions, time-encoded structured light (TLE) has become the most widely used approach for close-range 3D reconstruction due to its high measurement accuracy, contactless operation, and robustness. However, due to bandwidth limitations and other factors, TLE-based 3D cameras based on traditional frame cameras currently face three trade-offs in practical applications: reconstruction speed and accuracy, reconstruction speed and ambient light immunity, and reconstruction speed and dynamic range. These trade-offs force 3D camera systems to strike a balance between reconstruction speed and quality, and also hinder their use in scenes with strong ambient light, such as those outdoors. Summary of the Invention

[0004] The present invention aims to provide a 3D imaging method and system based on a hybrid event camera to achieve high-speed, high-precision, and high-dynamic range 3D imaging.

[0005] A first aspect of the embodiments of the present disclosure provides a 3D imaging method based on a hybrid event camera, comprising:

[0006] Controlling the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of structured light stripes that alternate between light and dark;

[0007] Controlling the hybrid event camera to collect an event stream of the structured light pattern on the surface of the object being measured in an event mode;

[0008] Determining, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured;

[0009] Controlling the projection module to project a second set of coded structured light patterns onto the object under test;

[0010] Controlling the hybrid event camera to capture a first image of the structured light pattern on the surface of the object under test in a traditional frame mode;

[0011] determining a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on brightness information of the first image;

[0012] determining a third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object to be measured based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object to be measured;

[0013] The object to be measured is reconstructed in three dimensions based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object to be measured and the calibration parameters of the hybrid event camera.

[0014] A second aspect of the present disclosure provides a 3D imaging system based on a hybrid event camera, comprising:

[0015] A first control module is used to control the projection module to project a first set of coded structured light patterns onto the object under test; the coded structured light patterns include a plurality of structured light stripes that alternate between light and dark;

[0016] An event stream acquisition module, used to control the hybrid event camera to acquire the event stream of the structured light pattern on the surface of the object being measured in event mode;

[0017] a first phase calculation module, configured to determine, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured;

[0018] a second control module, configured to control the projection module to project a second set of coded structured light patterns onto the object under test;

[0019] a brightness image acquisition module, configured to control the hybrid event camera to acquire a first image of the structured light pattern on the surface of the object under test in a traditional frame mode;

[0020] a second phase calculation module, configured to determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image;

[0021] a third phase calculation module, configured to determine a third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object under test based on the first phase and the second phase; the third phase being used to indicate a precise phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object under test;

[0022] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the object under test based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object under test and the calibration parameters of the hybrid event camera.

[0023] According to a third aspect of an embodiment of the present disclosure, a 3D camera is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the steps of the above-mentioned 3D imaging method based on a hybrid event camera are implemented.

[0024] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned 3D imaging method based on a hybrid event camera are implemented.

[0025] The beneficial effects of the 3D imaging method and system based on the hybrid event camera provided by the embodiments of the present disclosure are:

[0026] The disclosed embodiments utilize the advantages of a hybrid event camera that can output both event streams like an event camera and brightness images like a traditional camera. The high temporal resolution of the event stream is used to quickly obtain approximate phase information, and the high spatial resolution of the traditional frame image is used to refine the phase, thereby obtaining more accurate phase information, ultimately achieving high-speed, high-precision, and high-dynamic range 3D imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] Figure 1 A schematic flow chart of a 3D imaging method based on a hybrid event camera provided in one embodiment of the present disclosure;

[0029] Figure 2 A schematic diagram of a square wave coded structured light pattern provided in one embodiment of the present disclosure;

[0030] Figure 3 A schematic diagram of sinusoidal structured light stripes provided in one embodiment of the present disclosure;

[0031] Figure 4 A schematic diagram of line-shifted structured light stripes provided in one embodiment of the present disclosure;

[0032] Figure 5 A schematic diagram of part of the data of an event stream image provided by an embodiment of the present disclosure;

[0033] Figure 6 A structural block diagram of a 3D imaging system based on a hybrid event camera provided in one embodiment of the present disclosure;

[0034] Figure 7 A schematic block diagram of a 3D camera provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary detail.

[0036] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0037] Please refer to Figure 1 , Figure 1 A schematic flow chart of a 3D imaging method based on a hybrid event camera provided in one embodiment of the present disclosure, the method comprising:

[0038] S101: Control the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of structured light stripes that alternate between light and dark.

[0039] In this embodiment, the projection module includes a light source, a projection structure and a coded structured light projection control system. The light source is used to generate projection light, the projection structure is used to generate structured light, and the coded structured light projection control system is used to control the light source and the projection structure to generate coded structured light.

[0040] Exemplarily, the coded structured light projection module may be a projection system such as a MEMS projection optical engine or a DLP optical engine.

[0041] The first group of coded structured light patterns can be a single sinusoidal fringe pattern or a single square wave fringe pattern, or multiple sinusoidal fringe patterns, multiple Gray code fringe patterns, multiple phase shift code fringe patterns, or multiple line shift code fringe patterns.

[0042] For example, the projection module can project a square wave coded structured light pattern with a period of n in a scanning mode, such as Figure 2 As shown; it is also possible to project 3 images with a period of n and a phase difference of Sinusoidal structured light stripes, such as Figure 3 As shown; it is also possible to project 4 images with a period of n and a phase difference of The line-shifted structured light stripes are as follows Figure 4 shown.

[0043] S102: Control the hybrid event camera to collect an event stream of the structured light pattern on the surface of the object being measured in an event mode.

[0044] In this embodiment, if Figure 2-Figure 4 As shown, each cycle of the coded structured light pattern consists of one bright stripe and one dark stripe. When projecting bright stripes, the structured light source is on; when projecting dark stripes, the structured light source is off. Therefore, only bright stripes can trigger the event camera to generate events (excluding noise). When the coded structured light projection module projects structured light stripes in scanning mode, ensure that the timestamps of structured light stripes in different cycles are different.

[0045] S103: Determine, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured.

[0046] In this embodiment, effective pixels refer to pixels corresponding to the overlapping field of view of the projection field of the coded structured light projection module and the field of view of the hybrid event camera in the imaging plane of the hybrid event camera.

[0047] Since the event stream records the time sequence and position information of the interaction between the encoded structured light and the object surface, by analyzing the timestamp differences between different events and the spatial distribution of the corresponding pixel coordinates, combined with the encoding rules of the structured light stripes (such as the stripe period, the relationship between phase change and time), the phase of the structured light stripes corresponding to each effective pixel in the imaging plane can be calculated.

[0048] For example, by comparing the timestamp sequence of different events and the spatial position relationship of the corresponding pixel coordinates, combined with the known structured light fringe encoding rules, the phase of the structured light fringe corresponding to each effective pixel in the imaging plane can be calculated.

[0049] It should be noted that when projecting the coded structured light pattern in step S101, the stripe pattern is projected in a scanning manner. That is, different stripes within the same structured light pattern are projected at different times. Furthermore, it is necessary to ensure that the transmission speed of the event stream is greater than the generation speed of the event stream, and that different structured light stripes within the event stream have different timestamps. This ensures that the timestamps of each event in the event stream can be used to locate the period of the structured light pattern in which the structured light stripe that triggered the event is located.

[0050] S104: Control the projection module to project a second set of coded structured light patterns onto the object to be measured.

[0051] In this embodiment, the second group of coded structured light patterns can be a single sinusoidal stripe pattern or a single square wave stripe pattern, or multiple sinusoidal stripe patterns, multiple Gray code stripe patterns, multiple phase-shifted code stripe patterns, or multiple line-shifted code stripe patterns, and the second group of coded structured light patterns can be the same as or different from the first group of coded structured light patterns.

[0052] It should be noted that step S103 may be placed before step S104, or after step S104 and step S105, or may be performed simultaneously with step S104 and step S105.

[0053] S105: Control the hybrid event camera to capture a first image of the structured light pattern on the surface of the object under test in a traditional frame mode.

[0054] In this embodiment, in traditional frame mode, the hybrid event camera captures images at a fixed frame rate, and is able to obtain complete brightness information of the structured light pattern on the object surface. This brightness information reflects the modulation of the structured light on the object surface and can be used to further analyze the phase of the structured light fringes.

[0055] S106: Determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image.

[0056] In this embodiment, image processing is performed on the collected first image, and the second phase of each effective pixel on the imaging plane in the corresponding structured light fringes is calculated according to the brightness information.

[0057] For example, the phase shift method commonly used in this field can be used to analyze the brightness changes of structured light patterns with different phase shifts in the image, and use trigonometric function relationships to calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripes.

[0058] The method of determining phase based on image brightness information can fully utilize the spatial resolution advantage of traditional frame images and provide more accurate phase information.

[0059] S107: Determine a third phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object under test based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object under test.

[0060] In this embodiment, the first phase obtained from the event stream and the second phase obtained from the brightness information of traditional frame images are combined to fully leverage the advantages of both methods and compensate for their respective shortcomings. For example, the phase information provided by the event stream has high temporal resolution, enabling rapid determination of the approximate phase; while the phase information from traditional frame images has high spatial resolution and accuracy, allowing for refined phase analysis.

[0061] Specifically, the third phase can be obtained by adding the first phase and the second phase. For example, the first phase of a bright stripe is , the second phase of the pixel (x, y) in the bright stripe is , then the exact phase of the pixel (x, y) is: .

[0062] S108: Perform three-dimensional reconstruction of the object to be measured based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object to be measured and the calibration parameters of the hybrid event camera.

[0063] In this embodiment, the principle of 3D reconstruction (also known as 3D imaging) is to utilize the geometric relationship between the structured light phase and the height of the object's surface, combined with the calibration parameters of the hybrid event camera (including intrinsic and extrinsic parameters), to calculate the 3D coordinates of each point on the object's surface (i.e., point cloud data) through triangulation and other methods. Therefore, accurate phase information and calibration parameters are key to achieving high-precision 3D reconstruction. Using the third phase obtained in the previous step and the pre-calibrated camera parameters, a 3D model of the object can be reconstructed, providing a foundation for subsequent analysis and application.

[0064] During the 3D reconstruction process, the object can be reconstructed in either a triggered or continuous mode. Triggered mode means the system executes steps S101 to S107 each time it receives a soft or hard trigger signal. Continuous mode means the system executes steps S101 to S107 in a loop after receiving a start command, without a trigger signal.

[0065] From the above, it can be concluded that this embodiment takes advantage of the hybrid event camera's ability to output both event streams like an event camera and brightness images like a traditional camera. It uses the high temporal resolution of the event stream to quickly obtain approximate phase information, and then refines the phase through the high spatial resolution of the traditional frame image to obtain more accurate phase information, ultimately achieving high-speed, high-precision, and high-dynamic range 3D imaging.

[0066] In one embodiment of the present disclosure, determining a first phase of a structured light fringe corresponding to each effective pixel in an imaging plane in a structured light pattern on a surface of a measured object based on a timestamp and pixel coordinates of an event stream includes:

[0067] Create a photo with all zero values ​​based on the resolution of the image sensor in the hybrid event camera;

[0068] Assign the timestamp of each event in the event stream to the pixel coordinates corresponding to the event, and obtain a second image with the timestamp value;

[0069] Filtering the second image to obtain valid pixels in the second image;

[0070] Clustering consecutive non-zero pixels and zero pixels in the second image respectively to obtain a plurality of bright stripes and a plurality of dark stripes in the structured light pattern on the surface of the object being measured;

[0071] Determine the period position of the bright stripe according to the timestamp of the bright stripe; the period position of the bright stripe is used to indicate the period in which the bright stripe is located in the structured light pattern on the surface of the object being measured;

[0072] The first phase of the structured light fringes corresponding to each effective pixel on the imaging plane in the structured light pattern on the surface of the measured object is determined based on the periodic position of the bright fringes.

[0073] In this embodiment, a specific implementation method for determining the first phase based on the event stream is provided. The specific steps include:

[0074] (1) Based on the resolution of the image sensor, a photo with all values ​​0 is created. The timestamp t of each event in a set of event streams is assigned to the pixel position (u, v) corresponding to the event. A picture with the value of the timestamp can be obtained.

[0075] (2) Filter the image to remove events triggered by noise.

[0076] (3) For any row in the image, cluster the consecutive non-zero pixels and zero pixels separately. The clustering results are as follows: Figure 5 As shown in the figure, continuous non-zero pixels represent bright stripes, continuous zero pixels represent dark stripes, and a continuous bright stripe and a dark stripe constitute a cycle of the structured light stripe.

[0077] (4) According to the timestamp of the bright stripes, calculate the period in which the bright stripes and dark stripes of the structured light stripes are located.

[0078] From the above, it can be concluded that this embodiment fully utilizes the timestamp and pixel coordinate information of the event stream data of the hybrid event camera, converts it into an image form that is easy to analyze, and mines the implicit structured light fringe phase information in the event stream, providing an important basis for subsequent three-dimensional reconstruction.

[0079] In one embodiment of the present disclosure, filtering the second image to obtain valid pixels in the second image includes:

[0080] For any pixel coordinate whose timestamp value is greater than zero, if its two adjacent pixel values ​​along the scanning direction are both zero, the pixel value is set to zero.

[0081] In this embodiment, considering that in normal structured light streak imaging, continuous bright streaks (regions where events occur) typically form a certain continuous pixel region, while isolated pixels are more likely caused by noise. Therefore, for any pixel value with a timestamp value greater than 0, if its two adjacent pixel values ​​along the scanning direction (defined as the row direction in this embodiment) are both 0, the pixel is considered an isolated event pixel triggered by noise and the pixel value is set to 0. Otherwise, the pixel is considered a valid event pixel and is retained.

[0082] It can be concluded from the above that this embodiment filters noise points based on the continuity of bright stripes and then filters the second image, thereby eliminating the interference of noise on structured light stripe analysis.

[0083] In one embodiment of the present disclosure, the second set of coded structured light patterns is three images with a phase difference of The sinusoidal fringe pattern determines the second phase of each effective pixel on the imaging plane in the corresponding structured light fringe based on the brightness information of the first image, including:

[0084] Extracting the light intensity value of each valid pixel in the three first images;

[0085] Calculating the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the first formula;

[0086] The first formula is:

[0087] ;in, represents the second phase, 、 、 are the light intensity values ​​of the effective pixels in the three first pictures respectively.

[0088] In this embodiment, the second set of coded structured light patterns is three phase difference Taking the sinusoidal fringe pattern as an example, the light intensity distribution functions of the three sinusoidal structured light stripes are:

[0089]

[0090] in, Indicates the background light intensity, Indicates the modulated light intensity.

[0091] right 、 、 Performing linear combination, we get:

[0092]

[0093] Will Substitution We can get:

[0094]

[0095] Similarly,

[0096]

[0097] Therefore, we get:

[0098]

[0099]

[0100] Then the phase is obtained by the inverse tangent function :

[0101] .

[0102] In one embodiment of the present disclosure, the second set of coded structured light patterns is four images with a phase difference of The method of determining a second phase of each effective pixel on an imaging plane in a corresponding structured light stripe based on brightness information of a first image includes:

[0103] Extracting the light intensity value of each valid pixel in the four first images;

[0104] Calculating the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the second formula;

[0105] The second formula is:

[0106] ;in, represents the second phase, 、 、 、 are the light intensity values ​​of the effective pixels in the four first pictures respectively.

[0107] In this embodiment, the second set of coded structured light patterns is four phase difference Taking the line-shifted structured light stripes as an example, the light intensity distribution functions of the four sinusoidal structured light stripes are:

[0108]

[0109] in, Indicates the background light intensity, Indicates the modulated light intensity.

[0110] right 、 、 , perform linear combination and get:

[0111]

[0112] Therefore, we get

[0113] Then the phase is obtained by the inverse tangent function :

[0114] .

[0115] In one embodiment of the present disclosure, before controlling the projection module to project the first set of coded structured light patterns onto the object to be measured, the method further includes:

[0116] If the pixels generating the event stream in the hybrid event camera are the same as the pixels collecting the brightness image, the 3D imaging system is calibrated in the traditional frame mode, and the event mode uses the same calibration parameters as the traditional frame mode;

[0117] If the pixels generating the event stream in the hybrid event camera are different from the pixels capturing the luminance image, then after the 3D imaging system is calibrated in the traditional frame mode, the calibration parameters of the 3D imaging system in the event mode are calculated based on the position difference between the pixels generating the event stream and the pixels capturing the luminance image.

[0118] In this embodiment, considering that traditional frame-based calibration methods are highly mature, theoretically sound, and extensively validated in practice, they can obtain highly accurate camera intrinsic and extrinsic parameters. For example, the Zhang Zhengyou calibration method is used. By capturing images of a calibration plate in different poses, feature points on the plate are extracted. The correspondence between the image coordinates of these feature points and their real-world coordinates is then used to calculate the camera's intrinsic parameters (such as focal length, principal point coordinates, distortion coefficients, etc.) and extrinsic parameters (rotation matrix and translation vector). These parameters describe the geometric model of the camera image and serve as the foundation for subsequent 3D reconstruction.

[0119] Therefore, when the pixels generating the event stream are the same as the pixels capturing the brightness image, it indicates that the imaging geometry of the camera in the two modes is consistent. The event mode can directly use the parameters calibrated in the traditional frame mode without repeated calibration, saving calibration time and cost. At the same time, it ensures the consistency of the imaging parameters in the two modes, which is beneficial to the subsequent processing based on the data fusion of the two modes.

[0120] Otherwise, when the pixels generating the event stream in a hybrid event camera differ from the pixels capturing the luminance image, the calibration parameters used in the traditional frame mode must be adjusted based on this positional difference to obtain the calibration parameters for the 3D imaging system in event mode. For example, if the event stream pixels have a fixed horizontal offset relative to the luminance image pixels, the impact of this offset on the object's spatial position calculation must be considered when calculating the extrinsic parameters in event mode. Through appropriate mathematical transformations, the calibration parameters used in the traditional frame mode can be converted to those suitable for event mode, ensuring accurate 3D reconstruction in event mode.

[0121] It can be concluded from the above that this embodiment performs event mode calibration based on the calibration method in the traditional frame mode, and can utilize the advantage of high accuracy of the traditional frame mode calibration to improve the accuracy of the event stream mode calibration.

[0122] Corresponding to the 3D imaging method based on the hybrid event camera of the above embodiment, Figure 6 This is a structural block diagram of a 3D imaging system based on a hybrid event camera according to an embodiment of the present disclosure. For ease of illustration, only the parts related to the embodiment of the present disclosure are shown. Figure 6 The 3D imaging system 20 based on the hybrid event camera includes: a first control module 21, an event stream acquisition module 22, a first phase calculation module 23, a second control module 24, a brightness image acquisition module 25, a second phase calculation module 26, a third phase calculation module 27 and a 3D reconstruction module 28.

[0123] The first control module 21 is used to control the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of alternating light and dark structured light stripes;

[0124] An event stream acquisition module 22 is used to control the hybrid event camera to acquire the event stream of the structured light pattern on the surface of the object being measured in an event mode;

[0125] A first phase calculation module 23 is used to determine the first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the measured object based on the timestamp and pixel coordinates of the event stream;

[0126] A second control module 24 is used to control the projection module to project a second set of coded structured light patterns onto the object under test;

[0127] a brightness image acquisition module 25 for controlling the hybrid event camera to acquire a first image of the structured light pattern on the surface of the object under test in a conventional frame mode;

[0128] A second phase calculation module 26 is configured to determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image;

[0129] a third phase calculation module 27 for determining a third phase of each effective pixel on the imaging plane in the structured light pattern based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel on the imaging plane in the structured light pattern;

[0130] The three-dimensional reconstruction module 28 is used to perform three-dimensional reconstruction of the object under test based on the third phase of each effective pixel in the imaging plane in the structured light pattern and the calibration parameters of the hybrid event camera.

[0131] In one embodiment of the present disclosure, the first phase calculation module 23 is specifically configured to:

[0132] Create a photo with all zero values ​​based on the resolution of the image sensor in the hybrid event camera;

[0133] Assign the timestamp of each event in the event stream to the pixel coordinates corresponding to the event, and obtain a second image with the timestamp value;

[0134] Filtering the second image to obtain valid pixels in the second image;

[0135] Clustering consecutive non-zero pixels and zero pixels in the second image respectively to obtain a plurality of bright stripes and a plurality of dark stripes in the structured light pattern on the surface of the object being measured;

[0136] Determine the period position of the bright stripe according to the timestamp of the bright stripe; the period position of the bright stripe is used to indicate the period in which the bright stripe is located in the structured light pattern on the surface of the object being measured;

[0137] The first phase of the structured light fringes corresponding to each effective pixel on the imaging plane in the structured light pattern on the surface of the measured object is determined based on the periodic position of the bright fringes.

[0138] In one embodiment of the present disclosure, the first phase calculation module 23 is further configured to:

[0139] For any pixel coordinate whose timestamp value is greater than zero, if its two adjacent pixel values ​​along the scanning direction are both zero, the pixel value is set to zero.

[0140] In one embodiment of the present disclosure, the first group of coded structured light patterns and the second group of coded structured light patterns are one or a combination of two of a single sinusoidal stripe pattern, a single square wave stripe pattern, multiple sinusoidal stripe patterns, multiple Gray code stripe patterns, multiple phase shift code stripe patterns, and multiple line shift code stripe patterns.

[0141] In one embodiment of the present disclosure, the second set of coded structured light patterns is three images with a phase difference of Sinusoidal fringe pattern, the second phase calculation module 26 is specifically used to:

[0142] Extracting the light intensity value of each valid pixel in the three first images;

[0143] Calculating the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the first formula;

[0144] The first formula is:

[0145] ;in, represents the second phase, 、 、 are the light intensity values ​​of the effective pixels in the three first pictures respectively.

[0146] In one embodiment of the present disclosure, the second set of coded structured light patterns is three images with a phase difference of The second phase calculation module 26 is specifically used for:

[0147] Extracting the light intensity value of each valid pixel in the three first images;

[0148] Calculating the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the second formula;

[0149] The second formula is:

[0150] ;in, represents the second phase, 、 、 are the light intensity values ​​of the effective pixels in the three first pictures respectively.

[0151] In one embodiment of the present disclosure, the first control module 21 is specifically configured to:

[0152] If the pixels generating the event stream in the hybrid event camera are the same as the pixels collecting the brightness image, the 3D imaging system is calibrated in the traditional frame mode, and the event mode uses the same calibration parameters as the traditional frame mode;

[0153] If the pixels generating the event stream in the hybrid event camera are different from the pixels capturing the luminance image, then after the 3D imaging system is calibrated in the traditional frame mode, the calibration parameters of the 3D imaging system in the event mode are calculated based on the position difference between the pixels generating the event stream and the pixels capturing the luminance image.

[0154] See also Figure 7 , Figure 7 This is a schematic block diagram of a 3D camera provided by an embodiment of the present disclosure. Figure 7 The 3D camera in this embodiment shown here primarily comprises three components: a hybrid event camera 1, a coded structured light projection module 2, and a system control module 3. The coded structured light projection module 2 projects coded structured light onto the object in a scanning manner; the hybrid event camera 1 captures the structured light event stream and structured light image reflected from the object; and the system control module 3 is electrically connected to the coded structured light projection module 2 and the hybrid event camera 1, receives the structured light event stream and structured light image captured by the hybrid event camera 1, and processes the event stream and image to generate point cloud data representing the color information of the object being measured.

[0155] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or replacements within the technical scope disclosed in this disclosure, and such modifications or replacements should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A 3D imaging method based on a hybrid event camera, characterized in that: include: Controlling the projection module to project a first set of coded structured light patterns onto the object to be measured; The coded structured light pattern includes a plurality of alternating light and dark structured light stripes; Controlling the hybrid event camera to collect an event stream of the structured light pattern on the surface of the object being measured in an event mode; Determining, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured; Controlling the projection module to project a second set of coded structured light patterns onto the object under test; Controlling the hybrid event camera to capture a first image of the structured light pattern on the surface of the object under test in a traditional frame mode; determining a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on brightness information of the first image; Determining a third phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the measured object based on the first phase and the second phase; The third phase is used to indicate the precise phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object being measured; The object to be measured is reconstructed in three dimensions based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object to be measured and the calibration parameters of the hybrid event camera.

2. The 3D imaging method based on a hybrid event camera according to claim 1, wherein: The determining, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object under test includes: Create a photo with all zero values ​​based on the resolution of the image sensor in the hybrid event camera; Assigning the timestamp of each event in the event stream to the pixel coordinates corresponding to the event to obtain a second picture with the timestamp value; Filtering the second image to obtain valid pixels in the second image; Clustering consecutive non-zero pixels and zero pixels in the second image respectively to obtain a plurality of bright stripes and a plurality of dark stripes in the structured light pattern on the surface of the object being measured; Determining a period position of the bright stripe according to a timestamp of the bright stripe; the period position of the bright stripe is used to indicate the period in which the bright stripe is located in the structured light pattern on the surface of the object being measured; The first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the measured object is determined based on the periodic position of the bright fringes.

3. The 3D imaging method based on a hybrid event camera according to claim 2, wherein: Filtering the second image to obtain valid pixels in the second image includes: For any pixel coordinate whose timestamp value is greater than zero, if its two adjacent pixel values ​​along the scanning direction are both zero, the pixel value is set to zero.

4. The 3D imaging method based on a hybrid event camera according to claim 1, wherein: The first group of coded structured light patterns and the second group of coded structured light patterns are one or a combination of a single sinusoidal fringe pattern, a single square wave fringe pattern, multiple sinusoidal fringe patterns, multiple Gray code fringe patterns, multiple phase shift code fringe patterns, and multiple line shift code fringe patterns.

5. The 3D imaging method based on a hybrid event camera according to claim 4, wherein: The second set of coded structured light patterns consists of three images with a phase difference of Sinusoidal fringe pattern, wherein determining the second phase of each effective pixel on the imaging plane in the corresponding structured light fringe based on the brightness information of the first image includes: Extracting the light intensity value of each valid pixel in the three first images; Calculating the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the first formula; The first formula is: ;in, represents the second phase, 、 、 are the light intensity values ​​of the effective pixels in the three first pictures respectively.

6. The 3D imaging method based on a hybrid event camera according to claim 4, wherein: The second set of coded structured light patterns consists of four images with a phase difference of The method of determining the second phase of each effective pixel on the imaging plane in the corresponding structured light stripe based on the brightness information of the first image includes: Extracting the light intensity value of each valid pixel in the four first images; Calculating the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the second formula; The second formula is: ;in, represents the second phase, 、 、 、 are the light intensity values ​​of the effective pixels in the four first pictures respectively.

7. The 3D imaging method based on a hybrid event camera according to claim 1, wherein: Before controlling the projection module to project the first set of coded structured light patterns onto the object to be measured, the method further includes: If the pixels generating the event stream in the hybrid event camera are the same as the pixels collecting the brightness image, the 3D imaging system is calibrated in the traditional frame mode, and the event mode uses the same calibration parameters as the traditional frame mode; If the pixels generating the event stream in the hybrid event camera are different from the pixels capturing the luminance image, then after the 3D imaging system is calibrated in the traditional frame mode, the calibration parameters of the 3D imaging system in the event mode are calculated based on the position difference between the pixels generating the event stream and the pixels capturing the luminance image.

8. A 3D imaging system based on a hybrid event camera, characterized in that: include: A first control module is used to control the projection module to project a first set of coded structured light patterns onto the object under test; The coded structured light pattern includes a plurality of alternating light and dark structured light stripes; An event stream acquisition module, used to control the hybrid event camera to acquire the event stream of the structured light pattern on the surface of the object being measured in event mode; a first phase calculation module, configured to determine, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured; a second control module, configured to control the projection module to project a second set of coded structured light patterns onto the object under test; a brightness image acquisition module, configured to control the hybrid event camera to acquire a first image of the structured light pattern on the surface of the object under test in a traditional frame mode; a second phase calculation module, configured to determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image; a third phase calculation module, configured to determine a third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object under test based on the first phase and the second phase; the third phase being used to indicate a precise phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object under test; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the object under test based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object under test and the calibration parameters of the hybrid event camera.

9. A 3D camera comprising a projection module, a hybrid event camera, a main control module, and a computer program stored in and running on the main control module, characterized in that: When the main control module executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images