3D imaging method and system based on hybrid event camera

Abstract

The invention provides a 3D imaging method and system based on a hybrid event camera. The method comprises the following steps: projecting a first group of coding structured light patterns to a measured object; collecting an event stream of the surface structured light pattern of the measured object; determining a first phase of the structured light stripe corresponding to each effective pixel of the imaging plane in the surface structured light pattern of the measured object based on the event stream; projecting a second group of coding structured light patterns to the measured object; collecting a first picture of a surface structured light pattern of the measured object; determining a second phase of each effective pixel of the imaging plane in the corresponding structured light stripe based on the brightness information of the first picture; determining a third phase of each effective pixel of the imaging plane in the surface structured light pattern of the measured object based on the first phase and the second phase; and performing three-dimensional reconstruction on the measured object based on the third phase and the calibration parameters of the mixed event camera. According to the 3D imaging method and system based on the hybrid event camera provided by the invention, high-speed, high-precision and high-dynamic-range 3D imaging can be realized.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, to a 3D imaging method and system based on a hybrid event camera. Background Art

[0002] Compared with 2D vision, 3D vision systems can obtain accurate three-dimensional information of the target, bringing unprecedented efficiency and accuracy to the detection field, thus opening up a new path for the development of other industries.

[0003] Among 3D vision solutions, the time-coded structured light solution has become the most widely used short-range 3D reconstruction solution due to its advantages such as high measurement accuracy, non-contact, and good robustness. However, due to the limitations of factors such as bandwidth, the time-coded structured light 3D camera based on the traditional frame camera has three sets of contradictions in practical applications: the contradiction between reconstruction speed and reconstruction accuracy, the contradiction between reconstruction speed and resistance to ambient light, and the contradiction between reconstruction speed and dynamic range. These contradictions require the 3D camera system to make a trade-off between reconstruction speed and reconstruction quality, and it is also difficult to use in scenes with strong ambient light such as outdoors. Summary of the invention

[0004] The purpose of the present disclosure is to provide a 3D imaging method and system based on a hybrid event camera to achieve high-speed, high-precision, and high-dynamic range 3D imaging.

[0005] According to a first aspect of an embodiment of the present disclosure, a 3D imaging method based on a hybrid event camera is provided, comprising: Controlling the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of structured light stripes that alternate between light and dark; Controlling the hybrid event camera to collect an event stream of a structured light pattern on the surface of the object being measured in an event mode; Based on the timestamp and pixel coordinates of the event stream, determining a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured; Controlling the projection module to project a second set of coded structured light patterns onto the object to be measured; Controlling the hybrid event camera to collect a first image of the structured light pattern on the surface of the object under test in a conventional frame mode; Determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image; Determine a third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object to be measured based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object to be measured; The object to be measured is reconstructed in three dimensions based on the third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object to be measured and the calibration parameters of the hybrid event camera.

[0006] A second aspect of the embodiments of the present disclosure provides a 3D imaging system based on a hybrid event camera, comprising: A first control module, used to control the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of structured light stripes that alternate between light and dark; An event stream acquisition module, used to control the hybrid event camera to acquire the event stream of the structured light pattern on the surface of the object under test in an event mode; A first phase calculation module, used to determine a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the measured object based on the timestamp and pixel coordinates of the event stream; A second control module is used to control the projection module to project a second set of coded structured light patterns onto the object to be measured; A brightness image acquisition module, used to control the hybrid event camera to acquire a first image of the structured light pattern on the surface of the object under test in a traditional frame mode; A second phase calculation module, used to determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image; A third phase calculation module, used to determine a third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the measured object based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the measured object; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the object under test based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object under test and the calibration parameters of the hybrid event camera.

[0007] According to a third aspect of an embodiment of the present disclosure, a 3D camera is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned 3D imaging method based on a hybrid event camera when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned 3D imaging method based on a hybrid event camera are implemented.

[0009] The beneficial effects of the 3D imaging method and system based on the hybrid event camera provided by the embodiments of the present disclosure are: The disclosed embodiments utilize the advantages of a hybrid event camera that can output both event streams like an event camera and brightness images like a traditional camera. The high temporal resolution of the event stream is used to quickly obtain approximate phase information, and the high spatial resolution of the traditional frame image is used to refine the phase, thereby obtaining more accurate phase information, and ultimately achieving high-speed, high-precision, and high-dynamic range 3D imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of a flow chart of a 3D imaging method based on a hybrid event camera provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a square wave coded structured light pattern provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of sinusoidal structured light stripes provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of line-shifted code structured light stripes provided in an embodiment of the present disclosure; Figure 5 A schematic diagram of part of the data of an event stream picture provided by an embodiment of the present disclosure; Figure 6 A structural block diagram of a 3D imaging system based on a hybrid event camera provided in an embodiment of the present disclosure; Figure 7 A schematic block diagram of a 3D camera provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0013] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0014] Please refer to Figure 1 , Figure 1A schematic flow chart of a 3D imaging method based on a hybrid event camera provided in an embodiment of the present disclosure, the method comprising: S101: Control the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of structured light stripes that alternate between light and dark.

[0015] In this embodiment, the projection module includes a light source, a projection structure and a coded structured light projection control system. The light source is used to generate projection light, the projection structure is used to generate structured light, and the coded structured light projection control system is used to control the light source and the projection structure to generate coded structured light.

[0016] Exemplarily, the coded structured light projection module may be a projection system such as a MEMS projection optical machine or a DLP optical machine.

[0017] The first group of coded structured light patterns may be a single sinusoidal fringe pattern or a single square wave fringe pattern, or may be multiple sinusoidal fringe patterns, multiple Gray code fringe patterns, multiple phase shift code fringe patterns, or multiple line shift code fringe patterns.

[0018] For example, the projection module can project a square wave coded structured light pattern with a period of n in a scanning mode, such as Figure 2 As shown; you can also project 3 images with a period of n and a phase difference of Sinusoidal structured light fringes, such as Figure 3 As shown; it is also possible to project 4 images with a period of n and a phase difference of The line-shifted structured light stripes are as follows Figure 4 shown.

[0019] S102: Control the hybrid event camera to collect an event stream of the structured light pattern on the surface of the object being measured in an event mode.

[0020] In this embodiment, if Figure 2-Figure 4 As shown in the figure, each cycle of the coded structured light pattern contains a bright stripe and a dark stripe. When the bright stripe is projected, the structured light source is turned on; when the dark stripe is projected, the structured light source is turned off. Therefore, only the bright stripe can trigger the event camera to generate an event (except for noise). When the coded structured light projection module projects structured light stripes in scanning mode, it is necessary to ensure that the timestamps of structured light stripes of different cycles are different.

[0021] S103: Determine, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel on the imaging plane in the structured light pattern on the surface of the object being measured.

[0022] In this embodiment, the effective pixel refers to: the pixel corresponding to the overlapping field of view of the projection field of the coded structured light projection module and the field of view of the hybrid event camera in the imaging plane of the hybrid event camera.

[0023] Since the event stream records the time sequence and position information of the interaction between the encoded structured light and the surface of the object, the phase of the structured light fringes corresponding to each effective pixel in the imaging plane can be calculated by analyzing the timestamp differences of different events and the spatial distribution of the corresponding pixel coordinates, combined with the encoding rules of the structured light fringes (such as the relationship between the fringe period, phase change and time).

[0024] For example, by comparing the timestamp sequence of different events and the spatial position relationship of the corresponding pixel coordinates, combined with the known structured light fringe encoding rules, the phase of the structured light fringe corresponding to each effective pixel in the imaging plane can be calculated.

[0025] It should be noted that when projecting the coded structured light pattern in step S101, the stripe pattern should be projected in a scanning manner, that is, the projection time of different stripes in the same structured light pattern is different. In addition, it is necessary to ensure that the transmission speed of the event stream is greater than the generation speed of the event stream, and the timestamps corresponding to different structured light stripes in the event stream are different, so as to ensure that the timestamps of each event in the event stream can be used to locate the period of the structured light pattern in which the structured light stripe that triggers the event is located.

[0026] S104: Control the projection module to project a second set of coded structured light patterns onto the object to be measured.

[0027] In this embodiment, the second group of coded structured light patterns can be a single sinusoidal fringe pattern or a single square wave fringe pattern, or multiple sinusoidal fringe patterns, multiple Gray code fringe patterns, multiple phase shift code fringe patterns, or multiple line shift code fringe patterns, and the second group of coded structured light patterns can be the same as the first group of coded structured light patterns or different.

[0028] It should be noted that step S103 may be located before step S104, or after step S104 and step S105, or may be performed simultaneously with step S104 and step S105.

[0029] S105: Control the hybrid event camera to collect a first image of the structured light pattern on the surface of the object to be measured in a traditional frame mode.

[0030] In this embodiment, in the traditional frame mode, the hybrid event camera captures images at a fixed frame rate, and can obtain complete brightness information of the structured light pattern on the surface of the object. This brightness information reflects the modulation of the structured light on the surface of the object and can be used to further analyze the phase of the structured light fringes.

[0031] S106: Determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image.

[0032] In this embodiment, by performing image processing on the collected first picture, the second phase of each effective pixel on the imaging plane in the corresponding structured light fringes is calculated according to the brightness information.

[0033] For example, the phase shift method commonly used in the art can be used to analyze the brightness changes of structured light patterns with different phase shifts in the image, and use trigonometric function relationships to calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripes.

[0034] The method of determining phase based on image brightness information can make full use of the spatial resolution advantage of traditional frame images and provide more accurate phase information.

[0035] S107: Determine a third phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object under test based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object under test.

[0036] In this embodiment, by combining the first phase obtained based on the event stream and the second phase obtained based on the brightness information of the traditional frame image, the advantages of the two methods can be fully utilized and their respective shortcomings can be compensated. For example, the phase information provided by the event stream has high temporal resolution and can quickly determine the approximate phase; while the phase information of the traditional frame image has high spatial resolution and accuracy and can be used to refine the phase.

[0037] Specifically, the third phase can be obtained by adding the first phase and the second phase. For example, the first phase of a bright stripe is , the second phase of the pixel point (x, y) in this bright fringe is , then the exact phase of the pixel (x, y) is: .

[0038] S108: Perform three-dimensional reconstruction of the object to be measured based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object to be measured and the calibration parameters of the hybrid event camera.

[0039] In this embodiment, the principle of three-dimensional reconstruction (i.e., 3D imaging) is to use the geometric relationship between the structured light phase and the height of the object surface, combined with the calibration parameters of the hybrid event camera (including internal parameters, external parameters, etc.), and calculate the three-dimensional coordinates of each point on the object surface (i.e., point cloud data) through triangulation and other methods. Therefore, accurate phase information and calibration parameters are the key to achieving high-precision three-dimensional reconstruction. The third phase obtained in the previous step and the pre-calibrated camera parameters can reconstruct the three-dimensional model of the object being measured, providing a basis for subsequent analysis and application.

[0040] During the 3D reconstruction process, the object under test can be reconstructed in a trigger mode or a continuous mode. The trigger mode means that the system runs steps S101 to S107 once each time it receives a soft trigger signal or a hard trigger signal; the continuous mode means that after the system receives a start instruction, it does not need a trigger signal and runs steps S101 to S107 in a loop.

[0041] From the above, it can be concluded that this embodiment uses the advantages of the hybrid event camera that can output event streams like an event camera and output brightness pictures like a traditional camera. It uses the high temporal resolution of the event stream to quickly obtain approximate phase information, and then refines the phase through the high spatial resolution of the traditional frame image, thereby obtaining more accurate phase information, and ultimately achieving high-speed, high-precision, and high-dynamic range 3D imaging.

[0042] In one embodiment of the present disclosure, determining the first phase of the structured light fringes corresponding to each effective pixel of the imaging plane in the structured light pattern on the surface of the measured object based on the timestamp and pixel coordinates of the event stream includes: Create a photo with all zero values ​​based on the resolution of the image sensor in the hybrid event camera; Assign the timestamp of each event in the event stream to the pixel coordinates corresponding to the event, and obtain a second picture with the timestamp as the value; Filtering the second image to obtain valid pixels in the second image; Clustering the continuous non-zero pixels and zero pixels in the second image respectively to obtain a plurality of bright stripes and a plurality of dark stripes in the structured light pattern on the surface of the object to be measured; Determine the period position of the bright stripe according to the timestamp of the bright stripe; the period position of the bright stripe is used to indicate the period of the structured light pattern on the surface of the object being measured where the bright stripe is located; The first phase of the structured light fringes corresponding to each effective pixel on the imaging plane in the structured light pattern on the surface of the measured object is determined based on the periodic position of the bright fringes.

[0043] In this embodiment, a specific implementation method for determining the first phase based on the event stream is provided, and the specific steps include: (1) Based on the resolution of the image sensor, create a photo with all values ​​​​zero, and assign the timestamp t of each event in a set of event streams to the pixel position (u, v) corresponding to the event. You can get a picture with the value of the timestamp.

[0044] (2) Filter the image to remove events triggered by noise.

[0045] (3) For any row in the image, cluster the consecutive non-zero pixels and zero pixels separately. The clustering results are as follows: Figure 5As shown in the figure, continuous non-zero pixels represent bright stripes, continuous zero pixels represent dark stripes, and a continuous bright stripe and a dark stripe constitute a cycle of the structured light stripe.

[0046] (4) Based on the timestamp of the bright stripes, calculate the period in which the bright stripes and dark stripes of the structured light stripes are located.

[0047] It can be concluded from the above that this embodiment makes full use of the timestamp and pixel coordinate information of the event stream data of the hybrid event camera, converts it into an image form that is easy to analyze, and mines the structured light fringe phase information implicit in the event stream, providing an important basis for subsequent three-dimensional reconstruction.

[0048] In one embodiment of the present disclosure, filtering the second picture to obtain valid pixels in the second picture includes: For any pixel coordinate whose timestamp value is greater than zero, if its two adjacent pixel values ​​along the scanning direction are both zero, the pixel value is set to zero.

[0049] In this embodiment, considering that in normal structured light stripe imaging, continuous bright stripes (areas where events occur) usually form a certain continuous pixel area, and isolated pixels are more likely to be caused by noise. Therefore, for any pixel value with a timestamp value greater than 0, if its two adjacent pixel values ​​along the scanning direction (defined as the row direction in this embodiment) are both 0, then the pixel is an isolated event pixel triggered by noise, and the pixel value is set to 0; otherwise, the pixel is a valid event pixel and is retained.

[0050] It can be concluded from the above that this embodiment screens noise points based on the continuity of bright stripes, and then filters the second image, thereby eliminating the interference of noise on the structured light stripe analysis.

[0051] In one embodiment of the present disclosure, the second group of coded structured light patterns is three images with a phase difference of The sinusoidal fringe pattern determines the second phase of each effective pixel on the imaging plane in the corresponding structured light fringe based on the brightness information of the first image, including: Extracting the light intensity value of each effective pixel in the three first images; Calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the first formula; The first formula is: ;in, represents the second phase, , , are the light intensity values ​​of each effective pixel in the three first pictures respectively.

[0052] In this embodiment, the second group of coded structured light patterns is three images with a phase difference of Taking the sinusoidal stripe pattern as an example, the light intensity distribution functions of the three sinusoidal structured light stripes are:

[0053] in, Indicates the background light intensity. Indicates the modulated light intensity.

[0054] right , , Performing linear combination, we get:

[0055] Will Substitution We can get:

[0056] Similarly,

[0057] Therefore, we get:

[0058]

[0059] Then the phase is obtained by the inverse tangent function : .

[0060] In one embodiment of the present disclosure, the second group of coded structured light patterns is four images with a phase difference of The method of determining a second phase of each effective pixel on an imaging plane in a corresponding structured light stripe based on brightness information of a first image comprises: Extracting the light intensity value of each effective pixel in the four first images; Calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the second formula; The second formula is: ;in, represents the second phase, , , , are the light intensity values ​​of each valid pixel in the four first pictures respectively.

[0061] In this embodiment, the second group of coded structured light patterns is four phase differences of Taking the line-shifted structured light stripes as an example, the light intensity distribution functions of the four sinusoidal structured light stripes are:

[0062] in, Indicates the background light intensity. Indicates the modulated light intensity.

[0063] right , , , and perform linear combination to obtain:

[0064] Therefore, we get

[0065] Then the phase is obtained by the inverse tangent function : .

[0066] In one embodiment of the present disclosure, before controlling the projection module to project the first group of coded structured light patterns onto the object to be measured, the method further includes: If the pixels of the event stream generated in the hybrid event camera are the same as the pixels of the acquired brightness image, the 3D imaging system is calibrated in the traditional frame mode, and the event mode uses the same calibration parameters as the traditional frame mode; If the pixels generating the event stream in the hybrid event camera are different from the pixels collecting the brightness image, after the 3D imaging system is calibrated in the traditional frame mode, the calibration parameters of the 3D imaging system in the event mode are calculated according to the position difference between the pixels generating the event stream and the pixels collecting the brightness image.

[0067] In this embodiment, considering that the calibration method in the traditional frame mode is very mature, the theory is perfect and has been verified by a large number of practices, it is possible to obtain high-precision camera internal and external parameters. For example, by using the Zhang Zhengyou calibration method, by shooting the calibration plate images in different postures, extracting the feature points on the calibration plate, and using the correspondence between the image coordinates of the feature points and the actual world coordinates, the camera's internal parameters (such as focal length, principal point coordinates, distortion coefficient, etc.) and external parameters (rotation matrix and translation vector) are calculated. These parameters describe the geometric model of camera imaging and are the basis for subsequent three-dimensional reconstruction.

[0068] Therefore, when the pixels generating the event stream are the same as the pixels collecting the brightness image, it indicates that the imaging geometry of the camera in the two modes is consistent. The event mode can directly use the parameters calibrated in the traditional frame mode without repeated calibration, saving calibration time and cost. At the same time, it ensures the consistency of the imaging parameters in the two modes, which is beneficial to the subsequent processing based on the data fusion of the two modes.

[0069] Otherwise, when the pixels of the event stream generated in the hybrid event camera are different from the pixels of the acquired brightness image, the calibration parameters in the traditional frame mode need to be adjusted according to the position difference to obtain the calibration parameters of the 3D imaging system in the event mode. For example, if the event stream pixels have a fixed offset in the horizontal direction relative to the brightness image pixels, then when calculating the external parameters in the event mode, it is necessary to consider the impact of this offset on the calculation of the spatial position of the object. Through reasonable mathematical transformation, the calibration parameters of the traditional frame mode are converted into calibration parameters suitable for the event mode to ensure accurate 3D reconstruction in the event mode.

[0070] It can be concluded from the above that this embodiment performs event mode calibration based on the calibration method under the traditional frame mode, and can take advantage of the high accuracy of the traditional frame mode calibration to improve the accuracy of the event stream mode calibration.

[0071] Corresponding to the 3D imaging method based on the hybrid event camera in the above embodiment, Figure 6 This is a block diagram of a 3D imaging system based on a hybrid event camera according to an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 6 The 3D imaging system 20 based on the hybrid event camera includes: a first control module 21, an event stream acquisition module 22, a first phase calculation module 23, a second control module 24, a brightness image acquisition module 25, a second phase calculation module 26, a third phase calculation module 27 and a three-dimensional reconstruction module 28.

[0072] Wherein, the first control module 21 is used to control the projection module to project a first set of coded structured light patterns onto the object to be measured; the coded structured light patterns include a plurality of alternating light and dark structured light stripes; An event stream acquisition module 22, used to control the hybrid event camera to acquire the event stream of the structured light pattern on the surface of the object under test in an event mode; A first phase calculation module 23, used to determine the first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the measured object based on the timestamp and pixel coordinates of the event stream; A second control module 24, used to control the projection module to project a second set of coded structured light patterns onto the object under test; The brightness image acquisition module 25 is used to control the hybrid event camera to acquire a first image of the structured light pattern on the surface of the object under test in a traditional frame mode; A second phase calculation module 26, configured to determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image; A third phase calculation module 27, used to determine a third phase of each effective pixel on the imaging plane in the structured light pattern based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel on the imaging plane in the structured light pattern; The three-dimensional reconstruction module 28 is used to perform three-dimensional reconstruction of the object under test based on the third phase of each effective pixel in the imaging plane in the structured light pattern and the calibration parameters of the hybrid event camera.

[0073] In one embodiment of the present disclosure, the first phase calculation module 23 is specifically used to: Create a photo with all zero values ​​based on the resolution of the image sensor in the hybrid event camera; Assign the timestamp of each event in the event stream to the pixel coordinates corresponding to the event, and obtain a second picture with the timestamp as the value; Filtering the second image to obtain valid pixels in the second image; Clustering the continuous non-zero pixels and zero pixels in the second image respectively to obtain a plurality of bright stripes and a plurality of dark stripes in the structured light pattern on the surface of the object to be measured; Determine the period position of the bright stripe according to the timestamp of the bright stripe; the period position of the bright stripe is used to indicate the period of the structured light pattern on the surface of the object being measured where the bright stripe is located; The first phase of the structured light fringes corresponding to each effective pixel on the imaging plane in the structured light pattern on the surface of the measured object is determined based on the periodic position of the bright fringes.

[0074] In one embodiment of the present disclosure, the first phase calculation module 23 is further configured to: For any pixel coordinate whose timestamp value is greater than zero, if its two adjacent pixel values ​​along the scanning direction are both zero, the pixel value is set to zero.

[0075] In one embodiment of the present disclosure, the first group of coded structured light patterns and the second group of coded structured light patterns are one or a combination of two of a single sinusoidal stripe pattern, a single square wave stripe pattern, multiple sinusoidal stripe patterns, multiple Gray code stripe patterns, multiple phase shift code stripe patterns, and multiple line shift code stripe patterns.

[0076] In one embodiment of the present disclosure, the second group of coded structured light patterns is three images with a phase difference of Sinusoidal fringe pattern, the second phase calculation module 26 is specifically used for: Extracting the light intensity value of each effective pixel in the three first images; Calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the first formula; The first formula is: ;in, represents the second phase, , , are the light intensity values ​​of each effective pixel in the three first pictures respectively.

[0077] In one embodiment of the present disclosure, the second group of coded structured light patterns is three images with a phase difference of The second phase calculation module 26 is specifically used for: Extracting the light intensity value of each effective pixel in the three first images; Calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the second formula; The second formula is: ;in, represents the second phase, , , are the light intensity values ​​of each effective pixel in the three first pictures respectively.

[0078] In one embodiment of the present disclosure, the first control module 21 is specifically used for: If the pixels of the event stream generated in the hybrid event camera are the same as the pixels of the acquired brightness image, the 3D imaging system is calibrated in the traditional frame mode, and the event mode uses the same calibration parameters as the traditional frame mode; If the pixels generating the event stream in the hybrid event camera are different from the pixels collecting the brightness image, after the 3D imaging system is calibrated in the traditional frame mode, the calibration parameters of the 3D imaging system in the event mode are calculated according to the position difference between the pixels generating the event stream and the pixels collecting the brightness image.

[0079] See also Figure 7 , Figure 7 FIG. 1 is a schematic block diagram of a 3D camera provided by an embodiment of the present disclosure. Figure 7 The 3D camera in the embodiment shown mainly includes the following three parts: a hybrid event camera 1, a coded structured light projection module 2 and a system main control module 3. The coded structured light projection module 2 projects coded structured light to the object to be measured in a scanning manner; the hybrid event camera 1 collects the structured light event stream and structured light picture reflected from the object to be measured; the system main control module 3 is electrically connected to the coded structured light projection module 2 and the hybrid event camera 1, receives the structured light event stream and structured light picture collected by the hybrid event camera 1, and processes the event stream and picture to obtain point cloud data of the color information of the object to be measured.

[0080] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A 3D imaging method based on a hybrid event camera, characterized in that: include: Controlling the projection module to project a first set of coded structured light patterns onto the object to be measured; The coded structured light pattern comprises a plurality of alternating light and dark structured light stripes; Controlling the hybrid event camera to collect an event stream of a structured light pattern on the surface of the object being measured in an event mode; Based on the timestamp and pixel coordinates of the event stream, determining a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the object being measured; Controlling the projection module to project a second set of coded structured light patterns onto the object to be measured; Controlling the hybrid event camera to collect a first image of the structured light pattern on the surface of the object under test in a conventional frame mode; Determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image; Determine a third phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the measured object based on the first phase and the second phase; The third phase is used to indicate the precise phase of each effective pixel on the imaging plane in the structured light pattern on the surface of the object being measured; The object to be measured is reconstructed in three dimensions based on the third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the object to be measured and the calibration parameters of the hybrid event camera.

2. The 3D imaging method based on a hybrid event camera according to claim 1, characterized in that: The determining, based on the timestamp and pixel coordinates of the event stream, a first phase of the structured light fringes corresponding to each effective pixel of the imaging plane in the structured light pattern on the surface of the measured object comprises: Create a photo with all zero values ​​based on the resolution of the image sensor in the hybrid event camera; Assigning the timestamp of each event in the event stream to the pixel coordinates corresponding to the event, to obtain a second picture with a value of the timestamp; Filtering the second picture to obtain valid pixels in the second picture; Clustering the continuous non-zero pixels and zero pixels in the second image respectively to obtain a plurality of bright stripes and a plurality of dark stripes in the structured light pattern on the surface of the object to be measured; Determine the period position of the bright stripe according to the timestamp of the bright stripe; the period position of the bright stripe is used to indicate the period of the structured light pattern on the surface of the object being measured in which the bright stripe is located; The first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the measured object is determined based on the periodic position of the bright fringes.

3. The 3D imaging method based on a hybrid event camera according to claim 2, characterized in that: Filtering the second picture to obtain valid pixels in the second picture includes: For any pixel coordinate whose timestamp value is greater than zero, if its two adjacent pixel values ​​along the scanning direction are both zero, the pixel value is set to zero.

4. The 3D imaging method based on a hybrid event camera according to claim 1, characterized in that: The first group of coded structured light patterns and the second group of coded structured light patterns are one or a combination of a single sinusoidal fringe pattern, a single square wave fringe pattern, multiple sinusoidal fringe patterns, multiple Gray code fringe patterns, multiple phase shift code fringe patterns, and multiple line shift code fringe patterns.

5. The 3D imaging method based on a hybrid event camera as claimed in claim 4, characterized in that: The second group of coded structured light patterns is three images with a phase difference of Sinusoidal fringe pattern, wherein determining the second phase of each effective pixel on the imaging plane in the corresponding structured light fringe based on the brightness information of the first picture comprises: Extracting the light intensity value of each effective pixel in the three first images; Calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the first formula; The first formula is: ;in, represents the second phase, , , are the light intensity values ​​of each effective pixel in the three first pictures respectively.

6. The 3D imaging method based on a hybrid event camera according to claim 4, characterized in that: The second group of coded structured light patterns is four images with a phase difference of The method of determining the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the brightness information of the first picture includes: Extracting the light intensity value of each effective pixel in the four first images; Calculate the second phase of each effective pixel in the imaging plane in the corresponding structured light stripe based on the second formula; The second formula is: ;in, represents the second phase, , , , are the light intensity values ​​of each valid pixel in the four first pictures respectively.

7. The 3D imaging method based on a hybrid event camera according to claim 1, characterized in that: Before controlling the projection module to project the first set of coded structured light patterns onto the object to be measured, the method further includes: If the pixels of the event stream generated in the hybrid event camera are the same as the pixels of the acquired brightness image, the 3D imaging system is calibrated in the traditional frame mode, and the event mode uses the same calibration parameters as the traditional frame mode; If the pixels generating the event stream in the hybrid event camera are different from the pixels collecting the brightness image, after the 3D imaging system is calibrated in the traditional frame mode, the calibration parameters of the 3D imaging system in the event mode are calculated according to the position difference between the pixels generating the event stream and the pixels collecting the brightness image.

8. A 3D imaging system based on a hybrid event camera, characterized in that: include: A first control module, used to control the projection module to project a first set of coded structured light patterns onto the object to be measured; The coded structured light pattern comprises a plurality of alternating light and dark structured light stripes; An event stream acquisition module, used to control the hybrid event camera to acquire the event stream of the structured light pattern on the surface of the object under test in an event mode; A first phase calculation module, used to determine a first phase of the structured light fringes corresponding to each effective pixel in the imaging plane in the structured light pattern on the surface of the measured object based on the timestamp and pixel coordinates of the event stream; A second control module is used to control the projection module to project a second set of coded structured light patterns onto the object to be measured; A brightness image acquisition module, used to control the hybrid event camera to acquire a first image of the structured light pattern on the surface of the object under test in a traditional frame mode; A second phase calculation module, used to determine a second phase of each effective pixel on the imaging plane in the corresponding structured light fringes based on the brightness information of the first image; A third phase calculation module, used to determine a third phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the measured object based on the first phase and the second phase; the third phase is used to indicate the precise phase of each effective pixel of the imaging plane in the structured light pattern on the surface of the measured object; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the object under test based on the third phase of each effective pixel in the imaging plane in the structured light pattern on the surface of the object under test and the calibration parameters of the hybrid event camera.

9. A 3D camera, comprising a projection module, a hybrid event camera, a main control module, and a computer program stored in and running on the main control module, characterized in that: When the main control module executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images