METHOD FOR 3D DETECTION OF OBJECTS USING EB SENSORS

Abstract

The invention relates to a method for point-by-point determination of the spatial coordinates of a three-dimensional object by means of at least two photogrammetrically and temporally synchronized image sensors, which record the object to be measured from different angles whereupon the spatial coordinate of the respective sampling point is determined from the disparity of the image information provided by the image sensors for the same sampling point of the object to be measured, characterized in that Event-based image sensors (EB sensors) are used as image sensors and a laser line is used as the active light source, which is moved column by column via a deflection device.

Description

[0001] The invention relates to a method for the 3D detection of objects using imaging EB sensors according to the preamble of claim 1 and a corresponding device for implementing this method. The term "object" in the context of this description of the invention is understood broadly; it also encompasses the detection of measurement objects and scenes. TECHNICAL BACKGROUND

[0002] Systems currently on the market for three-dimensional, areal capture of scenes based on triangulation can be roughly divided into passive and active methods.

[0003] Passive methods are based on classical photogrammetry: Two image sensors capture a scene from different angles. Points in the left sensor are assigned manually or automatically by an algorithm to corresponding points in the right sensor. The resulting disparity image, along with calibration parameters (external and internal orientation), is further processed into a point cloud. The point cloud contains the surface points of the observed scene in three dimensions (x, y, z). Finding the corresponding points is often difficult.

[0004] For texture-free surfaces ("white wall"), it is fundamentally impossible to find corresponding points, and the method fails. Even for texture-rich scenes, error-minimizing, automated analysis is computationally complex, as large areas around a point usually have to be examined for similar features. Common methods include block matching or semi-global block matching, which attempt to minimize a cost function.

[0005] The active procedures are characterized as follows: They always contain a projection unit that projects structured light onto the scene and at least one other camera with an image sensor.

[0006] If only one pattern is projected onto the scene, the classic photogrammetry method described above can be applied to texture-free scenes, since the structured lighting provides the necessary texture. However, finding the correspondences remains complex. A variant of the active method involves using a projection unit and a camera. The projector projects a sequence of stripe light patterns onto the scene and encodes each point on the scene surface with a Gray code that corresponds to its column number. This automatically establishes a correspondence between projector pixels and camera pixels, and analysis of the corresponding points is no longer necessary. The method is therefore fast in terms of computing time, but typically requires at least 10 consecutive pattern projections.Further problems with the active methods are regularly dynamic problems (overexposed image areas) and intermodulation (coded light is reflected multiple times, resulting in artifacts in the point cloud).

[0007] The current methods can be described as slow and are not suitable for moving scenes. Time intervals from the capture to the point cloud are typically 0.1 seconds to several seconds, during which the measured object must be stationary. Even with time-optimized image processing, the lower limit for the acquisition time is determined by the maximum frame rate. For typical commercial image sensors, this is a few tens of milliseconds.

[0008] High-speed image sensors are usually ruled out due to their high price and complex electronics. THE PROBLEM UNDERLYING THE INVENTION

[0009] Area-contact sensors with short acquisition times and minimal time to point cloud output are particularly important for industrial applications (robotic applications, moving objects on conveyor belts, depalletizing, separation, packaging, machine loading, bin picking, SLAM, obstacle detection, etc.). Other applications include sports and biodynamics (e.g., 3D analysis of golf swings, tennis, etc.). THE TASK OF THE INVENTION

[0010] In light of what has been said so far, the object of the invention is to provide a method and a corresponding device which are able to output the required 3D information or the point cloud representing it more quickly. THE INVENTIVE SOLUTION

[0011] Therefore, a novel method is proposed based on an array of event-based sensors and an active source.

[0012] The patent proposes a method for determining the spatial coordinates of a three-dimensional object, usually point by point. It uses at least two, and ideally only two, photogrammetrically and temporally synchronized image sensors. The image sensors are usually spatially separated from each other, ideally at least 25 cm and even better at least 50 cm apart. This is an option, not currently a requirement.

[0013] These record the object to be measured from different angles. The spatial coordinates of the respective scanning point can then be determined from the disparity in the image information provided by the image sensors for the same scanning point of the object to be measured. The method according to the invention is characterized by the use of event-based image sensors (EB sensors) as image sensors. However, the invention does not stop there; it also provides for the use of a laser line as the active light source, which is moved column by column via a deflection device.

[0014] By using event-based image sensors paired with a laser beam that exposes only one or a reduced number of pixels at any given time, it is possible to determine significantly more quickly which pixels of the two sensors belong together in pairs and are therefore decisive for determining the disparity, i.e. the required 3D information can be calculated.

[0015] The following must be known about the event-based image processing used in the invention: Conventional systems capture images at a fixed frame rate and then forward them to high-performance processors, which process them with high required bandwidth. This current approach requires large amounts of data to be transferred and processed periodically, even though it is redundant because the individual pixels do not change from image to image. In contrast, systems with event-based technology do not record images in a fixed cycle. Instead, each pixel independently provides information, but only when the pixel's intensity has changed.

[0016] Ultimately, and crucially, EB sensors mimic human vision: the human eye or brain is very sensitive to rapid changes, while the brain ignores most motionless data.

[0017] To explain this in more detail, event-based sensors (EB sensors) differ from traditional frame-based sensors in that each pixel "fires" when a grayscale change occurs, sending a data packet consisting of the (x,y) coordinate, polarity (grayscale transition), and timestamp. This happens completely asynchronously. Each pixel can also fire very quickly, typically less than a microsecond.

[0018] Another advantage of EB sensors is their extremely high dynamic range of >120 dB. This allows objects with high intensity dynamics (e.g., reflective metal sheets) to be easily detected.

[0019] The invention consists in arranging two such event-based sensors photogrammetrically and time-synchronized, using a laser line as the active light source. The laser line is then moved very quickly from left to right in columns using a polygon scanner. Pixels in both EB sensors are excited accordingly, column by column.

[0020] This has the crucial advantage that this excitation directly solves the correspondence problem, since points excited in a column with a timestamp in the left EB sensor can be directly identified in the right EB sensor with the same timestamp and the same row number – because only those pixels fire whose gray value has changed due to the laser irradiation. Since the bulk of the pixels is outside the laser irradiation, the bulk of the pixels remains "inactive." This results in a reduced data stream from both EB sensors, from which the disparity can be determined without great computational effort, because it is easily determined which pixels of the two EB sensors form a matching pixel pair and therefore provide values ​​that are relevant for determining the disparity.

[0021] The laser line can be moved from left to right in a very short time.

[0022] The generation of the scene's point cloud depends almost exclusively on the pixel response time and the speed of the laser line sweep. Typically, one million 3D points of the scene can be generated within one millisecond. Dynamic performance issues are eliminated due to the high dynamic range of the EB sensor.

[0023] Intermodulation problems are largely eliminated because the scene does not have to be illuminated across the entire area.

[0024] The polygon scanner has only minimal requirements for stabilizing the speed, since the EB sensors are self-synchronizing.

[0025] Since the contrast threshold of a pixel is relatively low, very low laser power can be used. A FURTHER PROBLEM UNDERLYING THE INVENTION

[0026] In some cases, there is a need for solutions that can be implemented with reduced resources, i.e. with less effort, and / or can be operated with reduced power consumption. ANOTHER INVENTIONAL SOLUTION

[0027] Instead of two EB sensors, a single EB sensor can be used, provided the beam direction of the laser line is precisely known. This can be achieved, for example, using an encoder in the polygon scanner. OPTIONAL OPTIONS FOR REFINING THE INVENTION

[0028] The amount of data can be reduced, for example, by targeted modulation of the laser (e.g. by hiding every second column).

[0029] Ideally, a polygon scanner, a galvanometer scanner or a MEMS or MEMS component is used as the deflection device, i.e. what is known to those skilled in the art as a “micro-electromechanical system” - e.g., by way of explanation, similar or identical to an array of individually controllable micromirrors, as is known from projectors.

[0030] A frame-based (color) camera can be used to subsequently impose the scene texture on the point cloud.

[0031] Another optional but advantageous approach involves positioning an optical bandpass filter, preferably between the lenses and the EB sensor(s), which isolates the EB sensor(s) from residual light. This further accelerates the determination of related pixels, as the number of currently firing pixels is further reduced. Furthermore, any ambient light present cannot excite any pixels. EXAMPLE OF IMPLEMENTATION

[0032] The following drawings show an example of an active event-based stereo sensor.

[0033] Fig. 2 shows two EB sensors (shown here as stereo normal case), e.g. from Sony, Inivision, with projection centers O L and O R .

[0034] The laser line (on the object approximately the width of the inversely transformed pixel) is moved across the scene within a very short time (ms) and generates lines of "firing" pixels in the left and right EB sensors. The correspondence of left / right pixels is immediately apparent line by line; in the image, for example, P L and P R . Thus, there are M correspondences per projected laser line.

[0035] Fig. Figure 1 shows the essential processing steps: The data stream from the left and right EB sensors is fed via an interface into a processing unit with a pipeline architecture (e.g., a Xilinx Ultrascale+ FPGA). In a non-standard stereo configuration, the R data is rectified, and then both data streams are combined to form a disparity map. Finally, together with the calibration parameters, they are calculated without delay into a point cloud.

[0036] The method typically does not require slowing down external memory accesses and is therefore characterized by its avoidance of memory access.

[0037] Fig. 3 shows an embodiment with two EB sensors with lenses, a polygon scanner with 6 facets, a laser (laser diode) with collimation optics and electronics (FPGA, control electronics).

[0038] Instead of the polygon scanner, other scanning units such as galvano scanners or MEMS components can be used. An optical bandpass filter can be placed between the lenses and the EB sensor to isolate the sensors from residual light. MISCELLANEOUS

[0039] In due course, a device for carrying out the method is also claimed either according to one or more of the claims set out in this application and / or with one or individual additional features from the introduction to the description and / or the description of the exemplary embodiment and / or the figures, wherein the device comprises at least one and preferably two, better only two spatially separated EB sensors.

Claims

[1] Method for determining the spatial coordinates of a three-dimensional object point by point by means of at least two photogrammetrically and temporally synchronized image sensors, which record the object to be measured from different angles whereupon the spatial coordinate of the respective sampling point is determined from the disparity of the image information provided by the image sensors for the same sampling point of the object to be measured, characterized by , that Event-based image sensors (EB sensors) are used as image sensors and a laser line is used as the active light source, which is moved column by column via a deflection device. [2] Method for determining the spatial coordinates of a three-dimensional object point by point using a single image sensor that captures the object to be measured characterized by , that an event-based image sensor (EB sensor) is used as the image sensor and a laser line is used as the active light source, which is moved column by column via a deflection device, wherein the beam direction of the laser line is determined and taken into account, preferably by means of an encoder, and then the spatial coordinate of the respective scanning point is determined from a comparison of the image information which the image sensor provides for a specific scanning point of the object to be measured and the current beam direction of the laser beam. [3] Method according to claim 1 or 2, characterized by that the deflection device is a polygon scanner, a galvanometer scanner or a MEMS component. [4] Method according to one of the preceding claims, characterized by that a reduction in the amount of data is achieved by arbitrary modulation of the laser, preferably by hiding every second column. [5] Method according to one of the preceding claims, characterized by that a frame-based (color) camera is used to subsequently impose the scene texture on the determined point cloud. [6] Method according to one of the preceding claims, characterized by that an optical bandpass filter is preferably located between the lenses and the EB sensor(s), which decouples the EB sensor(s) from residual light influences. [7] Device for carrying out the method according to one of the preceding claims with at least one EB sensor.

Images