3D measuring system and methods for 3D measurement

The 3D measurement system improves accuracy and reduces complexity by using an event-based image sensor with a static pattern and position detection, enabling precise 3D geometry capture with enhanced resolution and faster scanning.

DE102022129836B4Active Publication Date: 2024-12-05CARL ZEISS INDUSTRIELLE MESSTECHNIKE GMBH
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
DE102022129836
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-11-11
Publication Date
2024-12-05
Estimated Expiration
2042-11-11

AI Technical Summary

Technical Problem

Existing 3D measurement systems using event-based image sensors require dynamic pattern projection to generate events, leading to complex structural designs, ineffective evaluation, and non-optimal capture of 3D geometries.

Method used

A 3D measurement system using an event-based image sensor with a static pattern and a movable sensor unit, where the sensor's position is detected relative to the measurement object, allowing the system to correlate events with position data to determine the absolute three-dimensional position of points on the object's surface without dynamic pattern projection.

Benefits of technology

This approach enhances measurement accuracy and resolution by leveraging the sensor's ability to generate more events in areas of significant change, reducing computational effort and structural complexity while allowing faster scanning speeds and a larger dynamic range.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

3D measuring system for 3D measurement of a measuring object, with a sensor unit comprising a projector and an event-based image sensor, and with an evaluation unit for determining 3D information from the output of the image sensor. wherein the projector projects a static pattern and the sensor unit is movable relative to the object being measured, wherein the 3D measuring system includes a position detection unit which detects the position of the sensor unit relative to the object being measured, wherein the evaluation unit determines an absolute three-dimensional position of points on the surface of the object being measured from the output of the image sensor and the position of the sensor unit relative to the object being measured, which is determined by the position detection unit and correlated in time to the output of the image sensor. wherein the evaluation unit correlates the three-dimensional position of the points on the surface of the object being measured relative to the sensor unit with a position of the sensor unit relative to the object being measured, determined by the position acquisition unit, and determines an absolute three-dimensional position of the points on the surface of the object being measured from the three-dimensional position of points on the surface of the object being measured relative to the sensor unit and the position of the sensor unit relative to the object being measured.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The present invention relates to a 3D measuring system and a method for 3D measurement of a measurement object with a sensor unit comprising a projector and an image sensor, and with an evaluation unit for determining 3D information from the output of the image sensor.

[0002] For example, the sensor unit can be a laser line scanner in which the projector projects one or more laser lines onto the measuring object in order to determine its three-dimensional shape based on the laser line that appears deformed in the image of the image sensor.

[0003] In particular, the projector of the sensor unit projects a static pattern and the sensor unit is moved relative to the measuring object so that the pattern covers the surface of the measuring object.

[0004] The acquisition of 3D geometries by laser line scanners guided by devices (coordinate measuring machines, robots, etc.) or handheld and tracked is known, for example, from DE 198 55 478 A1 and DE 10 2018 127 221 A1.

[0005] Typically, at least one laser line is projected onto a planar image sensor. Conventional image sensors operate with a fixed time frame; the two most important types are called "global shutter" and "rolling shutter." With "global shutter," the image projected onto the image sensor is captured in its entirety at a specific point in time and then output. With "rolling shutter," the capture occurs line by line (column by column would also be possible, but is uncommon). In industrial environments, image sensors with "global shutter" are typically used, as all image elements have the same temporal reference.

[0006] What these two modes of operation have in common, however, is that the timing of image capture is predetermined for the image sensor and follows a fixed pattern. The image to be captured, projected onto the image sensor, therefore has no effect on the timing of its capture.

[0007] In contrast, the temporal progression of image capture by “event-based vision sensors” or, in the terminology of the present application, “event-based image sensors” (EVS, sometimes also referred to as “dynamic image sensors”) is determined by the current image content – ​​in particular by the temporal progression of the brightness values ​​of its image components.

[0008] For example, an event is generated for each brightness change of an image element that exceeds a minimum value. Image elements with faster brightness changes generate a larger number of such events than image elements with slower changes.

[0009] In another implementation, the event rate is directly linked to the number of photons detected or the current generated by the photodiode of the respective pixel, so that brighter image areas generate more events than darker ones. However, this behavior can be transformed into the former by differentiating with respect to time.

[0010] The invention described below can be used with both types of event-based image sensors.

[0011] Typically, scenarios are described for the beneficial use of event-based image sensors in which their inherent ability to separate moving objects from stationary backgrounds can be used to enhance performance ("surveillance"). However, if the image sensors themselves are moving, the depth structure of a scene can be reconstructed from the events ("SLAM").

[0012] Furthermore, compared to conventional image sensors, the manageable brightness dynamic range can be increased because the individual image elements do not have a saturation limit; the only limiting effect may be a technically determined maximum event rate.

[0013] Another promising area is the capture of 3D geometries. According to the state of the art, a dynamically generated pattern is used for this purpose, i.e., a pattern that constantly changes in the image sensor image, regardless of the measurement situation. DE 10 2018 129 009 A1 describes how a change in the projected pattern has a direct impact on the generated events. This allows additional measurement information to be obtained. A similar approach is described in EP 3 907 466 A1, which involves deflecting the illumination line.

[0014] CN 1 12 525 107 A shows a scanner with an event-based camera in which a relative movement occurs between the scanner and the measuring object.

[0015] Further prior art is known from DE 10 2021 212 654 A1 and WO 2023 / 089 788 A1.

[0016] According to the state of the art, it was assumed that event-based image sensors could only be used for 3D measurement in combination with such a dynamic projector, which projects a changing pattern, since only the changing pattern generates corresponding events along the pattern, from which the pattern and thus the 3D geometry of the measurement object can be determined. However, known solutions have disadvantages with regard to complex structural design, ineffective analysis, and suboptimal capture of 3D geometries.

[0017] The object of the present invention is therefore to provide an improved 3D measurement system with an event-based image sensor. In particular, the inventor set himself the task of actually exploiting the advantages inherent in an event-based image sensor for 3D measurement.

[0018] This object is achieved by a 3D measuring system according to claim 1 and a method for 3D measurement of a measurement object according to claim 14. Preferred embodiments of the present invention are the subject of the dependent claims.

[0019] The present invention comprises a 3D measuring system for 3D measurement of a measurement object, comprising a sensor unit that includes a projector and an event-based image sensor, and comprising an evaluation unit for determining 3D information from the output of the image sensor. According to the invention, the projector projects a static pattern and the sensor unit is movable relative to the measurement object. The 3D measuring system comprises a position detection unit that detects the position of the sensor unit relative to the measurement object. The evaluation unit determines an absolute three-dimensional position of points on the surface of the measurement object from the output of the image sensor and the position of the sensor unit relative to the measurement object, determined by the position detection unit and temporally correlated to the output of the image sensor.The projected pattern may be a pattern of one or more laser lines, whereby the sensor unit may in particular be a laser line scanner.

[0020] The solution according to the invention can therefore dispense with the temporal variation of projected patterns or illumination lines, which were used according to the prior art to stimulate the generation of events by means of brightness changes in the event-based image sensor. Instead, the movement of the sensor unit during scanning is used for this purpose. The sensor unit can, for example, be guided by a device such as a coordinate measuring machine or a robot or can be handheld. The position of the sensor unit is preferably tracked internally or externally. By scanning the 3D surface contour of the measurement object to be detected, changes in the pattern imaged on the image sensor, in particular the laser line(s), arise solely through the movement of the sensor unit.

[0021] With conventional image sensors, this effect is rather undesirable, as the predetermined time grid during image acquisition usually leads to smearing effects that can reduce the resolution and accuracy of the 3D capture. To reduce such undesirable effects, the shortest possible image acquisition time is often aimed for. Reducing the movement speed, which also leads to a reduction, is avoided whenever possible, as it results in lower scanning performance. With event-based image sensors, however, this effect can be used to increase resolution and accuracy, as these image sensors also deliver more events when moving over areas with a more pronounced structure (i.e., higher rate of change), thus generating a larger number of measurement points.

[0022] The invention therefore proposes the use of an event-based image sensor in a sensor unit that is movable relative to the measurement object, for example, guided or handheld by a device (coordinate measuring machine, robot, or similar) and tracked internally or externally. The events provided by the image sensor and the position values ​​provided by the device or tracking system are correlated with each other in a time-correct manner to determine the three-dimensional shape of the measurement object. Suitable technology for this has been available for some time, e.g., synchronization through distributed clocks, etc. Changing the projection to specifically stimulate event generation is not necessary.

[0023] Compared to laser line scanners with global shutter image sensors, higher measurement accuracy during movement (i.e., faster permissible scanning speeds) and a larger number of measurable materials (greater dynamic range) can be achieved. Compared to designs with dynamic pattern projection, the setup is simpler and more cost-effective, as no variable projection units are required, and the computational effort is reduced.

[0024] According to a possible embodiment of the present invention, it is provided that the evaluation unit determines at least a partial area of ​​a pattern projected onto the measurement object based on the events detected by the image sensor and, based on the pattern, the three-dimensional position of points on the surface of the measurement object relative to the sensor unit along the pattern.

[0025] According to one possible embodiment of the present invention, the evaluation unit generates a complete internal two-dimensional representation of the pattern projected onto the measurement object in the image of the image sensor and / or a complete internal three-dimensional representation of the three-dimensional position of points on the surface of the measurement object relative to the sensor unit along the pattern. The representation is complete in the sense that it completely reproduces the previously acquired pattern or reproduces the three-dimensional position of points on the surface of the measurement object relative to the sensor unit along the entire previously acquired pattern, and not only where the pattern is currently recognizable.

[0026] This design takes into account that a complete image of the pattern may not be able to be determined at all times from the data generated by the event-based sensor, since in areas where the shape of the surface of the measurement object does not change relative to the sensor unit, the pattern in the image sensor also remains unchanged, so that no events are generated by which the pattern could be recognized. This is compensated for by the internal representation of the pattern and / or the three-dimensional representation of the three-dimensional position of points on the surface of the measurement object relative to the sensor unit along the pattern, which is maintained in the evaluation unit.

[0027] According to one possible embodiment of the present invention, it is provided that the complete internal two-dimensional representation of the pattern projected onto the measurement object in the image of the image sensor and / or a complete internal three-dimensional representation of the three-dimensional position of points on the surface of the measurement object is locally updated or supplemented based on the events detected by the image sensor and / or the 3D data determined from these. "Local" in the sense of the present invention does not refer to the coordinate system used, but rather to the limited extent of the referenced area within the pattern. Therefore, a local update or supplementation generally does not occur across the complete representation at all points, but only in sub-areas of the pattern.The internal representation is preferably updated or supplemented only in those sub-areas in which a sufficient number of events indicate a change in the pattern.

[0028] In one possible embodiment of the invention, initialization of the internal representation is omitted, as it is gradually built up from those areas in which the pattern influenced by the movement first appears on the object. For example, if the sensor unit is moved to a measuring position from which the object is to be measured, the internal representation is usually already fully established.

[0029] In one possible embodiment of the invention, however, the projection of the pattern can also be switched on during image acquisition to initialize the internal representation, so that when switched on, it generates events across its entire extension and is therefore recorded. In measurement mode, however, the projection remains statically switched on, and therefore a pattern is only recorded where it changes.

[0030] Therefore, if a change in the pattern is detected based on the events emitted by the image sensor, or if a change in the three-dimensional position of points on the surface of the measuring object is detected after converting the pattern into three-dimensional positions, the internal representation is locally updated or supplemented in the corresponding sub-areas. In areas where no change occurs and therefore no events are recorded, the data of the previously generated representation is retained or constantly updated.

[0031] In a first variant, the complete internal three-dimensional representation of the three-dimensional position of points on the surface of the measurement object is a representation in the coordinate system of the sensor unit along the pattern, i.e., the position of the points relative to the sensor unit along the pattern. In this case, this internal representation is updated locally where the pattern changes in the image sensor image and therefore reflects the changed position of the points on the surface of the measurement object relative to the sensor unit along the pattern.

[0032] In a second variant, the complete internal three-dimensional representation of the three-dimensional position of points on the surface of the measurement object is a representation in the coordinate system of the measurement object and preferably reflects the previously recorded areas of the surface of the measurement object. This representation is preferably supplemented locally where the surface geometry changes, which is indicated by a change in the pattern in the corresponding sub-area of ​​the image from the image sensor, and therefore supplements the previously recorded surface where the position of the points on the surface of the measurement object relative to the sensor unit changes along the pattern. In those areas in which no change is recorded, the geometry previously recorded along the pattern is preferably constantly updated in the direction of the relative movement between the sensor unit and the measurement object.

[0033] Preferably, the addition occurs more frequently in areas where a change in the surface geometry is detected than in areas where no change is detected. For example, the frequency of the addition based on a change can depend on the number of recorded events and / or a change rate, and in particular can increase with these, while the update in areas without change is performed, for example, based on a specified minimum resolution. Therefore, more points are generated in areas where the surface geometry changes than in areas without such a change.

[0034] According to one possible embodiment of the present invention, the frequency of updating and / or supplementation depends on the number of events registered by the image sensor and / or a rate of change of the pattern. This allows the accuracy of the detection to be adapted to the rate of change of the three-dimensional shape in the case of a change in the pattern caused by a change in the three-dimensional shape of the surface of the measurement object, thus increasing the accuracy of the detection.

[0035] According to one possible embodiment of the present invention, the frequency of the update or supplementation depends locally on the number of events registered by the image sensor and / or a rate of change of the pattern and / or the surface geometry, i.e., the local update rate depends on the local number of events registered by the image sensor and / or a rate of change of the pattern and / or the surface geometry. In particular, the frequency of the update or supplementation differs in different sub-regions of the pattern if the number of events registered by the image sensor and / or the rate of change of the pattern and / or the surface geometry differs between the sub-regions.

[0036] According to a possible embodiment of the present invention, it is provided that the frequency of the update is increased with increasing number of events and / or increasing rate of change.

[0037] In particular, the update rate increases in areas where the number of events registered by the image sensor and / or the rate of change of the pattern increases locally.

[0038] According to one possible embodiment of the present invention, the internal representation is read out in full and evaluated together with a position of the sensor unit relative to the measurement object, determined using the position detection unit, at the time the internal representation is read out to determine the absolute three-dimensional position of the points on the surface of the measurement object. This takes into account that, in order to determine the 3D geometry of the surface of the measurement object, in addition to the three-dimensional position of points on the surface of the measurement object relative to the sensor unit, the position of the sensor unit relative to the measurement object must also be known. Therefore, these two pieces of information must be temporally correlated.

[0039] Alternatively or additionally, it is intended that the internal representation already describes the absolute three-dimensional position of the points on the surface of the measuring object and is supplemented completely along the pattern and / or locally in those areas in which the surface changes along the pattern.

[0040] Regardless of the question of which type of representation is used and whether an internal representation is maintained at all, according to a possible embodiment of the present invention, a three-dimensional position of the points on the surface of the measurement object along the pattern is determined from the pattern captured in the image of the image sensor or the captured partial areas of the pattern and evaluated together with a time-correlated position of the sensor unit relative to the measurement object, which position detection unit determines, to determine the absolute three-dimensional position of the points on the surface of the measurement object.

[0041] According to one possible embodiment of the present invention, it is provided that the evaluation unit repeats the determination of the three-dimensional position of points on the surface of the measurement object relative to the sensor unit from at least a partial area of ​​the pattern and / or the determination of the absolute three-dimensional position of the points on the surface of the measurement object from the three-dimensional position of points on the surface of the measurement object relative to the sensor unit depending on the number of events registered by the image sensor and / or a rate of change of the pattern and / or the three-dimensional position of points on the surface of the measurement object relative to the sensor unit. Here, too, the calculation of the three-dimensional data is therefore made dependent on the rate of change of the three-dimensional shape of the surface of the measurement object, which is covered by the pattern.

[0042] According to one possible embodiment of the present invention, the frequency of the determination is increased with an increasing number of events and / or an increasing rate of change. This increases the accuracy for geometries with rapidly changing characteristics and reduces the required computing power for geometries with little change.

[0043] According to the present invention, the evaluation unit temporally correlates the three-dimensional position of the points on the surface of the measurement object relative to the sensor unit with a position of the sensor unit relative to the measurement object determined by the position detection unit, and determines an absolute three-dimensional position of the points on the surface of the measurement object from the three-dimensional position of points on the surface of the measurement object relative to the sensor unit and the position of the sensor unit relative to the measurement object. The absolute three-dimensional position of the points on the surface of the measurement object is, in particular, a position in the coordinate system of the measurement object.

[0044] According to one possible embodiment of the present invention, the position detection unit determines the position of the sensor unit relative to the measurement object at a fixed and / or predetermined sampling rate. The sensor unit of the present invention can therefore be used with position detection units known from the prior art, in which the position is detected at a fixed and / or predetermined sampling rate.

[0045] According to one possible embodiment of the present invention, the evaluation unit determines and / or reads the three-dimensional position of points on the surface of the measurement object relative to the sensor unit exactly at the time of position detection and uses this information together with the position to determine the absolute three-dimensional position of the points on the surface of the measurement object. Since, according to the invention, there are no longer any specific times at which the relative three-dimensional geometry along the pattern would have to be determined, the times can be selected so that they coincide with the times of position detection.

[0046] According to one possible embodiment of the present invention, the evaluation unit determines and / or reads the three-dimensional position of points on the surface of the measurement object relative to the sensor unit more frequently than the position detection unit determines and / or reads the position of the sensor unit relative to the measurement object. This takes advantage of the fact that more data is available, particularly in areas where the surface of the measurement object and thus the pattern changes, and therefore a higher frequency of determining the three-dimensional position of points on the surface of the measurement object relative to the sensor unit is possible.

[0047] According to a possible embodiment of the present invention, it is provided that between two successive times of detecting the position, the three-dimensional position of points on the surface of the measuring object relative to the sensor unit is determined and / or read out at a plurality of times and used together with an interpolated position to determine the absolute three-dimensional position of the points on the surface of the measuring object.

[0048] According to one possible embodiment of the present invention, the sensor unit is a laser line scanner, in which the projector projects one or more laser lines onto the measurement object in order to determine its three-dimensional shape based on the laser line appearing deformed in the image sensor image. In particular, the projector and the image sensor are rigidly mounted on a base at a projection angle.

[0049] According to one possible embodiment of the present invention, the position detection unit is an external tracking system. Such an external tracking system can be used in particular with handheld sensor units.

[0050] According to a possible embodiment of the present invention, it is provided that the external tracking system determines the position of the sensor unit based on the detection of reference points and / or reference objects arranged on the sensor unit.

[0051] According to one possible embodiment of the present invention, the position of the sensor unit is detected optically, in particular by triangulation. In particular, the external tracking system can comprise a plurality of optical sensors that detect reference points and / or reference objects arranged on the sensor unit. For example, the reference points and / or reference objects can be active optical markers.

[0052] According to a further possible embodiment of the present invention, the sensor unit is movable via a measuring arm of a coordinate measuring machine and / or via a robot, and the position detection unit determines the position of the sensor unit by determining the position of the measuring arm and / or the robot. In particular, the measuring arm is movable via three linear axes, in particular by motor drive. Alternatively or additionally, the sensor unit is pivotable via the measuring arm, in particular by means of a rotary joint arranged on the measuring arm.

[0053] According to one possible embodiment of the present invention, the position detection unit determines the position of the sensor unit relative to the measurement object based on the output of an image sensor. In particular, optical tracking is used to determine the position of the sensor unit, which can be internal or external tracking.

[0054] According to one possible embodiment of the present invention, the position detection unit determines the position of the sensor unit relative to the measurement object using reference points arranged on or in the vicinity of the measurement object. In particular, this allows internal tracking of the sensor unit by determining the position of the sensor unit relative to the reference points.

[0055] Reference points within the meaning of the present invention can be, for example, retroreflectors, active or passive markers, reference objects or circular reference points.

[0056] According to one possible embodiment of the present invention, the image sensor used by the position detection unit is rigidly mounted on the sensor unit. As a result, the coordinate system of the position detection unit is also rigidly mounted relative to the coordinate system of the sensor unit.

[0057] According to one possible embodiment of the present invention, the image sensor used by the position detection unit is an event-based image sensor. Such a sensor also offers advantages in position detection, since the detection accuracy can be at least maintained or even increased even when the sensor unit moves.

[0058] In particular, the image sensor used by the position detection unit is the image sensor of the sensor unit. This allows the structure of the sensor unit to be significantly simplified.

[0059] According to a possible embodiment of the present invention, it is provided that the evaluation unit generates an internal two-dimensional representation of the reference points or their position in the image of the image sensor, wherein the evaluation unit determines the three-dimensional position of the sensor unit relative to the measurement object from the two-dimensional position of the reference points in the image of the image sensor.

[0060] Since an event-based image sensor only provides data where the image changes during position detection, it is also useful to maintain an internal representation here.

[0061] According to a possible embodiment of the present invention, it is provided that the frequency of the determination of the three-dimensional position of the sensor unit relative to the measurement object by the image sensor used by the position detection unit depends on a number of events registered by the position detection unit and / or the image sensor used by the sensor unit and / or a rate of change of the pattern detected by the sensor unit and / or the position of the reference points in the image of the image sensor used by the position detection unit.

[0062] In particular, the frequency of position determination can be increased if the rate of change of the position of the sensor unit relative to the measurement object increases and / or if the rate of change of the pattern detected by the sensor unit increases and vice versa.

[0063] According to a possible embodiment of the present invention, it is provided that the frequency of determining the three-dimensional position of the sensor unit relative to the measurement object is synchronized with a reading and / or determination of a current three-dimensional position of points of the surface of the measurement object relative to the sensor unit.

[0064] The present invention further comprises a method for 3D measurement of a measurement object using a measuring system with a sensor unit comprising a projector and an event-based image sensor, and with an evaluation unit for determining 3D information from the output of the image sensor. For 3D measurement of a measurement object, a static pattern is projected onto the measurement object and the sensor unit is moved relative to the measurement object. The 3D measuring system comprises a position detection unit that detects the position of the sensor unit relative to the measurement object. The evaluation unit determines an absolute three-dimensional position of points on the surface of the measurement object from the output of the image sensor and the position of the sensor unit relative to the measurement object, determined by the position detection unit and temporally correlated to the output of the image sensor.

[0065] The method can preferably be carried out as already described above with regard to the inventive design of the 3D measuring system. In particular, the method can be carried out using a 3D measuring system as described above.

[0066] In one possible embodiment of the present invention, all of the steps and / or procedures described above or below, possibly with the exception of generating the relative movement between the sensor unit and the measurement object, which can also be generated manually, are performed, in particular automatically, by the sensor unit, the position detection unit, and / or the evaluation unit. In particular, these are designed and / or programmed to perform the corresponding functions and, in particular, the steps and / or procedures described above or below.

[0067] The 3D measurement system comprises, in particular, a controller with one or more microprocessors and software stored in one or more memory areas. The software, when running on the microprocessor(s), implements the functions described above and below. The controller communicates with the image sensor and evaluates the events generated by the image sensor accordingly.

[0068] If the sensor unit is moved by a motor, the controller preferentially controls the drives used to move the sensor unit to automatically execute a measurement program for measuring the object being measured, in which the pattern projected by the projector sweeps over the surface of the object being measured. With a handheld sensor unit, however, the sensor unit is moved by an operator along the surface of the object being measured, causing the pattern to sweep over it.

[0069] The present invention will now be described in more detail with reference to embodiments and drawings.

[0070] Showing: Fig. 1: a first embodiment of a measuring system according to the invention when carrying out an embodiment of a method according to the invention, Fig. 2a and Fig. 2b: which is achieved by using a global shutter sensor according to the state of the art ( Fig. 2a) and in an event-based sensor according to the present invention ( Fig. 2b) generated data, Fig. 3: a first variant of a measuring system according to the invention, in which the sensor unit is arranged on a coordinate measuring machine and moved over it, Fig. 4: a second variant of a measuring system according to the invention, in which the sensor unit is hand-held and the position of the sensor unit is detected by means of an external tracking system and Fig. 5: a third variant of a measuring system according to the invention, in which the sensor unit is hand-held and determines its position based on reference points on or around the measuring object.

[0071] Fig. Figure 1 shows an embodiment of a 3D measuring system according to the invention with a sensor unit 10 comprising a projector 12 and an image sensor 11. The projector 12 projects a static pattern onto the measurement object 20, which is captured by the image sensor 11 in order to determine the three-dimensional shape of the measurement object from the pattern.

[0072] In the exemplary embodiment, the pattern emitted by the projector 12 is at least one laser line 15. The sensor unit therefore functions as a laser line scanner. Instead of a single laser line, the sensor unit can also emit a plurality of laser lines or more complex patterns.

[0073] The 3D geometry of the measurement object along the pattern, in the exemplary embodiment along the laser line 15, is determined by triangulation. Based on the geometry of the pattern deformed by the three-dimensional surface of the measurement object 20 and the known projection angle between the projector and the image sensor 11, the three-dimensional position of the points on the surface of the measurement object along the pattern, in particular along the lines of the pattern, is determined, taking into account the respective imaging optics of the projector and image sensor.

[0074] To measure the measurement object 20 or the areas of interest on its surface, the sensor unit 10 is moved relative to the measurement object 20 so that the pattern 15 sweeps over the surface of the measurement object. For this purpose, the sensor unit 10 is preferably moved while the measurement object 20 remains stationary. However, the reverse case, in which the measurement object 20 is moving and the sensor unit remains static, is also conceivable.

[0075] In order to determine the three-dimensional shape of the measurement object from the three-dimensional position of the points on the surface of the measurement object along the pattern, which is determined locally in the coordinate system of the sensor unit 10, the relative position between the sensor unit and the measurement object must be known. For this purpose, the 3D measuring system further comprises a position detection unit that detects the position of the sensor unit relative to the measurement object. Exemplary embodiments of this are described in more detail below.

[0076] From the three-dimensional position of the points along the pattern in the sensor coordinate system determined by the sensor unit and the position of the sensor unit determined by the position detection unit and correlated therewith in time, the surface of the measuring object in an absolute coordinate system, in particular in the rest coordinate system of the measuring object 20, can be composed from the three-dimensional positions of the surface of the measuring object along the pattern relative to the sensor unit determined when the measuring object is scanned.

[0077] The advantages and specifics of using an event-based sensor are now explained using Fig. 2a and Fig. 2b, where Fig. 2a represents the use of a sensor according to the state of the art, Fig. 2b, on the other hand, shows the use of an event-based sensor according to the invention.

[0078] Fig. 2a shows the images of the pattern generated by a global shutter according to the prior art at times t1 - t3, wherein the image of the part 16 of the laser line 15 falling onto the ramp 21 of the measurement object 20, which part changes its position within the image due to the three-dimensional shape of the measurement object during the movement, is blurred by this movement. Fig. Figure 2b, on the other hand, shows the events 18, which are generated in the sensor image due to the movement of the laser line hitting the measuring object and which reflect the exact position of the laser line in space.

[0079] While the speed of the change in the shape of the laser line detected by the image sensor caused by the relative movement between the sensor unit and the measurement object was problematic according to the prior art because it reduced the accuracy of the 3D detection, in the event-based sensor according to the invention it only leads to the laser line becoming visible and creates an increased data density and thus improved resolution in areas with a higher rate of change in the geometry.

[0080] Since, according to the invention, a static pattern is projected which would not change in the image of the image sensor without a relative movement between the sensor unit 10 and the measurement object 20, the pattern is not necessarily recognized over its entire extent by the recorded events, even with such a relative movement. Fig. 2b, the pattern is not imaged in those areas to the right and left of the ramp 21 of the measurement object 20, since the constant position of the surface of the measurement object or the base lying next to the ramp 21 does not change the relative position between the surface and the sensor unit. In a sensor according to the prior art, as shown in Fig. 2a, the sections of the pattern to the right and left of ramp 21 are also clearly visible and, due to the static relative position, are depicted more sharply than the sections on ramp 21.

[0081] For this reason, the state of the art also assumed that event-based image sensors can only be used in combination with a dynamic pattern projector, i.e. a projector whose pattern changes in the image of the image sensor even when the sensor unit and the measuring object are arranged statically to each other, since only then can the pattern projected onto the measuring object actually be captured over its entire extent.

[0082] However, the inventor of the present invention has recognized that the advantages of an event-based sensor can also be used in a 3D measurement system with a statically projected pattern, and in contrast, there are even significant advantages compared to dynamic pattern projection.

[0083] In particular, the inventor recognized that for the 3D capture of a measurement object, it is not necessary to be able to constantly recognize the projected pattern across its entire extent in the sensor image, as is the case with an image sensor with a global shutter approach or a dynamic pattern projection. On the contrary, with a prior art approach, unnecessary computing power is consumed by the complete capture and processing of the pattern, even in areas where it does not change. Because an event-based image sensor in combination with a static pattern projector only generates events where the image of the pattern in the image sensor changes due to the change in the three-dimensional shape of the surface of the measurement object being scanned, data is only generated where a relevant change has actually occurred.In contrast, the data that is completely determined in the state of the art but is partly superfluous for determining the three-dimensional surface is not even generated.

[0084] In addition, the sensor according to the invention generates a particularly large amount of data precisely where the surface of the measurement object changes significantly, enabling particularly accurate detection of the three-dimensional surface. However, with dynamic pattern projection, all of these advantages of an event-based image sensor could not be utilized, since dynamic pattern generation generates events in the respective area completely and independently of the shape of the measurement object.

[0085] In order to achieve a complete capture of the three-dimensional shape of the measurement object despite the only local generation of events in areas of the measurement object where the three-dimensional shape of the surface changes, an instance is required within the evaluation system for the three-dimensional reconstruction of the entire surface of the measurement object that determines the three-dimensional shape of the measurement object relative to the sensor unit completely along the pattern. For this purpose, the two variants described below are provided, which can also be used in combination.

[0086] In a first variant, the evaluation system creates an internal representation of the pattern captured by the image sensor. This can be a data set that reproduces the pattern across its entire previously captured extent in a two-dimensional image. This internal representation is continuously updated locally based on the events captured by the image sensor in the areas where the pattern in the image sensor changes and therefore generates events. This ensures that the internal representation completely reproduces the pattern mapped to the image sensor, even if the image sensor only outputs local events. In areas where no current events are present, however, the existing internal representation of the pattern is retained.

[0087] The internal representation of the pattern is evaluated to determine the three-dimensional position of the surface of the measuring object along the pattern relative to the sensor unit.

[0088] In one possible embodiment, this evaluation is carried out completely, so that the entire pattern acquired so far is used to determine the three-dimensional position of the surface of the measurement object along the pattern relative to the sensor unit. Alternatively, only those areas of the pattern that have been updated are used to determine the current three-dimensional position of the surface of the measurement object along the pattern relative to the sensor unit, and in particular, to update this position.

[0089] In a second variant, the evaluation system generates an internal, complete representation of the three-dimensional position of the points on the surface of the measurement object relative to the sensor unit along the previously recorded pattern. This can be a data set that three-dimensionally reproduces the three-dimensional position of the points on the surface of the measurement object relative to the sensor unit along the previously recorded pattern over the entire extent of the pattern already recorded. This complete representation of the three-dimensional position of the points on the surface of the measurement object relative to the sensor unit is only updated locally in those areas in which a change in the pattern and thus a change in the three-dimensional position of the surface relative to the sensor unit is detected.

[0090] This variant therefore does not require a complete representation of the pattern. Rather, those areas of the pattern that are currently being captured by the event-based image sensor due to their change can be used to determine the three-dimensional position of the surface relative to the sensor unit along the currently captured area of ​​the pattern, and this three-dimensional data can be used to locally update the complete representation.

[0091] However, a combination of the two variants is also possible. In particular, even when maintaining a complete representation of the pattern, only the newly updated regions of the pattern can be used to update the complete representation of the three-dimensional position of the surface relative to the sensor unit along these regions.

[0092] Furthermore, it is conceivable to determine the three-dimensional position of the surface relative to the sensor unit along the pattern alternately completely and locally, provided that a complete representation of the pattern is available.

[0093] In a third variant, the evaluation system generates an internal, complete representation of the three-dimensional position of the points on the absolute surface of the measurement object in the previously recorded area of ​​the measurement object. This can be a data set that three-dimensionally reproduces the absolute three-dimensional position of the points on the surface of the measurement object in the areas of the measurement object previously covered by the pattern. This complete representation of the absolute three-dimensional position of the points on the surface of the measurement object relative to the sensor unit is preferably supplemented locally in those areas in which a change in the pattern and thus a change in the three-dimensional position of the surface relative to the sensor unit is detected.At specified times, the absolute three-dimensional position of the points on the surface of the measuring object can be constantly updated in those areas where no change in the pattern, and thus no change in the three-dimensional position of the surface relative to the sensor unit, was detected in order to maintain a specified minimum data density. In those areas where a change is detected, however, an increased data density results.

[0094] The third variant also does not require a complete internal representation of the three-dimensional position of the surface relative to the sensor unit and can constantly update the three-dimensional positions in the direction of movement, particularly in those areas where there is no change and therefore no pattern is detected.

[0095] However, the third variant can also be combined with the first and / or second variant, whereby the updating occurs, for example, by updating the internal complete representation of the three-dimensional position of the points of the absolute surface of the measurement object over the entire extent of the pattern at predetermined intervals based on a complete internal representation of the pattern and / or the three-dimensional position of the surface relative to the sensor unit along the pattern. Between these predetermined intervals, supplementation preferably occurs only where a change is detected.

[0096] In all described variants, the frequency of updating and / or supplementation preferably depends on the number of recorded events or, equivalently, on the rate of change of the pattern and / or the three-dimensional position of the points. In particular, the frequency can be increased if the number of recorded events and / or the rate of change increases and vice versa. The frequency of updating is preferably determined locally based on the locally recorded events and / or the local rate of change of the pattern and / or the three-dimensional position of the points. In areas with a smaller number of recorded events and / or a lower rate of change, the update therefore occurs less frequently than in areas with a larger number of recorded events and / or a larger rate of change.

[0097] This allows the spatial resolution to be increased in first areas where there is a higher rate of change in the direction of the relative movement between the measurement object and the sensor unit, and reduced in second areas where there is a lower rate of change in the direction of the relative movement between the measurement object and the sensor unit. This ensures consistent quality of 3D acquisition, even with changing geometries of the measurement object, while optimally utilizing the available resources.

[0098] In order to determine the three-dimensional shape of the entire surface of the measuring object to be examined from the data on the three-dimensional position of the surface relative to the sensor unit along the pattern, the data on the three-dimensional position of the surface relative to the sensor unit must be combined with temporally corresponding data on the position of the sensor unit relative to the measuring object and evaluated together in order to determine the absolute position of the surface of the measuring object in its own rest coordinate system.

[0099] The way in which such a temporal correlation can be achieved also depends on the type of position detection unit used.

[0100] In a first variant, the position detection unit detects the position of the sensor unit relative to the measuring object at a predetermined frequency, for example a measuring cycle and / or a frame rate of a video camera with global shutter in the case of optical detection.

[0101] In this case, it is possible, on the one hand, to evaluate and / or read out the three-dimensional position of the surface relative to the sensor unit always at the times at which the position is detected, in particular to evaluate and / or read out completely along the pattern.

[0102] Alternatively or additionally, the three-dimensional position of the surface relative to the sensor unit can also be evaluated and / or read between the times at which the position is recorded, and used together with an interpolated position value to determine the absolute position of the points. This allows the spatial resolution in the direction of the relative movement to be flexibly adjusted, and especially locally increased, in those areas where the pattern or surface of the measurement object changes, independent of the frequency of the position determination.

[0103] In a second variant, the frequency of position detection is variable. There are several options here: In a first sub-variant, the frequency of position detection depends on the number of recorded events and / or a rate of change of the pattern and / or the three-dimensional position of the points on the surface of the measurement object along the pattern relative to the sensor unit, and increases in particular with increasing number and / or rate of change. The frequency of position detection can be changed based on a complete criterion relating to the entire image or pattern, for example, a maximum of the local number and / or local rate of change across the entire image and / or pattern, and / or an average across the entire image and / or pattern.

[0104] In a second sub-variant, the frequency of position detection depends on the rate of change of the position. Specifically, the frequency of position detection increases with increasing rate of change of the position, and vice versa. This improves the spatial resolution.

[0105] Several different configurations are conceivable for generating the relative movement of the sensor unit to the measuring object and for detecting this relative movement. Three exemplary configurations are described below using the Fig. 3 to 5.

[0106] In the embodiment in Fig. 3, the sensor unit 10 is arranged on the measuring arm 61 of a coordinate measuring machine 60 and is linearly movable along its movement axis(es). The position detection unit 30 detects the position of the movement axis(es) of the coordinate measuring machine 60 and uses this to determine the position in an absolute coordinate system of the measuring machine, which preferably corresponds to the coordinate system of the measuring object.

[0107] In this exemplary embodiment, the evaluation unit 40 specifies a measurement cycle for the position detection unit 30 and reads the position of the sensor unit 10 at predetermined times. The sensor unit uses the event-based image sensor to detect the pattern projected onto the measurement object and also generates synchronization information to temporally synchronize the positions of the surface of the measurement object relative to the sensor unit, determined based on the events, with the position information.

[0108] In the examples in Fig. 4 and Fig. 5, the sensor unit 10 is hand-held and its position is controlled externally ( Fig. 4) or internally ( Fig. 5) tracked.

[0109] When designing according to Fig. 4, an external tracking unit 30 is provided, which detects the position of the sensor unit 10. The tracking unit 30 has one or more sensors 31, which detect reference points 25 arranged on the sensor unit and use them to determine the position of the sensor unit. The sensors are preferably optical sensors, in particular linear or planar image sensors, which detect the position of the reference points 25 and determine the position of the sensor unit by means of triangulation. The reference points 25 can, in particular, be several passive and / or active markers and / or reference objects and / or retroreflectors.

[0110] When designing according to Fig. 5, internal tracking of the sensor unit 10 is provided based on reference points 25, which are arranged on and / or around the measurement object 20. Here, too, the detection of the reference points 25 is preferably carried out via one or more optical sensors, in particular linear or planar image sensors, which are arranged on the sensor unit 10. In particular, the image sensor 11, which is also used to detect the pattern, can be used to detect the reference points 50. Alternatively, an additional sensor can be used, in particular a sensor with a larger measuring range and / or image angle than the image sensor 11, which is also used to detect the pattern.

[0111] The optical sensor used for internal or external tracking can be a conventional image sensor with a global shutter, which outputs an image at a specified clock rate and with which the position information is also determined at a specified clock rate.

[0112] Preferably, the optical sensor used for internal or external tracking is also an event-based image sensor. With external tracking, an event-based image sensor allows for particularly simple separation of the sensor unit moving relative to the measurement object from the stationary background. Furthermore, an event-based sensor also offers similar advantages for position determination as for pattern detection. In particular, the event-based sensor allows the frequency of position detection to be dependent on the rate of change of the position. Preferably, the frequency of position detection increases with the rate of change of the position.The rate of change of the position can be determined as the number of events captured by the image sensor, or as the rate of change of the resulting two-dimensional image of the reference points, or the resulting two-dimensional position of the reference points in the image sensor image. This rate can be used to determine the frequency of position detection, i.e., the calculation of the position of the sensor unit from the position of the reference points in the image sensor image. Since an event-based sensor also generates more events with faster movement, it allows for significantly improved spatial resolution, especially with fast scanning movements of the sensor unit, whereas conventional sensors also lose accuracy here.

[0113] Whenever reference is made to a position of the sensor unit relative to the measurement object in the context of this application, this preferably refers to the pose of the sensor unit, i.e., the position and orientation of the sensor unit in three-dimensional space. However, the position detection unit does not necessarily have to determine all six fundamental degrees of freedom of this pose. Rather, several of these degrees of freedom, for example, the orientation, can be determined by the arrangement of the sensor unit on a device such as the measuring arm of a coordinate measuring machine.

[0114] The sensor unit can also be mounted on a robot arm. In particular, the robot arm can have multiple rotational axes. Preferably, it is a robot arm of an industrial robot with at least 6 axes. The position of the sensor unit can be determined via the axis position of the robot arm and / or by internal or external tracking using optical sensors.

Claims

[1] 3D measuring system for 3D measurement of a measuring object, with a sensor unit comprising a projector and an event-based image sensor, and with an evaluation unit for determining 3D information from the output of the image sensor, wherein the projector projects a static pattern and the sensor unit is movable relative to the measurement object, wherein the 3D measuring system comprises a position detection unit which detects the position of the sensor unit relative to the measurement object, wherein the evaluation unit determines an absolute three-dimensional position of points on the surface of the measurement object from the output of the image sensor and the position of the sensor unit relative to the measurement object determined by the position detection unit and temporally correlated to the output of the image sensor, wherein the evaluation unit correlates the three-dimensional position of the points of the surface of the measuring object relative to the sensor unit in time with a position of the sensor unit relative to the measuring object determined by the position detection unit and determines an absolute three-dimensional position of the points of the surface of the measuring object from the three-dimensional position of points of the surface of the measuring object relative to the sensor unit and the position of the sensor unit relative to the measuring object. [2] 3D measuring system according to claim 1, wherein the evaluation unit uses the events detected by the image sensor to determine at least a partial area of ​​a pattern projected onto the measurement object and uses the pattern to determine the three-dimensional position of points on the surface of the measurement object relative to the sensor unit along the pattern. [3] 3D measuring system according to one of the preceding claims, wherein the evaluation unit generates a complete internal two-dimensional representation of the pattern projected onto the measurement object in the image of the image sensor and / or a complete internal three-dimensional representation of the three-dimensional position of points on the surface of the measurement object, which is preferably locally updated or supplemented based on the events detected by the image sensor and / or 3D data determined therefrom. [4] 3D measuring system according to claim 3, wherein the frequency of the update or addition depends on the number of events registered by the image sensor and / or a rate of change of the pattern, wherein preferably the frequency of the update or addition is increased with increasing number of events and / or increasing rate of change. [5] 3D measuring system according to claim 3 or 4, wherein the internal representation is read out completely and evaluated together with a position of the sensor unit relative to the measurement object determined using the position detection unit at the time of reading the internal representation to determine the absolute three-dimensional position of the points of the surface of the measurement object, and / or wherein the internal representation describes the absolute three-dimensional position of the points of the surface of the measurement object and is supplemented completely along the pattern and / or locally in those areas in which the surface changes along the pattern. [6] 3D measuring system according to one of the preceding claims, wherein the evaluation unit repeats the determination of the three-dimensional position of points of the surface of the measurement object relative to the sensor unit from at least a partial area of ​​the pattern and / or the determination of the absolute three-dimensional position of the points of the surface of the measurement object from the three-dimensional position of points of the surface of the measurement object relative to the sensor unit as a function of the number of events registered by the image sensor and / or a rate of change of the pattern and / or the three-dimensional position of points of the surface of the measurement object relative to the sensor unit, wherein the frequency of the determination is preferably increased with an increasing number of events and / or an increasing rate of change. [7] 3D measuring system according to one of the preceding claims, wherein the position detection unit determines the position of the sensor unit relative to the measuring object with a fixed and / or predetermined sampling rate, wherein preferably the evaluation unit determines and / or reads out the three-dimensional position of points of the surface of the measuring object relative to the sensor unit exactly at the time of the position detection and uses it together with the position to determine the absolute three-dimensional position of the points of the surface of the measuring object. [8] 3D measuring system according to one of the preceding claims, wherein the evaluation unit determines and / or reads out the three-dimensional position of points on the surface of the measuring object relative to the sensor unit more frequently than the position detection unit determines and / or reads out the position of the sensor unit relative to the measuring object, wherein in particular between two successive times of detecting the position at a plurality of times the three-dimensional position of points on the surface of the measuring object relative to the sensor unit is determined and / or read out and used together with an interpolated position to determine the absolute three-dimensional position of the points on the surface of the measuring object. [9] 3D measuring system according to one of the preceding claims, wherein the sensor unit is a laser line scanner. [10] 3D measuring system according to one of the preceding claims, wherein the position detection unit is an external tracking system which preferably determines the position of the sensor unit based on the detection of reference points arranged on the sensor unit, wherein the detection of the position of the sensor unit is preferably carried out optically, in particular by triangulation. [11] 3D measuring system according to one of the preceding claims, wherein the sensor unit is movable via a measuring arm of a coordinate measuring machine and / or via a robot and the position detection unit determines the position of the sensor unit by determining the position of the measuring arm and / or robot. [12] 3D measuring system according to one of the preceding claims, wherein the position detection unit determines the position of the sensor unit relative to the measuring object based on the output of an image sensor, in particular based on reference points arranged on or in the surroundings of the measuring object, wherein the image sensor is preferably arranged rigidly on the sensor unit. [13] 3D measuring system according to claim 12, wherein the image sensor is an event-based image sensor, in particular the image sensor of the sensor unit, wherein the evaluation unit preferably generates an internal two-dimensional representation of the reference points or their position in the image of the image sensor, wherein the evaluation unit determines the three-dimensional position of the sensor unit relative to the measurement object from the two-dimensional position of the reference points in the image of the image sensor, wherein the frequency of determining the three-dimensional position of the sensor unit relative to the measurement object preferably depends on a number of events registered by the image sensor and / or a rate of change of the pattern and / or the position of the reference points in the image of the image sensor and / or is synchronized with a reading and / or determination of a current three-dimensional position of points of the surface of the measurement object relative to the sensor unit. [14] Method for 3D measurement of a measurement object using a measuring system with a sensor unit comprising a projector and an event-based image sensor, and with an evaluation unit for determining 3D information from the output of the image sensor, in particular using a 3D measuring system according to one of the preceding claims, characterized bythat for the 3D measurement of a measurement object, a static pattern is projected onto the measurement object and the sensor unit is moved relative to the measurement object, wherein the 3D measuring system comprises a position detection unit which detects the position of the sensor unit relative to the measurement object, wherein the evaluation unit determines an absolute three-dimensional position of points on the surface of the measurement object from the output of the image sensor and the position of the sensor unit relative to the measurement object, which position is determined by the position detection unit and is correlated in time to the output of the image sensor,wherein the evaluation unit temporally correlates the three-dimensional position of the points on the surface of the measuring object relative to the sensor unit with a position of the sensor unit relative to the measuring object determined by the position detection unit and determines an absolute three-dimensional position of the points on the surface of the measuring object from the three-dimensional position of points on the surface of the measuring object relative to the sensor unit and the position of the sensor unit relative to the measuring object.

Citation Information

Patent Citations

  • Structured light three-dimensional measurement method based on event camera

    CN112525107A

  • Coordinate measuring system

    DE102018127221A1

  • Dynamic vision sensor and projector for depth mapping

    DE102018129009A1

  • Method and measuring device for measuring an object using a measuring device with a light source and a sensor unit having a plurality of detection pixels

    DE102021212654A1

  • Method and appliance for optical detection of surface contrast line employing laser scanner

    DE19855478A1