System and method for three-dimensional scanning of object

Abstract

The invention relates to a 3D scanning of an object (200) to determine a location (d201) of a feature (201) of the object (200) in at least a z-direction. The method applies a DVS camera whose focal length varies according to a predefined pattern during the scan. The resulting imaging data IMA comprises an event signal SIG generated by a given sensor element SE of a sensor chip (111) of the camera (110), the event signal SIG generating an identification ID of the sensor element SE, and a timestamp TS representing a point in time t0 at which the corresponding event signal SIG has been generated and evaluated to determine the location (d201).

Description

[0001] The present invention relates to camera-based scanning of objects in three dimensions.

[0002] Three-dimensional (3D) measurement from a fixed viewing direction using a single camera is useful for measuring the 3D shape of an object in various applications such as computer vision and robotics, where speed and versatility are key elements of measurement in the context of tasks such as inspection and detection.

[0003] 3D measurement of a scene typically applies any kind of stereoscopic imaging principle. One known stereoscopic imaging principle involves two cameras viewing a scene from different viewing directions, where the two cameras can be two physical cameras at different poses imaging simultaneously, or one camera positioned at different poses at different times. However, this principle is not well suited for moving objects. Another approach is to apply an active triangulation system to the scene using one camera and structured lighting, instead of using two cameras. The system takes a series of images with different illumination patterns in order to evaluate depth information from the image stack.

[0004] Another way to capture depth information is to use image stacks created by one camera or imaging system by taking images at different nominal focus positions and by evaluating the intensity variation of each pixel or binned area in the image for the maximum or minimum intensity reading depending on the instantaneous defocus. The extreme intensities can be assigned to the physical location of the real object surface points in 3D space. Taking a focus series takes a considerable time, since the number of images in the stack depends on the spatial separation and the depth range to be covered. Therefore, this principle is also less suitable for moving objects due to its high time requirements. Additionally, the measurement generates a relatively large amount of data to be processed.

[0005] Other 3D measurement principles, such as color (confocal) sensing or laser triangulation probes, suffer from the fact that they cannot capture object height information in the region of interest in parallel. Scanning in one or even two directions would be required, and therefore makes these techniques less suitable for many applications.

[0006] Therefore, a solution is required which allows fast and precise scanning of objects in three dimensions. This is achieved by the method proposed in claim 1 and the scanning system according to claim 11 .

[0007] A method SCMETH for determining the location of an object feature relative to a camera reference location (e.g., the surface of a sensor chip of the camera) based on a camera, assuming that the object is located in the field of view of the camera, which extends from the camera in the z direction. The camera applied in the method is a DVS camera comprising a sensor chip, the sensor chip having a plurality of sensor elements SE(j), i.e., pixels, where i=1,...,I, and I represents the total number of sensor elements of the sensor chip. The camera is configured to generate imaging data DAT(j) including event signals SIG(j) for each sensor element SE(j) individually (i.e., pixel by pixel). Each event signal is selected from a predefined group GRP of event signal types, which group GRP may include at least positive events and negative events. An event signal is only generated if the light flux or light intensity to the corresponding sensor element SE(j) changes by more than a given threshold value THRES (e.g., 15%), while no event signal is generated if the flux change is less than the threshold value THRES. The method SCMETH comprises a scanning stage SCAN during which the focal length f(t) of the optical segment of the DVS camera is varied in the z direction according to a predefined time pattern f(t). It can therefore be assumed that for each time point t in the variation range between a minimum value fmin and a maximum value fmax, the instantaneously applied focal length f(t) as well as the object distance g(t) and the image distance b(t) are known, for example according to the lens manufacturer equation 1 / f=1 / g+1 / b. Furthermore, the method comprises an evaluation stage EVAL during which any one or more imaging data DAT(j) generated by the DVS camera during the focal length variation of stage SCAN are evaluated to determine at least the z position d201 of the feature in the z direction.

[0008] Therefore, the special properties of a DVS camera described below are beneficially applied to determine at least the z-position of a feature of interest.

[0009] Each imaging data DAT(j) transmitted from the DVS camera to the evaluation system for further processing in the evaluation level EVAL comprises the generated event signal SIG(j) itself, being either positive DOS or negative NEG, an exact timestamp TS(j) representing the point in time t at which the sensor element SE(j) generated the corresponding event signal SIG(j), and an unambiguous identifier ID(j) of the corresponding individual event signal SIG(j) generating sensor element SE(j), said unambiguous identifier ID(j) representing the positioning of the corresponding sensor element SE(j) on the sensor chip of the DVS camera.

[0010] Based on the time point t0, determine the z - positioning d201 to be determined for the feature based on the instantaneous focal length f(t0) and / or the associated g(t0) and / or b(t0). The time point t0 is selected to represent the time when one or more event signals SIG(j) are generated by the sensor element SE(j) during the focal length change and / or the time when the temporal and / or spatial pattern of the sensor element SE(j) correspondingly satisfies a predefined condition or corresponds to a predefined pattern.

[0011] Among them, in the simplest case, the "temporal and / or spatial pattern" in which the event signal SIG(j) is generated by the sensor element SE(j) stipulates that one of the sensor elements SE(j) must generate an event signal at some time point t0: In this case, it is assumed that the predefined condition is satisfied when any event signal type of event signal SIG(j), such as a positive event POS or a negative event NEG, is generated at the time point tp, where t0 is selected to be the same as the time point tp, that is, t0 = tp.

[0012] In a more detailed case, the pattern can be a temporal pattern, which can consider the same sensor element SE(j) over time and observe the temporal development of the even signal SIG(j) generated by this sensor element SE(j). The spatial pattern can consider different sensor elements SE(j1), SE(j2) and observe the event signals SIG(j1), SIG(j2) generated by them. A hybrid pattern that is both a spatial pattern and a temporal pattern can consider different sensor elements SE(j1), SE(j2) over time t.

[0013] In a more complex scenario, for the first of the sensor elements SE(j), a first sequence SEQ(j) including one or more event signals SIG(j) of a first event signal type (such as a positive event POS) is generated between the time points t1 and t2, where t2≥t1, and for the sensor element SE(j’) that is explicitly (i.e., in a given neighborhood) associated with the first sensor element SE(j) among the multiple sensor elements SE(j), a second sequence SEQ(j’) including one or more event signals SIG(j’) of a second event signal type (such as a negative event NEG) is generated between the time points t3 and t4, where t4≥t3 and t3>t2, and t0 is selected in the range of t2 < t0 < t3, especially t0=(t2 + t3) / 2. This should include the case where one or both of the sequences SEQ include only one event signal.

[0014] A sensor element SE(j') that is "explicitly related" to the first sensor element SE(j) can be the same as the first sensor element SE(j), i.e., SE(j') = SE(j). Due to the effects that will be exemplarily explained below in the context of the "first effect" in the first operating mode, the expression "explicitly related" also applies to sensor elements SE(j), SE(j') that are not the same but are at a finite, predefined distance DIST from each other.

[0015] In short, in the case where both of these sensor elements are located within a predefined distance DIST from each other on the sensor chip, the sensor elements SE(j), SE(j') are explicitly related to each other, where the predefined distance DIST is determined based on the rate of change of the focal length f(t) during a change in the focal length.

[0016] It is possible that the projection of the feature is not limited to only one sensor element, but it can extend across a region covering multiple sensor elements on the sensor chip. In such a scenario, it can be assumed that a predefined condition is met when, for one or more sensor elements SE in a first region on the sensor chip, a first event signal SIG(j) of a first event signal type (e.g., positive event POS) is generated between time points t1 and t2, where t2 ≥ t1, and for one or more sensor elements SE in a second region on the sensor chip that is explicitly related (i.e., within a given neighborhood) to the first region, a second event signal SIG(j') of a second event signal type (e.g., negative event NEG) is generated between time points t3 and t4, where t4 ≥ t3 and t3 > t2, and where t0 is selected in the range t2 < t0 < t3, in particular t0 = (t2 + t3) / 2.

[0017] For example, in the case where the first region and the second region overlap or are adjacent to each other, i.e., in the case where at least one sensor element in the first region is adjacent to at least one sensor element in the second region, the first region and the second region are explicitly related to each other. "Adjacent" means that there are basically no further sensor elements located between those "adjacent" sensor elements.

[0018] The z - positioning d201 of the feature can be determined as the distance of the instantaneous focus of the optical segmentation from the surface of the sensor chip at the determined time point t0, where the instantaneous distance of the focus is derived from the instantaneous focal length f(t0) according to the selected time point t0. In the case where the optical segmentation and its properties - including their behavior over time due to the change in the focal length f(t) and the associated b(t) and g(t) - are known, the distance d201 can be calculated as d201 = b(t0)+g(t0).

[0019] In a first operating mode, the optical segment is configured to perform a focus change according to a predefined pattern f(t) in time, so that its focus changes, while its focus position and the focal plane move accordingly in the z direction due to the focus change, so that the clarity of the projection of a given feature onto the sensor chip changes during the focus change.

[0020] In a second operating mode, the optical segment is configured to perform a focus change according to a predefined pattern f(t) in time, so that its focal length changes, while its focus position and the focal plane accordingly do not move in the z direction, but remain constant at a selected position in the z direction, for example at a predefined distance from the main plane, so that during the focus change, the clarity of the projection of the feature at the selected position in the z direction onto the sensor chip remains constant, while the magnification of the projection changes with the focus change.

[0021] In a second operating mode, the stage SCAN may be executed multiple times, wherein different constant focus positions in the z-direction are selected for different executions of the stage SCAN. Furthermore, for each execution of the stage SCAN, the positions POS(j,t) of those sensor elements SE(j) on the sensor chip which successively generate event signals SIG(j) during a change in focus corresponding to the projection of the feature onto the sensor chip may be observed over time t to determine a lateral propagation velocity VEL(PCS(j,t),t) of the projection of the feature across the sensor chip due to the change in focus, wherein a representation of the z-position d201 to be determined for the feature (201) is estimated based on the lateral propagation velocity VEL(POS(j,t),t).

[0022] In addition to the z positioning d201, the lateral positioning x201, y201 of the feature 201 in the x and y directions is derived from the positioning POS(j) of the sensor element SE(j) generated from the event signal SIG(j) on the sensor chip 111, where the positioning POS(j) is derived from the identifier ID(j) of the sensor element SE(j) generated from the corresponding individual event signal SIG(j).

[0023] A corresponding scanning system for camera-based determination of the positioning of an object feature relative to a camera reference position (e.g., a surface of a sensor chip of the camera) comprises a DVS camera having a field of view for scanning the object, the field of view extending from the camera in the z-direction. The DVS camera comprising a sensor chip having a plurality of sensor elements SE(j) is configured to generate imaging data DAT(j) comprising an event signal SIG(j) from a group GRP of event signal types for each sensor element SE(j) individually in case a change in the light flux onto the respective sensor element SE(j) exceeds a given threshold value THRES. The scanning system comprises an evaluation system configured to control an optical segmentation of the DVS camera for a change in the focal length and to receive and evaluate the imaging data DAT(j) for the determination of the positioning. In particular, the evaluation system is configured to perform the method as described above.

[0024] The present invention advantageously implements a camera that applies the principle of a "dynamic vision system" (DVS). Typically, such DVS cameras, also called "event cameras" or "neuromorphic cameras", are full-field-of-view cameras with sensor chips having an aspect ratio of 4:3 or 16:9, etc., and are currently available in the market with >10 6 The DVS camera is available in versions with individual sensor elements (e.g. Prophesee's "MetaVision" sensor with Sony chip). The basic properties of the DVS camera applied in the solution described in this article include that the DVS camera detects events, i.e. changes in the light flux or intensity, respectively exceeding a predefined threshold THRES of, for example, 15%, and detects the light flux pixel by pixel with the given threshold THRES, so that in fact the DVS camera has an effective dynamic range of 120 dB or even higher as a result of the full well capacity of the sensor elements and the exposure time adopted according to the minimum and maximum allowable values. The high dynamics of the signal of the event on the sensor element is significantly faster than the typical camera frame readout time, and the high dynamic range of the camera and / or the sensing principle is a core aspect of the advantageous solution provided in this article. The signal rate of the event can be in the range of as low as single microseconds, while the typical frame rate of Mpxl cameras today is in the range of 10 to 5 ms for frame rates of 100 or 200 Hz. As a result, the speed of the measurement system is improved by 3 to 4 orders of magnitude with respect to today's technology.

[0025] In summary, the main differences of DVS camera based 3D scanning with respect to conventional systems are based on the properties of DVS cameras as described below to detect events rather than full frame imaging, the speed of DVS camera data acquisition is significantly higher compared to the typical frame rate of full frame cameras, there is no need for tight and precise timing coupling in the accuracy of conventional cameras because DVS cameras are much faster, and there is a significant reduction in the amount of data because only relevant events are reported rather than a complete image capture of the overall surface of the inspected object.

[0026] The proposed system allows very fast and precise measurements, where lateral and depth resolution can be controlled mainly by the parameters of the optical segmentation and the operating parameters of the system. Among others, the application of event-based or DVS cameras for precise 3D measurements benefits from the very high dynamic nature of DVS cameras, which allows very fast and continuous changes in the effective focal length f(t) of the optical segmentation and still takes the associated changes in the projection to generate an accurate 3D view of the scene. The 3D topography of the object is reconstructed from the event signal SIG(j) based on: maximum or minimum peak detection of the sensor element SE(j) at the corresponding intensity including fitting capability; edge points or contours by a sequence of adjacent positive POS and negative events NEG on the side of the contour; and sub-pixel lateral resolution by lateral evaluation of positive or negative events when defocusing the corresponding structure.

[0027] It should be understood that the elements and features described in the appended claims can be combined in different ways to produce new claims that also fall within the scope of the invention. Therefore, although the dependent claims attached below are dependent on specific independent or dependent claims, it should be understood that these dependent claims can be alternatively dependent on alternatives from any previous or subsequent claims, whether independent or dependent, and such new combinations should be understood to form part of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the following, possible embodiments of different aspects of the present invention are described in more detail with reference to the attached drawings. The purpose and further advantages of this embodiment will become more obvious and easy to understand from the following description of the preferred embodiment in conjunction with the attached drawings, in which:

[0029] Figure 1 shows a scanning system for 3D scanning of an object,

[0030] Figure 2 shows a flow chart of the scanning method SCMETH,

[0031] Figure 3 shows a top view of the sensor chip of a DVS camera,

[0032] Figure 4a shows a blurred projection onto the sensor chip at a first point in time,

[0033] Figure 4b shows a blurred projection onto the sensor chip at a second point in time,

[0034] Figure 5a shows a clear projection onto the sensor chip at a first point in time,

[0035] Figure 5b shows a clear projection onto the sensor chip at a second point in time,

[0036] Figure 6 A diagram illustrating the development of the luminous flux FX as a function of the focal length f(t) is shown,

[0037] Figure 7 shows a diagram illustrating the change in the light flux depending on the focal length f(t) and the associated incident signal,

[0038] Figure 8 A top view of a segment of the sensor chip is shown. DETAILED DESCRIPTION

[0039] Figure 1 Shown is an arrangement of a scanning system 100 and an object 200 to be scanned in three dimensions (3D) by the scanning system 100. The scanning system 100 comprises a DVS camera 110 and an evaluation system 120, wherein the DVS camera 110 is configured and arranged such that it captures a field of view FOV in the z-direction positioned in front of the DVS camera 110. The object 200 to be scanned is arranged in the field of view FOV of the DVS camera 110.

[0040] The scanning system 100 is not only configured to capture 2D data of the object 200, i.e., representing the properties of the object 200 in the xy direction, but is also configured to derive depth information, i.e., information about the object 200 in the third dimension and z direction accordingly, so as to achieve 3D scanning based on the imaging data DAT(j) provided by the DVS camera 110.

[0041] Taking the outline or edge e201 of an object 200 as an example, it can be assumed that the edge e201 consists of multiple spatial features 201. Determining the 3D position of the edge e201 means that the 3D position of the feature 201 can be determined individually, and the edge e201 can extend in any direction in space. In the following, the determination of the 3D position is described for one feature 201. However, it will be clear that the 3D position and extension of the entire edge e201 in space is nothing more than the overall 3D position of its individual features 201. The solution described below of course enables the 3D position of several features 201 to be determined in parallel.

[0042] The 3D scan will provide the position POS201=(x201, y201, z201) of the feature 201 in three dimensions. The depth information and the position z201 of the feature 201 will respectively correspond to the distance d201 of the corresponding feature 201 in the z direction from a reference, such as Figure 1 Surface 111 p of the sensor chip 111 of the DVS camera 110 is indicated in FIG.

[0043] For the purpose of 3D scanning, the evaluation system 120 is connected to the DVS camera 110 using a wired or wireless connection 121. The evaluation system 120 and its control unit 124 are configured to control the DVS camera 110 using a control signal CTRL, for example, regarding the setting of the instantaneous focal length f(t) of the optical segment 112 of the camera 110, and to receive imaging data DAT(j) from the DVS camera 110, both via the connection 121.

[0044] Furthermore, the evaluation system 120 is configured to determine 3D spatial information about the object 200 based on an evaluation of the imaging data DAT(j) provided by the DVS camera 110 .

[0045] For the purpose of 3D scanning and subsequent evaluation, the evaluation system 120 applies Figure 2 The scanning method SCMETH described in the context of , comprises a stage SCAN of scanning an object 200 and a stage EVAL of evaluating the data DAT(j) collected in the SCAN stage. The scanning method SCMETH may be implemented as corresponding software executed by a processor 122 of an evaluation system 120. The evaluation system 120 may further comprise a memory 123 for storing any data to be processed, such as imaging data DAT(j), as well as software to be executed by the processor 122, and / or any other suitable or required data.

[0046] The DVS camera 110 has a sensor chip 111 having a plurality of light-sensitive sensor elements SE(j), where j=1, . . . , J, and J represents the total number of sensor elements SE(j), for example J ≥ 10. 6 As a simplified example, Figure 3A top view of a sensor chip 111 with only 16*24=384 sensor elements SE(j) is shown in FIG. Only a few sensor elements SE(j) have been marked with reference numerals. The sensor chip 111 is a rectangular chip 111, wherein its sensor elements SE(j) are arranged in rows and columns perpendicular to one another. The extension of the planar, essentially two-dimensional sensor chip 111 defines a coordinate system x, y, z, wherein the plane of the sensor chip 111 defines the x direction in the first dimension and the y direction in the second dimension, while the z direction in the third dimension is correspondingly perpendicular to the plane and surface 111p of the sensor chip 111.

[0047] In contrast to conventional cameras, the signal SIG(j) generated by the DVS camera 110 for a given sensor element SE(j) does not directly or, for example, proportionally represent the light intensity or light flux FX onto the respective sensor element SE(j), but it depends on the relationship between the light flux FX onto the sensor element SE(j) at the time point t(i) and the light flux FX onto the same sensor element SE(j) at the earlier time point t(i-1). More specifically, it shows whether the light flux FX at the sensor element SE(j) for the time point t(i) differs from the flux FX at the same sensor element SE(j) at the earlier time point t(i-1) by more than a given threshold value THRES. In case such a difference is indeed above the threshold value THRES, a positive or negative event is generated for the respective sensor element SE(j).

[0048] Therefore, the DVS camera 110 is configured to generate a signal SIG(j) for each sensor element SE(j) individually, in case the light intensity or light flux onto the corresponding sensor element SE(j) respectively changes more than a given threshold value THRES (e.g. 15%), hereinafter referred to as an "event signal" SIG(j), while not generating an event signal in case the flux change DELTAFX is less than or equal to the threshold value THRES. Therein, the event signal SIG(j) is not assigned a free floating value, but each event signal SIG(j) is selected from a predefined group GRP of event types, which group GRP may include at least positive events and negative events. Typically, the group GRP contains only two event types, namely "positive events" and "negative events". Therefore, in case the event signal SIG(j) is generated, it can only be a positive event in case the flux FX changes to a higher flux by an amount DELTAFX where DELTAFX>THRES, or a negative event in case the flux changes to a lower flux by an amount DELTAFX where DELTAFX>THRES. In case the change DELTAFX of the light flux onto the sensor element SE(j) is lower than or equal to the threshold value THRES (i.e. DELTAFX≤THRES), no event signal SIG(j) is generated for the corresponding sensor element SE(j). Therefore, the event signal SIG(j) is only generated in case of a significant flux change (i.e. DELTAFX>THRES).

[0049] More specifically, considering a specific sensor element SE(j) of the sensor chip 111 of the DVS camera 110: in the case where the instantaneous flux FX(t(i)) onto the sensor element SE(j) is higher than the earlier flux FX(t(i-1)) onto the same sensor element SE(j) by more than THRES (i.e., FX(t(i))-FX(t(i-1))>THRES), the event signal SIG(j) generated for the specific sensor element SE(j) is a positive event. In the case where the instantaneous flux FX(t(i)) onto the sensor element SE(j) is lower than the earlier flux FX(t(i-1)) onto the same sensor element SE(j) by more than THRES (i.e., FX(t(i-1))-FX(t(i))>THRES), the event signal SIG(j) generated for the specific sensor element SE(j) is a negative event. In all other cases, where the amount of the difference between the fluxes FX(t(i)), FX(t(i-1)) |FX(t(i))-FX(t(i-1))| is less than THRES, i.e. |FX(t(i))-FX(t(i-1))|≤THRES, no signal is generated at all for the corresponding sensor element SE(j). Correspondingly, the sensor element SE(j) for which no event signal is generated is not assigned any symbol in the corresponding figure.

[0050] The scanning method SCMETH for determining 3D spatial information of an object 200 advantageously utilizes the properties of such a DVS camera 110 .

[0051] The DVS camera 110 has a body 113, which may include typical electronics including the sensor chip 111 described above and an optical segment 112. Just like a conventional camera lens, the optical segment 112 may be made of an arrangement of one or more optical devices such as a lens, an aperture, etc., and it has an effective main plane 112'. Furthermore, the optical segment 112 is adapted to the size of the sensor chip 111 and is selected such that imaging the object 200 in the field of view FOV results in the object 200 being projected onto the sensor chip 111, corresponding to the size of the sensor chip 111.

[0052] The optical segment 112 and the DVS camera 110 are accordingly configured to vary the focal length f in the stage SCAN in a variation range between a minimum value fmin and a maximum value fmax. Such a focal length variation moves the focus in the z direction and typically also results in a shift of the main plane 112', wherein the relevant parameters f, g, b can be approximated based on the lens manufacturer equation 1 / f=1 / g+1 / b, where f represents the focal length, b represents the image distance, and g represents the object distance. For a known optical segment 112, it can be assumed that for each time point t, f(t), g(t) and b(t) are known. For example, b(t) can be fixed and known, while f(t) is variable and known, so that g(t) can be calculated based on the lens manufacturer equation.

[0053] Preferably, the object 200 is arranged relative to the DVS camera 110 so that it can be completely scanned in the z-direction. For example, the point of the object 200 closest to the DVS camera 110 is at a distance from the main plane 112' of no less than fmin. With respect to the distance of the point of the object 200 farthest from the DVS camera 110 from the main plane 112' and the required maximum focal length fmax, the well-known lens manufacturer equation 1 / f=1 / b+1 / g applies again. This ensures that the DVS camera 110 is configured to completely scan the object 200 in the third dimension and the z-direction accordingly.

[0054] In the following, it is assumed that the DVS camera 110 and the object 200 do not move relative to each other, while the focal length f=f(t) of the optical segment 112 of the DVS camera 110 changes over time t. For example, the focal length f(t) may increase from a minimum value fmin to a maximum value fmax in a first phase of the focal length change, wherein the increase may be a linear function over time, i.e., f(t)=fmin+DF*t, wherein DF=DF1 is a suitable constant value representing the desired focal length increase rate. After the first phase, once f(t)=fmax is achieved, the focal length may be reduced again in a second phase of the focal length change, e.g., following the function f(t)=fmax-DF*t, until the minimum focal length fmin is achieved to complete the first cycle of the focal length change. In an alternative embodiment, the function f(t) is not a linear function as described above, but is, for example, a sinusoidal function, e.g., f(t)=Cl*sin(DF*(t+t0))+C0. Where Cl = (fmax - fmin) / 2 and C0 = (fmax + fmin) / 2 may be useful, with DF representing the desired rate of focus change, and t0 being a fixed but arbitrary time offset.

[0055] In the following, one cycle of focal length variation may start at f(t=0)=fmin, reach f(t=T / 2)=fmax, and return to f(t=T)=fmin, where T represents the time required to complete one cycle. Alternatively, the cycle may start at f(t=0)=(fmax+fmin) / 2, increase f(t) to reach fmax, then decrease f(t) to reach fmin, and increase f(t) again to return to f(t=T)=(fmax+fmin) / 2. Of course, other arrangements are conceivable. In any case, a "stage" of the cycle may be defined such that all available focal lengths f(t) between fmin and fmax will be applied at least once during one stage.

[0056] In any case, multiple cycles of focus variation can be applied to improve the accuracy of the method, i.e. the focus f(t) can be increased and decreased multiple times, although in principle one stage of a cycle would be sufficient, since all focus f(t) between fmin and fmax would be applied once during one stage. For example, DF and T can be selected so that 300 cycles are completed per second. For each feature 201, each cycle provides two insights into the location z201 of the corresponding feature in the z direction, since the focus f(t) corresponding to the location of the corresponding feature in the z direction is applied twice per cycle - except for the two extreme points at fmin and fmax, which are only applied once per cycle. It can then be assumed that the feature location in the z direction is the average of the individual values, and the reliability increases with the number of values ​​to be averaged. Therefore, each complete cycle, and indeed, each stage of such a focus variation cycle, improves the reliability of the 3D scan and the location determination.

[0057] However, in the following, only one phase of a cycle of focal length variation is explained, wherein the explanation can easily be applied to other phases and cycles accordingly.

[0058] In addition to the z-direction, only the x-direction is considered in the explanation, while the y-direction is not explicitly mentioned. However, the considerations regarding the x-direction can be easily and unambiguously applied to the y-direction.

[0059] In a first operating and execution mode of the method SCMETH, the focal length f(t) of the optical segment 112 is varied during a phase, for example starting with f(t=0)=fmin (i.e. with a minimum focal length), and in this exemplary case it is assumed that the focal length varies in a linear manner, i.e. f(t)=fmin+DF*t. In the first operating mode, the variation of the focal length is performed such that the focal position is moved in the z direction so that different segments of the object 200 in the z direction are sharply imaged over time t. However, it can be assumed that for each point in time t, the instantaneously applied focal length f(t) as well as g(t) and b(t) are known and can therefore be used for further processing in the evaluation system 120. At the same time, i.e. during the stage SCAN, any event signal SIG(j) generated by the sensor element SE(j), i.e. a positive POS or a negative event NEG, is transmitted to the evaluation system 120 as part of the imaging data DAT(j) for further processing in the stage EVAL. The imaging data DAT(j) transmitted from the DVS camera 110 to the evaluation system 120 for further processing in the evaluation level EVAL comprises the event signal SIG(j) itself, the exact time stamp TS(j) representing the point in time at which the sensor element SE(j) generated the event signal SIG(j) and an unambiguous identifier ID(j) of the respective individual event signal SIG(j) generating sensor element SE(j) to ensure that the position POS(j) at which such event signal SIG(j) of the sensor element SE(j) is generated is known on the sensor chip 111. The position POS(j) which can be derived from the identifier ID(j) is required to determine the lateral position x201, y201 of the feature 201, as described later.

[0060] In the first mode of operation, varying the focal length f(t) during stage SCAN results in several effects:

[0061] The first effect exploits the fact that when the focal length f(t) changes due to a corresponding change in magnification, the projection P201 of a feature 201 (e.g. an edge or border) of the object 200 onto the sensor chip 111 moves accordingly across the surface 111p of the sensor chip 111 and the sensor element SE(j) in a lateral direction (i.e. in the x and / or y direction).

[0062] Taking into account the special behavior of the DVS camera 110 as described above, namely its sensitivity to changes in the light flux, rather than to the absolute value of the flux, the first effect of moving across the sensor chip 111 while changing the focal length f(t) may result in those sensor elements SE(j) across which the projection P201 moves actually generating an event signal SIG(j). However, this depends on whether the change DELTAFX=dFX in the flux FX onto the respective sensor element SE(j) is above a threshold value THRES. As explained below, this requirement may not be met as long as the projection P201 of the feature 201 is too blurred. As a result, no sensor element SE(j) will generate an event signal SIG(j) corresponding to the feature 201.

[0063] The first effect is Figure 4a , 4b For clarity, the results of the second effect are presented below in Figure 4a , 4b , 5a, 5b and related explanations are ignored. Figure 4a , 4b and Figure 5a , 5b Both show in their lower segments a side view in the y direction of some of the sensor elements SE(j) of the sensor chip 111. The sensor elements SE(j) are arranged adjacent to each other in the x direction and it can be assumed that the light flux FX corresponding to the respective projection P201 falls on the sensor elements SE(j) from above. The upper segments of the figures show the flux FX onto the individual sensor elements SE(j).

[0064] Figure 4a , 4b The blurred projection P201 at the time points t1, t1+dt is shown, where the instantaneous focal lengths f(t=t1), f(t=t1+dt) are correspondingly such that the projection P201 of the feature 201 onto the sensor chip 111 is blurred and has low definition. Here and in the following, the terms "clarity" and "definition" of the projection P201 essentially refer to the spatial extension of the projection segment representing a specific feature (e.g. edge 201) in the lateral direction. High definition means that the lateral extension of the projection of the feature edge is minimal, while low definition means a wide extension, i.e. a blurring effect. From Figure 4a As can be seen in FIG. 1 , the slope of the projection P201 representing the edge 201 essentially extends from the signal element SE(j+2) to the signal element SE(j+9), i.e. across eight signal elements SE, which is interpreted as low definition and strong blur. In contrast, Figure 5aA situation is shown where the slope of the projection P201 representing the edge 201 again extends substantially from the signal element SE(j+5) to the signal element SE(j+6) or even less, i.e., across one or two signal elements SE, which is interpreted as high definition and low blur.

[0065] Returning to Figure 4a 、 4b and considering the time point t1, the difference dFX in the flux FX between the adjacent sensor elements SE(j+5) and SE(j+6) is small and less than the threshold THRES. Thus, after a time span dt, i.e., at the time point t1+dt, when the projection P201 has moved in the x direction by a distance corresponding to the extension of the sensor element SE(j), as Figure 4a 、 4b shown in, the light flux FX onto the adjacent sensor element SE(j+6) will change by an amount dFX due to the movement of the projection P201. In the case where dFX is less than THRES, the sensor element SE(j+6) will not generate any event signal in this scenario.

[0066] In other words, due to the movement of the projection in the x direction, the change dFX in the flux FX at the sensor element SE(j+6) between the time points t1 and t1+dt is small only due to the blur of the projection P201. Assuming dFX < THRES, no event signal will be generated.

[0067] In contrast, Figure 5a 、 5b a situation of the projection P201 at the time points t2, t2+dt is shown, where the instantaneous focal lengths f(t=t2), f(t=t2+dt) are such that the projection P201 of the feature 201 onto the sensor chip 111 is correspondingly sharp and has high definition while having low blur. In the example shown, the projection P201 clearly illustrates that the feature 201 belongs to the edge e201 of the object 200.

[0068] Thus, at the time point t2, the difference dFX in the flux FX between the adjacent sensor elements SE(j+5) and SE(j+6) is large and especially higher than the threshold THRES. Thus, after a time span dt, i.e., at the time point t2+dt, when the projection P201 has moved in the x direction by a distance corresponding to the extension of the sensor element SE(j), as Figure 5a 、 5b shown in, the light flux FX onto the sensor element SE(j+6) will change by an amount dFX > THRES due to the movement of the projection P201. In the case where dFX is higher than THRES, the sensor element SE(j+6) will generate an event signal SIG(j+6) at a dedicated time point t0 = f2+dt.

[0069] in other words, Figure 5a , 5b A scene with an instantaneous focal length f(t) is shown that makes the projection P201 of the feature 201 onto the sensor chip 111 sharp and does not suffer from significant blurring. Due to the sharpness of the projection P201, the change dFX of the flux FX at the sensor element SE(j+6) due to the movement of the projection in the x-direction between the time points t2 and t2+dt is large. Assuming dFX>THRES, the event signal SIG(j+6) will be generated by the sensor element SE(j+6).

[0070] Therefore, considering only the first effect, it becomes possible to identify the instantaneous focal length f(t0) corresponding to a specific feature 201 and, thereby, to identify the expected depth information d201 about this feature 201 based on the occurrence of the event signal SIG(j+6) of the sensor element SE(j+6), which is to be registered by the evaluation system 220 at the time point t0=t2+dt: Assuming that for each time point t the instantaneous setting of the focal length f(t) is known, and assuming that the time point t0=t2+dt of the occurrence of the event signal SIG(j+6) is also known, it can be assumed that the actual focal length f(t0) at the dedicated time point t0=t2+dt is known.

[0071] Therefore, the position z201 of the feature 201 of the object 200 in the z direction can be derived directly from the actual focal length f(t0). For example, still assuming that the surface 111p of the sensor chip 111 represents a reference for determining the depth information, it can be assumed that z201=d201=g(t0)+b(t0), where 1 / f(t)=1 / g(t)+1 / b(t) still applies. As mentioned earlier, b(t) can be correspondingly constant and fixed, while f(t) is variable and known, so that g(t) can be calculated based on the lens manufacturer equation.

[0072] Summarizing the first effect, any change in the effective focal length f(t) will also change the magnification and lateral imaging scale of each part of the object 200, and therefore each geometrical feature e201, e202, e203 of the object 200. As a result, the size of the corresponding projection on the sensor chip 111 changes accordingly. In practice, this results in a lateral shift of the object features e201, e202, e203 relative to the grid of sensor elements SE(j) on the sensor chip 111. Therefore, in case the instantaneous focal length results in a sufficiently high resolution of the projection P201 of the feature 201, this lateral shift due to the scaling of the image size may result in a significant change of the intensity and flux pattern as imaged on the sensor element SE(j). Thus, the DVS camera 110 will detect the change to create a corresponding event signal SIG(j) also depending on the rate and amount of the change, especially in the case of structural elements such as edges, corners, border lines or borders of geometric structures such as circles or holes, e.g. for drilling with or without phases, threads, etc., and even defects such as dents or scratches. Due to this effect, structural elements tend to create a much higher number of events than homogeneous planes or smooth surfaces. Additionally, the DVS camera 110 is able to detect events at a much higher rate than ordinary CCD or CMOS cameras (e.g. a thousand times or up to ten thousand times faster), and thus the detection efficiency of changes in the imaged scene is significantly increased, while the total amount of data created is reduced, typically by a factor of one hundred up to a thousand, because the event signal is only generated if an event actually occurs, so that a large number of sensor elements do not experience a significant change in content and thus no events are created.

[0073] The second effect exploits the fact that the sharpness of the projection P201 of the feature 201 on the sensor chip 211 changes with a varying focal length f(t) and that this sharpness has a direct influence on the light flux FX onto the sensor element SE(j). As introduced earlier, the term "sharpness" of the projection P201 essentially corresponds to the spatial extension of a segment of such a projection in the lateral directions x, y. A high sharpness means a minimal extension, while a low sharpness means a wide extension, i.e. a blurring effect. The sharpness of the projection P201 depends on the instantaneous focal length f(t) and the distance d201 of the feature 201 in the z direction from a reference, e.g. the surface 111p of the sensor chip 111. Therefore, knowledge of the sharpness of the projection P201 of the feature 201 can be used to derive the distance d201 of the feature 201 from the reference 111p. As with the first effect, this distance d201 represents the desired depth information about the object 200.

[0074] The second effect is Figure 6 , 7 For clarity, Figure 6 , 7In the explanation related to the second effect, the result of the first effect introduced above, namely the lateral movement of the projection P201 across the sensor chip 111, is ignored. Figure 6 A diagram is shown of the development of the intensity and the light flux FX onto a given sensor element SE(j1) depending on the instantaneous focal length f(j). The maximum value of the flux FX is found at f(t0), corresponding to the maximum sharpness of the projection P201 of the feature 201.

[0075] Figure 7 The corresponding change dFX of the flux FX due to a change df of the focal length f(t) is shown in FIG. 1 , which is the basic measure for a given sensor element SE(j1) to generate an event signal SIG(j1). In other words, Figure 7 express Figure 6 The slope of the curve shown. In addition, Figure 7 Also included is the change dFX of the flux FX required to cause the event signal SE(j1), which is expressed by the threshold THRES introduced earlier. In case the change dFX is above the threshold THRES (i.e. dFX>THRES), a positive event is generated. If the change dFX is less than the negative threshold -THRES, i.e. dFX<-THRES, a negative event is generated.

[0076] Figure 7 It is clarified that increasing the instantaneous focal length f(t) leads to an increase in dFX, which exceeds a threshold value THRES between time points t1 and t2, so that at least one of several event signals SIG(j1) of a first type (e.g., positive event) or typically a sequence SEQ(j1) of several event signals SIG(j1) of a first type (e.g., positive event) is generated, such as Figure 7 As illustrated in the lower part of , symbolized by “(POS)”. Therein, t1 denotes the time point at which the first of the positive event signals of the sequence SEQ(j1) is generated, and t2 denotes the time point at which the last of the positive event signals of the sequence SEQ(j1) is generated. Between the time points t2 and t3, dFX is less than THRES, so that SE(j1) does not generate an event signal. In fact, the time point t0 at which the instantaneous focal length f(t0) has the maximum clarity of the projection P201 of the feature 201 is located between t2 and t3. Between the time points t3 and t4, the flux change dFX is less than -THRES, that is, the amount |dFX| is higher than THRES, so that at least one of the event signals of the second type SIG(j1) or typically a sequence SEQ(j1) of event signals of the second type SIG(j1) is generated, such as a negative event, as Figure 7As illustrated in the lower part of, it is symbolized by "(NEG)". Among them, t3 represents the time point of the first of the negative event signals in which the sequence SEQ(j1) is generated, while t4 represents the time point of the last of the negative event signals in which the sequence SEQ(j1) is generated.

[0077] A pattern PAT(j1) formed by one or more positive event signals SIG(j1)=POS for the focal lengths between f(t1) and f(t2) with t1 < t < t2 and one or more negative event signals SIG(j1)=NEG for the focal lengths between f(t3) and f(t4) with t3 < t < t4 allows, for example, the estimation of the focal length f(t0) corresponding to the maximum sharpness via an interpolation method. This is based on the fact that the time dependence of the focal length f(t) during the focal length change is known, that is, f(t0) can be derived from the knowledge of the time point t0 at which the sharpness of the projection P201 is the highest for the instantaneous focal length f(t). For this purpose, t0 can be derived from the knowledge of two or more of the time points t1, t2, t3, t4. Assuming that the pattern PAT(j1) is essentially symmetric, where f(t0) is the center point of the symmetric pattern, a simple exemplary interpolation method for calculating t0 can be t0 = (t2 + t3) / 2. However, other methods considering t1 and / or t4 are also conceivable.

[0078] The knowledge of t0 now allows the determination of the corresponding instantaneous focal length f(t0) based on the known time correlation of f(t), for example, f(t0) = fmin + DF * t0. Similar to the first effect, the positioning z201 of the feature 201 of the object 200 in the z direction can be directly derived from the actual focal length f(t0). For example, again assuming that the surface 111 of the sensor chip 111 represents the reference for determining the depth information, it can be assumed that z201 = d201 = g(t0) + b(t0), where 1 / f(t) = 1 / g(t) + 1 / b(t) still applies. For example, where b(t) is correspondingly constant and fixed, while f(t) is variable and known, such that g(t) can be calculated based on the lens manufacturer's equation.

[0079] As a summary of the second effect, varying the focal length f(t) results in a projection P201 structure of the object 200 such as an edge e201 and, as a result, the projection of the feature 201 as described above is blurry, then becomes increasingly sharper, reaching maximum sharpness at time point t0 at the instantaneous focal length f(t0) and then becoming blurry again. Reaching maximum sharpness means, among other things, that the light flux FX changes drastically in the time span around t0, which ultimately results in a sequence of positive and negative event signals during the time span from which t0 can be interpolated. Knowledge of t0 and knowledge of the temporal behavior of f(t) allow the instantaneous focal length f(t0) to be determined and, therefore, the positioning of the conjugate plane of the sensor chip 111 in the object space, which represents the distance d201 from the reference 111p and, correspondingly, the z-positioning of the feature 201.

[0080] Thus, the second effect is based on the fact that the sharpness of each object feature 201, 202, 203 changes with a change in the effective focal length f(t) of the optical segment 112. This results in that the steepness of the grayscale profile of, for example, an edge on the sensor originating from the feature 201, 202, 203 also changes, which results in a change in the lateral intensity and flux distribution on the sensor chip 111. In other words, the focal length change affects the spatial redistribution of the light flux due to a mere defocusing effect, i.e. the flux around the edge is redistributed by defocusing. Furthermore, this redistribution of the energy of the optical light field will create events in adjacent sensor elements SE when the flux FX rises or falls due to a redistribution of energy due to the focus quality. This effect will characterize an edge or feature profile, etc., by adjacent positive and negative event signals along the edge direction or feature profile line.

[0081] In an alternative second mode of operation and execution of the method SCMETH, the focal length f(t) of the optical segment 112 may be varied as described earlier, but while maintaining the focus position in the object space. In this case, a third effect is induced, namely the magnification of the projection on the sensor chip 111 (e.g. the projection P201 of the feature 201) changes with the changing focal length f(t), while the sharpness of the projection P201 of the feature 201 remains substantially constant. However, the sharp projection P201 of the feature 201 shows a lateral shift across the sensor chip 111 and the sensor element SE(j) due to the change in magnification, thereby causing the events of all features 201 and the corresponding event signals SIG(j) to approach the instantaneous focus position in the z-direction.

[0082] In a second mode of operation, the propagation speed of the lateral movement of the projection P201 of the feature 201 is a measure of the corresponding distance d201 of the feature to the reference 111p. By comparing the relative speeds of the lateral movement of at least two different features 201, 202, an indicator of the different distances d201, d202 of the two features 201, 202 from the reference 111p is generated. The comparison takes into account lateral offsets of the corresponding projections P201, P202 from the optical axis, since these offsets also affect the lateral movement via changes in focal length and thus via changes in magnification. This second mode of operation is well suited for features 201, 202 in the far field of the optical segment 112, i.e., where the distances d201, d202 are significantly larger than the effective focal length f(t) of the optical segment 112.

[0083] For a change in focal length, a change in magnification is also intrinsic to the first effect and the second effect and can also be evaluated there as an intrinsic depth zoom in the image data as initiated by the change in focal length. The scanning method SCMETH is built on the effects summarized in the context of the first and second operating modes above. For the sake of clarity, the effects have been explained independently of each other, i.e. without taking into account the corresponding other effects. In fact, all effects simultaneously influence whether the sensor element SE(j) generates an event signal SIG(j). However, in particular, when the instantaneous focal length f(t) makes the projection P201 of the feature 201 onto the sensor chip 111 sufficiently clear, the first and second effects utilize the sharp increase and decrease of the flux and projection P201 associated with the feature 201.

[0084] The main effects used in this article are the change of the flux FX to extreme values ​​at the best focus location and the rapid change of the flux value near the best focus. Therein, the strong blur, i.e. the low definition, leads to the fact that even the movement of the projection due to the first effect does not lead to a sufficiently high change of the light flux onto the sensor element. Therefore, as long as the instantaneous focal length f(t) is significantly different from the distance d201, the influence of the first effect is low due to the blur, so that no event signal corresponding to the feature 201 is generated.

[0085] For all the effects introduced above, the instantaneous focal length f(t0) is determined based on the selected time point t0 at which the specific event signal SIG(j) has been generated due to the specific feature 201. Features 201, 202, 203 in different depths and therefore different distances d201, d202, d203 from the reference 111p will cause events at time points t0_201, t0_202, t0_203 associated with different, known focal lengths f(t0_201), f(t0_202), f(t0_203). However, such a time point t0 expressed by a time stamp TS(j) or any other suitable means may be provided by the DVS camera 110 as part of the imaging data DAT(j), and the evaluation system 120 processes the time stamp TS(j) to derive the instantaneous focal length f(t0=TS(j)). This assumes that the control of the focal length change and the readout of the event signal SIG(j) are synchronized. In other words, the clock of the DVS camera 110 and the clock of the control unit 124 for controlling the focal length f(t) of the optical segment 112 and / or the evaluation system 120 are synchronized and use the same accurate time base, so that the instantaneous focal length f(t) can be accurately determined for the time stamp TS(j) belonging to the event signal SIG(j). Since the DVS camera 110 can detect events very quickly and therefore in a tight time sequence, the change of the effective focal length f(t) can be very fast, for example 300 cycles per second, in order to have a high performance 3D scan for capturing a given depth range.

[0086] In general, the z-position d201 to be determined for the feature is determined based on the instantaneous focal length f(t0) and / or the related g(t0) and / or b(t0) according to the time point t0. The time point t0 is selected such that t0 represents a time point during the change of the focal length f(t) at which the temporal and / or spatial pattern of one or more event signals SIG(j) generating sensor elements SE(j) satisfies a predefined condition or corresponds to a predefined pattern, respectively.

[0087] Therein, in the simplest case, the “temporal and / or spatial pattern” of the event signal SIG(j) generating the sensor elements SE(j) may dictate that one of the sensor elements SE(j) must generate an event signal at time point t0: in this case, it is assumed that the event signal SIG(j) of any event signal type, e.g. a positive event POS or a negative event NEG, satisfies the predefined condition when generated at time point tp, where t0 is selected to be the same as time point tp, i.e. t0=tp.

[0088] In more detail, the pattern may be a temporal pattern which may consider the same sensor element SE(j) over time and observe the temporal development of the even signal SIG(j) generated by this sensor element SE(j).

[0089] That is, the temporal pattern is characterized in that the sensor element SE(j) must generate an event signal at a first time point t1 and a further but different event signal at a second time point t2≠t1. In a similar manner, the temporal pattern can be characterized in that the sensor element SE(j) must generate a sequence SEQ(j) of event signals SIG(j) of a first type, e.g. SIG(j)=POS, during a first time span dt12 between the time points t1, t2 and a sequence SEQ(j) of further but different event signals SIG(j), e.g. SIG(j)=NEG, during a second time span dt34 between the time points t3, t4.

[0090] In more general terms, the temporal pattern can be characterized in that the sensor element SE(j) must generate a sequence SEQ(j) of one or more first type of event signals SIG(j), for example SIG(j)=POS, during a first time span dt12 between time points t1, t2, and generate one or more further but different event signals SIG(j), for example SIG(j)=NEG, during a second time span dt34 between time points t3, t4.

[0091] The spatial pattern may take into account different sensor elements SE(j1), SE(j2) and observe the event signals SIG(j1), SIG(j2) generated by them. Such a spatial pattern may be characterized in that one of the sensor elements SE(j1) must generate an event signal SIG(j1) of a first type and the other sensor element SE(j2) must generate an event signal SIG(j2) of a different type, substantially at the same point in time t0.

[0092] As a hybrid pattern of both spatial and temporal patterns, different sensor elements SE(j1), SE(j2) may be considered over time t. Such a hybrid pattern may be characterized, for example, in that a first sensor element SE(j1) generates a first event signal SIG(j1) of a first type or a sequence SEQ(j1) of first event signals SIG(j1), such as a positive event signal POS, at a first point in time or time span t1, while a related second event signal SIG(j2) or a sequence SEQ(j2) of event signals SIG(j2) of a second type, such as a negative event signal NEG, is generated at a second point in time or time span t2≠t1, but is generated by a second different sensor element SE(j2) unlike in the scenario of a pure temporal pattern. The second sensor element SE(j2) may be in the neighborhood of the first sensor element SE(j1), but in case of a hybrid or pure spatial pattern it is not the same sensor element.

[0093] In a more complex scenario, it is assumed that a predefined condition is met if, for the first of the sensor elements SE(j), a first sequence SEQ(j) of one or more event signals SIG(j) comprising a first event signal type (e.g., positive event POS) is generated between time points t1 and t2, where t2 ≥ t1, and for the sensor elements SE(j’) that are clearly related to the first sensor element SE(j) among the plurality of sensor elements SE(j) (i.e., in a given neighborhood), a second sequence SEQ(j’) of one or more event signals SIG(j’) comprising a second event signal type (e.g., negative event NEG) is generated between time points t3 and t4, where t4 ≥ t3 and t3 > t2, where t0 is selected in the range t2 < t0 < t3, in particular t0 = (t2 + t3) / 2. This includes the case where one or both of the sequences SEQ include only one event signal. Then, it is assumed that a predefined condition is met if, for the first of the sensor elements SE(j), a first event signal of the first event signal type (e.g., positive event POS) is generated at time point t1, and for the sensor elements SE(j’) that are clearly related to the first sensor element SE(j) among the plurality of sensor elements SE(j), a second event signal of the second event signal type (e.g., negative event NEG) is generated at time point t2, where t2 > t1, where t0 is selected in the range t1 < t0 < t2, in particular t0 = (t1 + t2) / 2. The sensor element SE(j’) that is “clearly related” to the first sensor element SE(j) can be the same as the first sensor element SE(j), i.e., SE(j’) = SE(j). However, as already exemplarily explained above in the context of the “first effect” in the first operating mode, due to the change in magnification and lateral imaging ratio associated with the change in focal length f(t), the change in focal length f(t) causes a lateral shift of the projection P201 of the feature 201 under consideration on the sensor chip 111. Thus, it can happen that the projection P201 of the feature 201 that has fallen on the sensor element SE(j) at time point t1 has moved to the sensor element SE(j’) at time point t2, such that the positive POS and negative event signals NEG are generated by different sensor elements SE(j), SE(j’). However, in the case where the nature and size, etc. of the optical segment 112 of the DVS camera 110 and the time points t1, t2 and the corresponding instantaneous focal lengths f(t1), f(t2) are known, it can be assumed that, in the case where an event of the first type has occurred at the sensor element SE(j), it is possible to easily predict at which sensor element SE(j’) an event of the second type must occur. Thus, such sensor elements SE(j’) and sensor element SE(j) are “clearly” related to each other.In short, in the case where both of these sensor elements are within a predefined distance DIST from each other on the sensor chip 111, the sensor elements SE(j), SE(j’) are clearly related to each other, where the predefined distance DIST is determined based on the rate of change of the focal length f(t) during the change of the focal length.

[0094] In this scenario, the mode in which the event signal SIG(j) is generated by the sensor element SE(j) is a temporal mode, considering the same sensor element SE(j) over time. In the case where the same sensor element SE(j) first generates an event signal of a first type, for example a positive event signal, at a time point t1 and subsequently generates an event signal of a different type, for example a negative event signal, at a time point t2, it can be assumed that a time point t0 must have occurred between those time points t1, t2. Among them, it is possible that the sensor element SE(j) does not generate an event signal between those time points t1, t2. It is possible that the projection P201 of the feature 201 is not limited to only one sensor element SE(j1), but it can extend across a region ARI on the sensor chip 111, and this region ARI covers multiple sensor elements SE(j1), SE(j2), SE(j3), …. This is depicted in Figure 8 shown in Figure 8 a segmented top view of the sensor chip 111 is shown. In such a scenario, it can be assumed that a predefined condition is met in the following case: for one or more sensor elements SE within a first region AR1 on the sensor chip 111, a first event signal SIG(j) of a first event signal type (for example, a positive event POS) is generated between a time point t1 and a time point t2, where t2 ≥ t1, and for one or more sensor elements SE within a second region AR2 on the sensor chip 111 that is clearly related (i.e., in a given neighborhood) to the first region AR1, a second event signal SIG(j’) of a second event signal type (for example, a negative event NEG) is generated between a time point t3 and a time point t4, where t4 ≥ t3 and t3 > t2, and where t0 is selected in the range t2 < t0 < t3, especially t0 = (t2 + t3) / 2.

[0095] For example, the first region AR1 and the second region AR2 are clearly related to each other in the case where they overlap or they are adjacent to each other, that is, at least one sensor element SE of the first region AR1 is adjacent to at least one sensor element SE of the second region AR2. "Adjacent" means that there are basically no further sensor elements SE located between those "adjacent" sensor elements SE.

[0096] The lateral position x201, y201 of the feature 201 in the x and y direction can be derived from the x and y position POS(j) of the sensor element SE(j) generating the event signal SIG(j) on the sensor chip 111, since the optical properties of the optical segment 112, such as for example magnification and zoom etc., again depend on the actual focal length f(t0) which is also known. As mentioned earlier, the position POS(j) of the sensor element SE(j) or its identifier ID(j) is transmitted together with the event signal SIG(j) to the evaluation system 120 with the imaging data DAT(j). Thus, the lateral position x201, y201 of the feature 201 can be derived from the identity ID(j) of the sensor element SE(j) generating the event signal SIG(j) and the knowledge of the actual focal length at the time point t0 when the signal element SE(j) generated the corresponding event signal SIG(j).

[0097] A method METH for 3D scanning of an object 200 has been described, which method METH focuses on determining the 3D location x201, y201, z201 of one specific feature 201 of the object, wherein the feature 201 is only one of several elements typical of an edge e201 or a contour or the like of the object 200. Therefore, determining the 3D location of the entire contour e201 means determining the 3D location of the individual features forming the contour e201. For a given feature of the object 200, such as a contour, such as an edge e201, there are two aspects in the imaging process of the DVS camera 110. The contour e201 can be characterized by a second series of positive event signals SIG=POS on one side of the contour e201 and negative event signals SIG=NEG on the other side. For reference purposes, by cross-checking, the contour e201 can be determined in an image stack of a classical full-frame camera image. The second aspect is the occurrence of a peak flux, which has a high or low value depending on the type of feature. When the focal length f(t) passes through the best focus position, for a given sensor element SE, there will be a sequence of positive POS and negative NEG event signals. Therefore, peak detection and interpolation can be applied in the time series of events. In order to fit the interpolated curve, the characteristic peak shape can be used to accurately assign the events on the valley focus curve of the corresponding optical segment 112. This has been described in the context of the second effect.

[0098] Accordingly, lateral shifts of the image caused by focal length changes and / or depth-dependent changes of the imaging scale and magnification can also be exploited for a more detailed analysis of the sub-pixel accuracy of feature or edge location in the image space. Due to the lateral redistribution of the image light flux with respect to the grid of sensor elements SE of the sensor chip 111, the event signal SIG can be evaluated according to the characteristic focus shift of the image intensity with respect to the best focus location in order to determine the sub-pixel shift of the curve and thus the real feature edge or contour location for a given edge location. This characteristic can be used to overcome the limitations of the DVS camera 110 for geometrically accurate measurements, such limitations as imposed by the quantitative nature of the event detection itself with a given threshold.

[0099] As becomes clear from the above, an essential feature of the optical segment 112 is its ability to vary the focal length f(t). This can be achieved via different implementations.

[0100] For example, the optical segment 112 may implement a classical zoom lens, where the effective focal length f(t) changes upon zooming.

[0101] Alternatively, the optical segment 112 may employ a lens with a fixed focal length, but the lens may be axially displaced, ie, axially displaced in the z direction, so as to change the distance from the lens to the image plane. The displacement may be physically achieved by relative mechanical movement or displacement.

[0102] As another alternative, the optical segment 112 may include an optical medium positioned between an arrangement of other optical devices of the optical segment 112, such as lenses and apertures, and the sensor chip 111. The effective optical thickness and length, respectively, may be varied by specific control parameters. Possible implementations of the optical medium may be based on, for example, electro-optical materials, magneto-optical materials, stress-, strain- or pressure-controlled optical thickness of corresponding material compartments in the optical beam path, birefringent materials, where the orientation of the birefringent optical elements is changed by an external field, physical rotation of the optical medium to change the effective thickness, and / or prisms that are displaced in such a way that the effective optical thickness changes in the imaging beam path. This results in an effective change of the plane in the object space that is in best focus when imaged onto the sensor chip 111.

[0103] In a further implementation, the optical segment 112 may include a liquid focusing lens and, optionally, a classical fixed-focus imaging lens. The change in the effective focal length f(t) may be introduced by a voltage or current controlling the optical power of the liquid lens via a control signal CTRL from a control unit 124 of the evaluation system 120. For example, in this setup, a total tuning range of four to more than twenty diopters may be realistic. Appropriate and tight coupling of the timing of the DVS camera 110 and the control voltage or current of the liquid lens will result in a correlation of the event detected at time t and the setting of the effective focal length f(t) of the optical segment 112 at the corresponding point in time t. For example, the control frequency may be in the range from 0.25 Hz to 2000 Hz, and the overall system may be as fast as 100 kHz to appropriately control a predefined change in the optical refractive power.

[0104] The optical imaging system and the optical segment 112 accordingly may include any combination of variable optical elements such as liquid lenses and / or classical lenses to achieve an optical system with appropriate imaging properties and the ability to change its refractive power or focal length. In a preferred embodiment, at least one variable optical element is located at or near the aperture plane of the optical segment 112 or near the aperture plane of the optical segment 112 to maximize the effect of changes in the refractive power or corresponding focal length of the optical segment 112.

[0105] Another implementation, preferably implemented with a microscope, may use a liquid lens as described above, or in a standard configuration, for example with a standard z-drive, to impose a continuous motion of the object to be scanned in the z direction along the focal axis, for example with a sinusoidal, triangular, trapezoidal or ramp type motion characteristic. As with other implementations, a common accurate time base or coupling of the data for the mechanical motion of the z-drive and the time stamp TS(j) of the event signal SIG(j) is required, which may have μs accuracy or even better, so that the event can be assigned to a corresponding and well-defined z-position, corresponding to the distance d201 introduced above, which represents the z-position z201 of the corresponding feature 201. In the case of a standard microscope, it may be necessary to add accurate timing and positioning information capabilities, because microscopes typically take camera images when the sample is stationary or in slow motion. In the DVS camera-based 3D scanning method introduced in this article, the z-motion can be much faster or much larger than a fraction, a few or several μm of the exposure time of a classic CCD or CMOS camera. The positioning of the z-drive may be derived from microscope control parameters such as the drive voltage of the piezoelectric unit, the number of steps of the stepper motor, an encoder or a positioning and / or focus sensor, such as a color focus sensor and the like.

[0106] Preferably, the scanning system 100 comprises an illumination system with one or more light sources 141 (e.g. LEDs), which needs to provide a constant light flux, which is preferably homogeneous over the entire field of view. Since typical DVS cameras 110 are monochromatic, the illumination can also be monochromatic, or it can be a polychromatic illumination, for example in the ultraviolet, short-wavelength infrared or visible spectrum. When the illumination system is capable of generating one wavelength at a time, a preferred mode of operation can be that one illumination color is used for one focus change phase, for example f(t) increases from fmin to fmax, and then another wavelength is used for the next focus change phase, for example in the opposite direction, i.e. from fmax to fmin. Switching the illumination during one focus change phase may create additional or artificial event signals SIG caused only by changing the illumination, rather than changing the imaging properties of the object 200 itself. Preferably, it is known at which wavelength each light source 141 operates at one focus change phase, and the corresponding information can be attached to the data set DAT. All known and typically applied illumination modalities can be combined, such as bright field transmitted light, bright field reflected light, dark field reflected light or dark field transmitted light, as well as any kind of mixed forms or modalities. Additional contrast methods such as polarization can also be combined for illumination. Whatever the illumination settings are with regard to wavelength, bandwidth, illumination modality, or what additional settings such as polarization may be, it is preferred to keep the settings constant at least for one focus change phase, so that all information from the DVS camera 110 can be primarily assigned directly to the object 200 itself.

[0107] Typically, the DVS camera 110 has a monochrome sensor 111 and does not have a color filter to capture a color image. To capture a color image, the illumination of the object 200 may be changed from one color to another, from one focus cycle consisting of, for example, one or more cycles or stages of focus variation to another, and later the final imaging data DAT is fused into a full-color image. Alternatively, a combination with a color camera is a beneficial embodiment in order to have a full-color image of the object 200, plus, at least at a single focus plane, plus event-based 3D information DAT of a fast-moving DVS imaging mode, including at least one stage, but preferably several cycles of focus variation. Of course, the color camera may also be operated in parallel with the 3D scanning of the DVS camera 110. This may make the image a bit blurry, but this information is still of great help in appropriately colorizing the event-based object image from the DVS camera 110. The resulting image may be prepared by fusing the data from the DVS and color cameras. In general, a DVS camera can be combined with any other camera for a complete image view in the same or even other spectral ranges, typically in the range from UV-VIS-NIR-SWIR, and it may be advantageous to decouple the measuring wavelength of the DVS camera from the additional camera by a spectral filter, preferably by a dichroic mirror. In one embodiment, the at least two cameras use the same imaging lens and the beam path is split between the cameras in the space between the lens and the first camera.

[0108] Thus, color images of the object may be generated with the aid of the lighting system or by combining the DVS camera 110 with at least a conventional camera to obtain color information of the corresponding features of the object 200. These color images do not necessarily need to be taken at an exact focus position and may be blurry, since fast focus changes that are safely detectable by the DVS are too fast for a classical camera. Thus, these images become blurry while maintaining the speed of the DVS camera.

[0109] Since the DVS cameras 110 detect changes in the flux FX of each sensor element SE and generate an event if the photocurrent rises or falls by a predefined amount, they automatically adapt themselves to the actual irradiance level of each sensor element SE in the operating range of the DVS camera 110 with respect to a speed given by the minimum readout time of the SE photocurrent of the sensor element and the maximum time for this evaluation. In particular, but to be understood as an example, this may result in an effective dynamic range of more than 100 dB or even 120 dB and above, compared to about 40 to 50 dB for a standard CMOS or CCD camera with full frame readout. This capability is a result of the inherent ability of the DVS sensor chip 111 to adapt locally to the respective signal strength, so that it uses the full well capacity of the sensor element SE with different exposure times, where the respective sensor element is being monitored for changes in exposure. In practice, this constitutes an effective dynamic range of 100 dB and above.

[0110] Although the present invention has been described above with reference to various embodiments, it will be appreciated that many changes and modifications may be made to the described embodiments. Therefore, it is intended that the foregoing description be considered illustrative rather than restrictive, and it will be appreciated that all equivalents and / or combinations of the embodiments are intended to be included in this description. Therefore, the present invention is not limited to the embodiments described above, but may be derived by those skilled in the art without departing from the scope of the present invention.

Claims

1. A method SCMETH for determining a position of a feature (201) of an object (200) relative to a reference position (111p) of the camera (110) based on a camera (110), wherein the object (200) is positioned in a field of view FOV of the camera (110), the field of view FOV extending from the camera (110) in a z direction, The camera is a DVS camera (110) - comprising a sensor chip (111) having a plurality of sensor elements SE(j), and - is configured to generate, for each sensor element SE(j), individually imaging data DAT(j), said imaging data DAT(j) comprising an event signal SIG(j) from a group of event signal types GRP, if the light flux FX onto the respective sensor element SE(j) changes by more than a given threshold value THRES, and - wherein said method comprises a scanning stage SCAN, in which the focal length f(t) of the optical segment (111) of the DVS camera (110) varies according to a predefined pattern f(t) in time, and -- Evaluation stage EVAL, in which any one or more imaging data DAT(j) generated by the DVS camera (110) during the focus variation of stage SCAN are evaluated to determine at least the z position d201 of the feature (201) in the z direction.

2. The method according to claim 1, wherein each imaging data DAT(j) comprises - the generated event signal SIG(j) itself, a time stamp TS(j) which represents the point in time t at which the sensor element SE(j) generates the corresponding event signal SIG(j), and - The respective individual event signal SIG(j) generates an identifier ID(j) of the sensor element SE(j).

3. A method according to any one of claims 1 to 2, wherein the z-position d201 to be determined for the feature (201) is determined based on the instantaneous focal length f(t0) at a time point t0, wherein the time point t0 is selected to represent a time point at which the temporal and / or spatial pattern of one or more event signals SIG(j) generating the sensor element SE(j) during a change in the focal length f(t) satisfies a predefined condition. 4 . The method according to claim 3 , wherein in case the event signal SIG(j) is generated at the time point tp it is assumed that a predefined condition is met, wherein t0 is selected to be identical to the time point tp.

5. A method according to any one of claims 3 to 4, wherein it is assumed that a predefined condition is met in the following case, - generating, for a first sensor element SE(j), between times t1 and t2, a first sequence SEQ(j) comprising one or more event signals SIG(j) of a first event signal type, wherein t2 ≥ t1, and For sensor elements SE(j') explicitly associated with the first sensor element SE(j), a second sequence SEQ(j') comprising one or more event signals SIG(j') of a second event signal type is generated between times t3 and t4, wherein t4 ≥ t3 and t3 > t2. Therein, t0 is selected in the range of t2<t0<t3, in particular t0=(t2+t3) / 2.

6. A method according to claim 5, wherein the sensor elements SE(j), SE(j') are unambiguously related to each other when both sensor elements are located within a predefined distance DIST from each other on the sensor chip, wherein the predefined distance DIST is determined based on the speed of change of the focal length f(t) during the focus change.

7. A method according to any one of claims 3 to 6, wherein it is assumed that a predefined condition is met if, - generating, for one or more sensor elements SE in a first area on the sensor chip, a first event signal SIG(j) of a first event signal type between times t1 and t2, wherein t2 ≥ t1, and - generating a second event signal SIG(j′) of a second event signal type between times t3 and t4 for one or more sensor elements SE in a second area of ​​the sensor chip which is unambiguously associated with the first area, wherein t4 ≥ t3 and t3 > t2, Therein, t0 is selected in the range of t2<t0<t3, in particular t0=(t2+t3) / 2.

8. The method according to claim 7, wherein the first area and the second area are unambiguously associated with each other in case the first area and the second area overlap or are adjacent to each other.

9. A method according to any one of claims 3 to 8, wherein the features The z-position of (201) is determined as the distance of the instantaneous focus of the optical segment (112) from the surface (111p) of the sensor chip (111) at the determined time point t0, wherein the instantaneous distance of the focus is derived from the instantaneous focal length f(t0) according to the selected time point t0.

10. The method according to any one of claims 1 to 9, wherein the optical segment (112) performs a change of the focal length f(t) according to a predefined pattern f(t) in time, such that - its focal length f(t) is variable, and - Due to the change of focal length f(t), its focal position moves in the z direction.

11. The method according to any one of claims 1 to 9, wherein the optical segment (112) performs a change of the focal length f(t) according to a predefined pattern f(t) in time, such that - its focal length f(t) is variable, and - Its focal position remains constant at a chosen position in the z-direction. 12 . The method according to claim 11 , wherein the stage SCAN is performed multiple times, wherein different constant focus positions in the z-direction are selected for different executions of the stage SCAN.

13. A method according to any one of claims 11 to 12, wherein for each execution of the level SCAN, the positioning POS(j,t) of those sensor elements SE(j) on the sensor chip (111) is observed over time t, said positioning POS(j,t) generating an event signal SIG(j) during a change in the focal length f(t) corresponding to the projection (P201) of the feature (201) onto the sensor chip (111), so as to determine a lateral propagation velocity VEL(POS(j,t),t) of the projection (P201) of the feature (201) across the sensor chip (111) due to the change in focal length f(t), wherein a representation of the z-position d201 to be determined for the feature (201) is estimated based on the lateral propagation velocity VEL(POS(j,t),t).

14. A method according to any one of claims 1 to 13, wherein the features 201 The lateral positioning x201, y201 in the x and y directions is derived from the positioning POS(j) of the sensor element SE(j) generated by the event signal SIG(j) on the sensor chip (111), wherein the positioning POS(j) is derived from the identifier ID(j) of the sensor element SE(j) generated by the corresponding individual event signal SIG(j).

15. A scanning system (100) for determining a position of a feature (201) of an object (200) relative to a reference position (111p) of the camera based on a camera, wherein a field of view (FOV) of the camera (110) for scanning the object (200) extends from the camera (110) in a z-direction, the scanning system (100) comprising - a DVS camera (110) comprising a sensor chip (111) having a plurality of sensor elements SE(j) and being configured to generate imaging data DAT(j) comprising an event signal SIG(j) from a group of event signal types for each sensor element SE(j) individually if the light flux onto the corresponding sensor element SE(j) changes by more than a given threshold value THRES, and an evaluation system (120) configured to control the optical segment (111) of the DVS camera (110) for varying the focal length f(t) and configured to receive and evaluate the imaging data DAT(j) for determining the positioning, The evaluation system (120) is configured to perform the method according to any one of claims 1 to 14.

Images