System and method for imaging of an object

Abstract

The invention relates to imaging of an object to generate a 2D image IMA of the object. The inventive approach applies a DVS camera and executes controlled lateral shifting of a projection of the object relative of a sensor chip of the camera. Polarities of resulting imaging data are processed to determine ratios of original light fluxes onto neighboring sensor elements of the sensor chip. The ratios are further processed to generate representations of the original light fluxes, from which the pixel values of the image's pixels are derived.

Description

[0001] Description

[0002] System and method for imaging of an object

[0003] The invention relates to imaging of an object to generate a 2D image IMA of the object.

[0004] Optical imaging in industrial applications suffers the problem of a huge amount of data being created when objects shall be inspected with high lateral resolution and / or by means of time series of several individual images. This amount of data needs to be transmitted and preevaluated in a smart camera or an edge device in order to make the data stream to an end device like a superordinated production control system manageable. Lower data rates would result in less resolution and, therewith, lower imaging accuracy which would have a negative impact on the quality of the production process. In the light of the data volume to remain manageable while, at the same time, the spatial and / or temporal resolution of the imaging task shall be high, one concrete problem to be solved might be how to achieve higher resolution without just ramping up the overall pixel number of the imaging system and thus blowing up the amount of generated data.

[0005] State of the art imaging systems capture the imaging data in high volume with a suitable camera and transmit captured data to an edge device or to the cloud which performs data preevaluation or even full evaluation. In some cases, smart cameras are used which enable very early data processing so that time critical applications are manageable. In many applications line-scan cameras are in use in order to have a more constant data rate than with full field cameras.

[0006] Such prior art imaging technologies provide the required high spatial and / or temporal resolution. However, they do not include solutions which, on the one side, provide images with low data volumes and which, on the other side, cover large dynamic ranges of e.g. 80 to 100dB or even higher. The high dynamic range is necessary to cover different illumination conditions in the field of view of the camera and to maintain good imaging properties to ensure the high imaging accuracy and quality. Typical full field or line cameras only have 40 to 60dB of maximal dynamic range. However, the so called “Dynamic Vision System” (DVS) cameras, e.g. described in PCT / EP2022 / 070785 and, for 3D applications, in PCT / EP2022 / 074649, promise to be an imaging technology with the potential to overcome those issues of data volume, dynamic range, and spatial resolution. However, DVS sensor chips act different from traditional full frame CCD or CMOS cameras etc. A single sensor element of the sensor chip issues an event EV based on changes of the light flux FX impinging onto single sensor element. Thus, DVS sensors do not capture image information according to external timing or frame signals. They do react on single sensor element level on changes of the light flux onto the respective sensor element and imaging data are provided from that sensor element as soon as such an event EV has occurred, i.e. the DVS concept does not foresee the limitation of image frames which are provided for the sensor chip as a whole. Thus, the concept of the DVS cameras does not foresee to generate an image frame like a traditional image or, in other words, the DVS concept does not generate full field images.

[0007] Therefore, a solution is required which allows fast and precise high-resolution full field imaging with low data volume. This is solved by the method suggested in claim 1 and by the imaging system as per claim 11. Dependent claims introduce advantageous embodiments.

[0008] A Method METH for camera based imaging of an object for generating an image IMA of the object applies an imaging system, which comprises a camera, which is embodied as a DVS camera, i.e. with a light sensitive sensor chip with a plurality of sensor elements SEn,m, each of which generates imaging data DATn,min case of an event EVn,mof change of light flux FX onto that sensor element SEn,mby more than a given threshold THRES. The method METH comprises a stage INSPECT of observing the object with the DVS camera and a stage EVAL of evaluating the imaging data DATn,m(k,s) generated during INSPECT.

[0009] Stage INSPECT includes execution of one or more shifting sequences SEQ(k) with k=1 , ... ,K and K>1 representing the number of shifting sequences SEQ(k). Each shifting sequence SEQ(k) comprises one or more shifting steps SEQST(k,s) with s=1 ,... ,SP(k) and SP(k) representing the number of shifting steps SEQST(k,s) of the shifting sequence SEQ(k). During each shifting step SEQST(k,s) a relative lateral shifting of a projection P200 of at least a section of the object and the sensor chip is executed in a shifting direction DIRSEQST(k.s) to generate the imaging data DATn,m(k,s).

[0010] In the stage EVAL, ratios RATFXn,m,n-i,m of original light fluxes FXn,m(T0), FXn-i,m(T0) onto sensor elements SEn,m, SEn-i,m being neighbored to each other in the shifting direction DIRSEQST(k.s) are determined for each pair of such neighboring sensor elements SEn,m, SEn.i,m,. The ratios RATFXn,m;n-i,m are then processed to determine representations f(FXn,m(T0)) of original light fluxes FXn,m(T0) onto each one of the sensor elements. Pixel values PXVALn,mof pixels PXn,mof the image IMA are determined based on the representations f(FXn,m(T0)) of original light fluxes, preferably PXVALn,m=f(FXn,m(T0)).

[0011] For the sake of clarity, the division of a shifting sequence into shifting steps still includes that the shifting sequence is executed continuously, i.e. without stopping the shifting between subsequent shifting steps. However, it nevertheless also includes an execution during which the shifting is interrupted between subsequent shifting steps. The same is applicable in case more than one shifting sequence is foreseen: The shifting might be interrupted or not when a first one of the shifting sequences ends and a subsequent one begins.

[0012] Imaging data DATn,m(k,s) of a sensor element SEn,m, in case generated in response to a shifting step SEQST(k,s), includes an event signal SIGn,m(k,s) which represents a polarity POLn,m(k,s) of the respective event EVn,m(k,s), wherein the ratios RATFXn,m;n-i,m of original light fluxes FXn,m(T0), FXn-i,m(T0) of the neighboring sensor elements SEn,m, SEn-i,m are determined based on the polarities.

[0013] For a shifting step SEQST(k,s) in a direction DIRSEQST(k.s) from a first sensor element SEn-i,m to a second sensor element SEn,m, the ratio RATFXn,m;n-i,m(k,s) between the original light flux FXn.i,m(T0) onto sensor element SEn-i,m and the original light flux FXn,m(T0) onto sensor element SEn.m(TO) is set to be

[0014] RATFXn,m;n-i,m(k,s)=FXn-i,m(T0) / FXn,m(T0)>(1 +THRES) in case the event signal SIGn,m(k,s) has positive polarity POS. However, in case the event signal SIGn,m(k,s) has negative polarity NEG, the ratio RATFXn,m;n-i,m(k,s) between the original light flux FXn-i,m(T0) onto sensor element SEn. i,mand the original light flux FXn,m(T0) onto sensor element SEn,m(T0) is set to be RATFXn,m;n- 1 ,m(k, s)= FXn-i ,m(T0) / FXn,m(T0)<(1 -TH RES) .

[0015] The shifting sequence SEQ(k) includes a plurality of shifting steps SEQST(k.si) with i=1 ,... ,ST(k) and ST(k) representing the number of shifting steps of the sequence SEQ(k) in different directions DIRSEQST(k.si). This enables higher accuracy and, in case the different directions are neither parallel nor anti parallel, the generation of a 2D image.

[0016] Preferably, the direction DIRSEQST(k,s1) of at least one SEQST(k,s1) of the shifting steps SEQST(k,s) extends along a row of sensor elements on the sensor chip (111) and the direction DIRSEQST(k,s2) of another one SEQST(k,s2) of the shifting steps SEQST(k,s) extends along a column of sensor elements on the sensor chip. This enables the generation of a 2D image. Ratios RATFXn,m;n-i,m(k,s1), RATFXn,m;n-i,m(k,s2) which are resulting from different shifting steps SEQST(k,s1), SEQST(k,s2) and which refer to the same sensor elements SEn,mand SEn.i,m, are averaged, so that better reliability and stability of the approach is achieved.

[0017] In one embodiment, the shifting sequence SEQ(k) includes a first shifting step SEQST(k,1) in positive x-direction by an amount ASEQST(k,1)=dPXx, a second shifting step SEQST(k,2) in negative x-direction by an amount ASEQST(k,2)=ASEQST(k,1)=dPXx, a third shifting step SEQST(1 ,3) in positive y-direction by an amount ASEQST(k,3)=dPXy, and a fourth shifting step SEQST(k,4) in negative y-direction by an amount ASEQST(k,4)=ASEQST(k,3)=dPXy.

[0018] In another embodiment, the shifting sequence SEQ(k) includes a first shifting step SEQST(k,1) in positive x-direction by an amount ASEQST(k,1)=dPXx, a second shifting step SEQST(k,2) in positive y-direction by an amount ASEQST(k,2)=dPXy, a third shifting step SEQST(k,3) in negative x-direction by an amount ASEQST(k,3)=ASEQST(k,1)=dPXx, and a fourth shifting step SEQST(k,4) in negative y-direction by an amount ASEQST(k,4)=ASEQST(k,2)=dPXy.

[0019] Preferably for both embodiments of the shifting sequence SEQ(k), the x-direction extends along the rows of the sensor chip while the y-direction extends along the columns and the amounts dPXx, dPXy are the pixel pitches in the respective x- and y-direction.

[0020] Preferably, the relative lateral shifting of the projection P200 and the sensor chip is executed with the help of a beam shifting device of the imaging system, which is configured to effect the relative lateral shift of the sensor chip and the projection P200 onto the sensor chip by shifting the projection P200 relative to the sensor chip, preferably by controlling an optics section of the DVS camera, and / or by shifting the sensor chip relative to the projection P200.

[0021] Therein, an optics section of the DVS camera might comprise one or more integrated optical devices for generating the projection P200 of the object from the light flux field LFL emitted by the object, wherein the beam shifting device is realized by at least one of the optical devices of the optics section, wherein the lateral shifting is achieved by changing optical properties of the at least one optical device, preferably by shifting, tilting, and / or deforming the at least one optical device.

[0022] An imaging system for camera based imaging of an object for generating an image IMA of the object is configured to execute the method METH described above and comprises the DVS camera, the beam shifting device for effecting a relative lateral shift of the sensor chip and a projection P200 of at least a section of the object onto the sensor chip, and a control system with a control unit for controlling the beam shifting device and the sensor chip to generate the imaging data DATn,m(k,s) and an evaluation unit for executing the stage EVAL for evaluating the imaging data DATn,m(k,s) to generate the image IMA.

[0023] The DVS camera might comprise an optics section for converting a light flux field LFL from the object into the projection P200 of the object onto the sensor chip, wherein the optics section might comprise one or more optical devices for generating the projection P200 from the light flux field LFL, wherein the beam shifting device can be realized by at least one of the optical devices of the optics section, the lateral shifting can be achieved by changing electrical and / or optical properties of the at least one optical device, preferably by shifting, tilting, and / or deforming the at least one optical device.

[0024] The approach described herein makes use of the high speed of the DVS concept to generate a 2D full field image IMA at high speed, allowing high temporal resolution, e.g. in case a time series of images shall be captured. Thus, the special properties of a DVS camera, also known as “event camera” or “neuromorphic camera”, are beneficially applied to image the object with high resolution. The essential properties of a DVS camera applied in the solution described herein include that the DVS camera detects events, i.e. changes of light flux FX or intensity, respectively, exceeding a pre-defined threshold value THRES and that such change of light flux FX is detected pixelwise, i.e. individually for each sensor element SE of the camera’s sensor chip.

[0025] It is to be understood that the elements and features recited in the appended claims may be combined in different ways to produce new claims that likewise fall within the scope of the present invention. Thus, whereas the dependent claims appended below depend on specific independent or dependent claims, it is to be understood that these dependent claims can, alternatively, be made to depend in the alternative from any preceding or following claim, whether independent or dependent, and that such new combinations are to be understood as forming a part of the present specification.

[0026] DESCRIPTION OF THE FIGURES

[0027] In the following, possible embodiments of the different aspects of the present invention are described in more detail with reference to the enclosed figures. The objects as well as further advantages of the present embodiments will become more apparent and readily appreciated from the following description of the preferred embodiments, taken in conjunction with the accompanying figure in which:

[0028] FIG 1 shows an exemplary application scenario of an imaging system with a DVS camera,

[0029] FIG 2 shows the method METH for imaging,

[0030] FIG 3 shows a top view onto a sensor chip,

[0031] FIG 4 shows a first embodiment of a beam shifting device,

[0032] FIG 5 shows a second embodiment of a beam shifting device,

[0033] FIG 6 shows a third embodiment of a beam shifting device.

[0034] DETAILED DESCRIPTION

[0035] It might be mentioned that dimensions and proportions in the figures are typically not true to scale but exaggerated in order to clarify procedures and effects.

[0036] FIG 1 shows an arrangement of an imaging system 100 for camera-based imaging of the object 200 as well as the object 200 to be scanned and inspected, respectively, by the imaging system 100. The imaging system 100 is configured to generate an image IMA of the object 200.

[0037] The imaging system 100 comprises a DVS camera 110 and a control system 120, wherein the DVS camera 110 is configured and arranged such that it captures a field of view FOV positioned in a z-direction in front of the DVS camera 110. The object 200 to be imaged is arranged at least in parts in the field of view FOV of the DVS camera 110. Thus, a light flux field LFL from the object 200 to the camera 110 falls onto an optics section 112 of the DVS camera 110 which is arranged and configured such that the light flux field LFL representing the object 200 is converted into a projection P200 of the object 200 onto a light sensitive sensor chip 111 of the camera 110. The light flux field LFL itself might be caused by ambient light or, beneficially, by illuminating the object 200 with an illumination system 140 with one or more light sources 140, e.g. LEDs, wherein the illumination system and its one or more light sources 140 are configured and arranged such that they provide a preferably constant light flux onto the object 200 which is preferably homogenous over the entire field of view FOV. The illumination can be monochromatic or it is a multicolor illumination, e.g. in the ultraviolet, short wavelength infrared, or in the visible spectrum. All known and typically applied illumination modalities can be used individually or in a combined manner, like bright field transmissive light illumination, bright field reflected light, dark field reflected light or dark field transmitted light, as well as any kind of mixed forms or modalities, e.g. a combination of transmissive and reflective light etc. In any case and even without a dedicated illumination system, it can be assumed that a light flux field LFL from the object 200 reaches the DVS camera 110 so that the object’s 200 projection P200 falls onto the sensor chip 111.

[0038] The control system 120 of the imaging system 100 comprises a control unit 122, an evaluation unit 123, a memory 124, and one or more processors 125. The control system 120 is connected to the DVS camera 110 via a wired or non-wired connection 121. The processor 125 might be utilized by the control unit 122 and by the evaluation unit 123 to execute a respective software to perform their respective tasks of controlling the camera 110 and evaluating imaging data DAT generated by the camera 110, both corresponding to the method METH introduced below. The software might be stored in the memory 124.

[0039] The control system 120 and its control unit 122, respectively, is configured to control the DVS camera 110 with a control signal CTRL, e.g. with regard to setting of imaging parameters, e.g. focal length etc., of an optics section 112 of the camera 110 and / or of a sensor chip 111 of the camera 110. Moreover, the control system 120 is configured to receive imaging data DAT from the DVS camera 110, both via the connection 121. Received imaging data DAT might be stored at least temporarily in the memory 124 and / or routed to the evaluation unit 123 for further processing.

[0040] The control system 120 and its evaluation unit 123, respectively, is configured to evaluate the imaging data DAT provided by the DVS camera 110 and potentially stored in the memory 123 to create one or more images IMA of the object 200.

[0041] In more general terms, the control system 120 and its evaluation unit 123, respectively, is configured to process imaging data DAT provided by the DVS camera 110 from scanning the object 200 in order to determine pixel values PXVAL(j) of pixels PX(j) of the image IMA. For that purpose, the evaluation unit 123 applies a method METH shown in FIG 2 including a stage INSPECT for generating the imaging data DAT with the DVS camera 110 and a stage EVAL of evaluating the imaging data DAT generated in the stage INSPECT to generate the image IMA. The method METH might be implemented as a respective software executed by the processor 125 of the control system 120 as indicated above. The stage INSPECT might be controlled by the control unit 122 while the stage EVAL might be performed by the evaluation unit 123.

[0042] However, the distribution of such tasks to the components of the overall system can be adapted to individual requirements of a measurement to be performed by the system. The DVS camera 110 has a body 113 which might comprise camera typical electronics including a light sensitive DVS sensor chip 111 as well as an optics section 112. Just as a regular camera lens, the optics section 112 can be a housing with an arrangement of one or more optical devices 112’ like lenses, apertures etc. and imaging parameters like magnification and focal length can be set as for a regular camera lens. Moreover, the optics section 112 is adapted and selected such that it generates a projection P200 of those parts of the object 200, which are arranged in the field of view FOV, onto the sensor chip 111. In addition, the optics section 112 might be suitable to be configured to manipulate the lateral position of the projection P200 relative to the sensor chip 111 as introduced in EP22199206.8 in the context of a beam shifting approach.

[0043] The DVS sensor chip 111 comprises a plurality of light sensitive sensor elements SEn,m with n=1 ,... ,N and m=1 ,... ,M which can be, preferably, arranged in N rows and M columns, perpendicular to each other. For example, the sensor chip 111 might comprise a total number of N*M>106sensor elements SE. As a simplified example, a top view of a sensor chip 111 with only 16*24=384 sensor elements SEn,mis shown in FIG 3. For the sake of clarity, only a few of the sensor elements SEn,mhave been marked with reference signs. The extensions of the planar, essentially two-dimensional sensor chip 111 define a coordinate system x, y, z, wherein the sensor chip’s 111 plane defines the x-direction in a first dimension, e.g. along the rows, as well as the y-direction in a second dimension, e.g. along the columns, while the z-direction in the third dimension is perpendicular to the sensor chip’s 111 plane and surface 111 p, respectively. Individual sensor elements SEn,m have a width wPXx and a height wPXy, e.g. wPXx=wPXy=wPX. In x-direction, the distance of the centers of neighboring sensor elements SEn,m, SEn+i.m, i.e. the x-pixel-pitch, amounts to dPXx. The corresponding distance in y- direction, i.e. the y-pixel-pitch between SEn,m, SEn,m+i, is dPXy. Preferably, dPXx=dPXy=dPX is applicable.

[0044] In contrast to regular cameras, a signal SIGn,m generated by a DVS camera 110 for a given sensor element SEn,m at a point T(i) in time does not directly represent the flux or intensity of light onto the respective sensor element SEn,m, but it depends on a relationship between a light flux onto the sensor element SEn,m at that point T(i) in time and a light flux FX onto the same sensor element SEn,m at a preceding point T(i-1) in time. More concrete, the signal SIGn,m shows whether a light flux FXn,m(T(i)) at the sensor element SEn,mfor a point T(i) in time differs from a light flux FXn,m(T(i-1)) at the same sensor element SEn,mat an earlier point T(i- 1 ) in time by more than a given threshold THRES. In case such ratio RATFX=FXn,m(T(i)) / FXn,m(T(i-1)) at the location of a particular signal element SEn,mis indeed higher than the threshold THRES, it is assumed that an “event” EVn,mhas occurred at the location of that signal element SEn,m- Thus, an event EVn,m detected by a sensor element SEn,m of the sensor chip 111 is the situation in which the light flux onto that sensor element SEn,m changes by more than the threshold THRES. Therein, in case the flux FX changes to a higher flux by an amount according to RATFX with RATFX>THRES, a “positive” event is assumed to have happened. A change of flux to a lower value by an amount according to RATFX with RATFX>THRES corresponds to a “negative” event.

[0045] Thus, the DVS camera 110 is configured to generate for each sensor element SEn,m individually a signal SIGn,m, in the following referred to as an “event signal” SIGn,m, in case a light flux onto the respective sensor element SEn,m changes by more than the threshold THRES, e.g. 15%. Therein, the event signals SIGn,m are not assigned free floating values, but each event signal SIGn.m is selected from a predefined group GRP of event signal types, which group GRP at least includes a positive signal type POS and a negative signal type NEG according to the positive and negative events EV introduced above, in the following occasionally referred to as “polarity” POL. Typically, the group GRP contains only the two polarities POL=POS and POL=NEG. Just for the sake of clarity, in case the intensity change according to RATFX is less than the threshold THRES, no event is assumed to have occurred and no event signal SIG would be created at all. Thus, a polarity value like POL=0 is not applicable and not required while the two polarities POL=POS, POL=NEG are sufficient for the operation of the DVS camera 110. Consequently, event signals SIGn,m are only generated in case of “significant” flux changes, i.e. RATFX>THRES.

[0046] More concrete, considering a specific sensor element SEn,m of the sensor chip 111 of the DVS camera 110: In case the momentary flux FXn,m(T(i)) onto the sensor element SEn,m is higher than the earlier flux FXn,m(T(i-1)) onto the same sensor element SEn,m by more than THRES, the sensor element SEn,m will generate an event signal SIGn,m=POS of positive polarity. In case the momentary flux FXn,m(T(i)) onto the sensor element SEn,m is lower than the earlier flux FXn,m(T(i- 1)) onto the same sensor element SEn,m by more than THRES, the sensor element SEn,m will generate an event signal SIGn,m=NEG of negative polarity. In all other cases, i.e. as long as RATFX<THRES is applicable, no signal and no “imaging data” DAT at all are generated for the respective sensor element SEn,m.

[0047] Any event signals SIGn,m generated by the sensor elements SEn,mduring the stage INSPECT are transferred as part of the imaging data DATn,mfrom the DVS camera 110 to the control system 120 for further processing in the stage EVAL. Such imaging data DATn,minclude the event signal SIGn,m itself, i.e. the polarity POLn,mof the respective event EVn,m, an accurate time stamp TSn.m which stands for the point in time at which the sensor element SEn,m generated the event signal SIGn,m, and an unambiguous identifier ID(SEn,m) of the respective individual event signal SIGn,m generating sensor element SEn,m to ensure that the position POS(SEn,m) of such event signal SIGn,m generating sensor element SEn,m on the sensor chip 111 is known.

[0048] In the end, in contrast to a regular camera, the DVS camera does not generate frames in a regular fashion according to a frame rate, but it immediately provides the output DATn,m to the evaluation unit 123 as soon as the respective event EVn,m occurred. Thus, the emergence of an event EVn,m is the trigger for generating and providing the respective imaging data DATn,m, without any binding to or limitation by a frame rate. The executions of the stages INSPECT and EVAL of the method METH can be performed in parallel or at least overlapping in time because imaging data DATn,m are transmitted immediately after they have been generated so that immediate evaluation is possible. It is not required to wait until a whole frame has been generated or all sensor elements of the chip 111 have been read out.

[0049] The functional principle of the DVS camera 110 necessitates changes of the intensity or light flux FX of the object 200 projection P200 onto the sensor chip 111 and its sensor elements SEn,m- Such a change in flux FX according to RATFX onto the sensor chip 111 can be caused by applying the “beam shifting approach”, e.g. according to EP22199206.8, which causes a flux change in a reproducible way, i.e. in a predefined, well known manner. The beam shifting approach achieves a relative lateral movement of the sensor chip 111 and the projection P200. For that, it foresees a lateral movement of the projection P200 onto the sensor chip 111 relative to the sensor chip 111 by a certain shifting amount.

[0050] The relative lateral movement of the sensor chip 111 and the projection P200 can be achieved by shifting the sensor chip 111 relative to the object 200 and / or by shifting the projection P200 relative to the sensor chip 111. Shifting the sensor chip 111 relative to the object 200 can be achieved by shifting the sensor chip 111 within the camera 110 and / or by shifting the whole camera 110 with the sensor chip 111. Shifting the projection P200 relative to the sensor chip 111 can be achieved by shifting the whole object 200 and with it its original light flux field LFL and / or by controlling the optics section 112 to manipulate the projection P200. Examples for the latter approach are explained in the context of FIGs 4-6. In any case, a “beam shifting device” 130 according to any one of the embodiments introduced in EP22199206 or in PCT / EP2023 / 084588 can be foreseen to be arranged and configured to perform one or more of the solutions introduced above or any other approach for achieving the relative lateral movement of the sensor chip 111 and the projection P200. For the invention introduced herein, the concrete implementation of the beam shifting device 130 is not relevant. Preferably, the sensor chip 111 is shifted, e.g. by means of a piezo drive, or the projection P200 is shifted by means of manipulating one or more components the optics section 112. Therein, the beam shifting device 130 might be integrated into the housing of the DVS camera 110, as indicated in FIG 1.

[0051] Independent from the concrete embodiment of the beam shifting device 130, the control unit 122 of the control system 120 of the imaging system 100 might be configured and connected to the respective beam shifting device 130 to control the beam shifting device 130 such that the aspired relative lateral shift is achieved in a controlled way. In other words, the shifting manner MAN can be assumed to be well known, i.e. the speed, the direction, and the shifting amount and, therewith, the relative position of the projection P220 on the sensor chip 111 are known for each point in time. Such synchronization between the DVS camera 110 and the beam shifting device 130 or, in other words, the fixed temporal connection between imaging data DAT(t) generated at times t and the actual amounts ASEQST(t) and directions DIRSEQST(t) of relative lateral shifting at those times t, and the knowledge of that connection is required in stage EVAL as described herein for reconstructing the image IMA.

[0052] In summary, the relative movement of the projection P200 of the object 200 and the sensor chip 111 can be caused by moving the object 200, moving the projection P200, and / or moving the sensor chip 111. In any case, it is assumed that a relative lateral movement of the projection P200 and the sensor chip 111 is effected in some way, preferably controlled by the control unit 122. Therein, it can be assumed that one or possibly more shifting sequences SEQ(k) with k=1 ,... ,K and K representing the total number of shifting sequences are executed, e.g. by means of the beam shifting device 130, in the stage INSPECT.

[0053] Each one of the shifting sequences SEQ(k) comprises a number SP of shifting steps SEQST(k,s) with s=1,... ,SP. In each shifting step SEQST(k,s), a relative lateral shift of the projection P200 and the sensor chip 111 by an amount ASEQST(k,s) and in a shifting direction DIRSEQST(k.s) is performed. Different shifting steps SEQST(k,s1), SEQST(k,s2) of the same shifting sequence SEQ(k) might, but don’t have to, differ from each other by the shifting amounts ASEQST and / or shifting directions DIRSEQST. The entirety of shifting steps SEQST(k,s) of a sequence SEQ(k) defines a trajectory TR(k) which describes the path along which the projection P200 travels across the sensor chip 111. Different shifting sequences SEQ(k1), SEQ(k2) might be identical to each other or they might comprise different shifting steps.

[0054] In the following, an exemplary shifting sequence SEQ(k) is considered which is executed with the help of the beam shifting device 130 such that the projection P200 and with it the respective light flux FX moves relative to the sensor chip 111 along the rows of sensor elements SE, i.e. in positive x-direction in FIG 3.

[0055] In the example, the shifting amount ASEQST per shifting step SEQST(k,s) is set to one pixel pitch dPX. This results in a shift of the projection P200 by one sensor element SE in the respective direction. In the following, a shift by one pixel pitch dPXx in positive x-direction, i.e. from a sensor element SEn,m to a sensor element with a higher x-index “n+1”, i.e. to sensor element SEn+i,m, is assumed for the explanation of the effect.

[0056] At the beginning of the stage INSPECT, at a point TO in time where no shifting has yet been performed, sensor elements SEn,mand SEn+i,m are exposed to light flux segments FXn,m(T0) and FXn+i,m(T0), respectively. Therein, the expression “light flux segment” as well as the parameter “FXn,m“, respectively, represent that part or “segment” of the whole light flux field which falls onto a particular sensor element SEn,m. In the following, the light fluxes FXn,m(T0) onto sensor elements SEn,m at that point TO in time before the first shifting step are designated “original” light fluxes. In the end, the pixel values PXVALn,m of the pixels PXn,m of the full frame image IMA to be generated might correspond to those original light fluxes FXn,m(T0), i.e. a pixel value PXVALn.m of pixel PXn,m the image IMA shall correspond to the light flux segment FXn,m(T0) onto sensor element SEn,m at the point TO in time, i.e. before the beam shifting has been started.

[0057] The concept introduced herein for reconstructing the image IMA builds on the following: At a point T1>T0 in time, i.e. after execution of the first shifting step SEQST(k,1) in positive x- direction by the amount ASEQST(k,1)=dPXx of one x-pixel-pitch, sensor element SEn,m is exposed to a light flux FXn,m(T1)=FXn-i,m(T0) and sensor element SEn+i,m witnesses a light flux FXn+i,m(T1)=FXn,m(T0). In the transition phase during the shift, any sensor elements SEjj will create an event EVjj in case the light flux FX onto that sensor element is changing by more than the threshold THRES. Therefore, due to the shift, a sensor element SEn+i,m determines the difference of the incoming light flux FX in units of relative thresholds THRES as compared to sensor element SEn,m- When a shifting step SEQST(k,s) is completed, the former light flux segment FXn,mfrom sensor element SEn,mhits sensor element SEn+i,m. Execution of a plurality of shifting steps allows to build a metric of light flux readings for sensor elements SEn+i,m and SEn.m etc. In addition, sensor element SEn-i,m also relates to sensor element SEn,m in the same way, but possibly with another difference than sensor element SEn,m and sensor element SEn+i,m do.

[0058] Thus, each shifting step SEQST(k,s) allows to derive the ratio or difference of fluxes for each pair of neighboring sensor elements SE. In concrete, the method METH for generating the full frame image IMA comprising a number N*M of pixels PXn,m with intensity or light flux information for each sensor element SEn,m of the sensor chip 111 foresees to apply a sequence SEQ(k) of shifting steps SEQST(k,s) in a stage INSPECT of the method METH. In each shifting step SEQST(k,s), a relative lateral shift of the projection P200 and the sensor chip 111 is performed.

[0059] Each shifting step SEQST(k,s) is characterized by the amount ASEQST(k,s) and the direction DIRSEQST(k.s) of shifting. Preferably, but not necessarily, the amounts ASEQST(k,s) can correspond to the pixel pitches of the sensor chip 111. E.g. ASEQST(k,s)=dPXx is preferred in case of a shift in x-direction and ASEQST(k,s)=dPXy is preferred in case of a shift in y-direction. However, shifting amounts ASEQST(k,s) other than one pixel pitch can be used. Regarding the directions DIRSEQST(k.s), it is not necessarily required that the directions DIRSEQST(k.s) are always along rows or columns of sensor elements SE, but they can also be diagonal. In more general terms, instead of linear shifts which are preferably aligned with the matrix pattern of the sensor chip’s 111 sensor elements SE, other motions like circular paths or diagonal shifts etc. are also possible. However, the shifting sequence’s SEQ(k) trajectory TR(k) is required to be known for subsequent processing of generated imaging data DATn,m(k).

[0060] A shifting sequence SEQ(k) comprising shifting steps SEQST(k,1), SEQST(k,2), ... , SEQST(k.SP) might be repeated several times. In that case, the stage INSPECT includes a plurality of identical shifting sequences SEQ(k). However, in case the stage INSPECT shall include a plurality of shifting sequences, it is also possible that different, i.e. not identical, shifting sequences SEQ(k) are included. Therein, shifting sequences SEQ(k) might differ from each other in the order of shifting steps SEQST(k,s) and / or in the characteristics of individual shifting steps SEQST(k,s), wherein the term “characteristics of individual shifting steps SEQST(k,s)” refers to the selection of the amounts ASEQST(k,s) and / or directions DIRSEQST(k.s) of the respective shifting steps SEQ(k). For example and as indicated above, a first shifting step SEQST(k,1) of sequence SEQ(k) might be executed in positive x-direction. In case sensor element SEn,mhas generated a positive event signal SIGn,m(T1)=POS in response to the first shifting step SEQST(k,1), i.e. at point T 1 >T0 in time, it can be derived for the point TO in time, i.e. for the original light flux, that the original light flux FXn-i,m(T0) onto sensor element SEn-i,m must be significantly higher than the original light flux FXn,m(T0) onto sensor element SEn,m. Therein and in the following, the threshold THRES represents the scale which is applicable when the term “significantly” is utilized. Thus, a second light flux FX2 being “significantly higher” or “significantly lower” than a first light flux FX1 means that the second flux FX2 is higher or lower than the first light flux by at least the threshold THRES, i.e. FX2 / FX1 >(1+THRES) or FX2 / FX1 <(1-THRES). However, as a consequence it can be derived for the original light fluxes FX(TO) at the point TO in time that the ratio RATFX between FXn-i,m(T0) and FXn,m(T0) must have been at least RATFX=FXn. i,m(T0) / FXn,m(T0)>(1+THRES).

[0061] Assuming moreover that sensor element SEn-i,m has generated a negative event signal SIGn-

[0062] 1,m(T1)=NEG in response to the first shifting step SEQST(k,1), i.e. at point T1 >T0 in time, it can be derived for the point TO in time, i.e. for the original light flux, that the original light flux FXn.

[0063] 2,m(T0) onto sensor element SEn.2,m must be significantly lower than the original light flux FXn.

[0064] 1,m(T0) onto sensor element SEn-i,m. As a consequence it can be derived for the original light fluxes FX(TO) at the point TO in time that the ratio RATFX between original light fluxes FXn.

[0065] 2,m(T0) and FXn-i,m(T0) onto sensor elements SEn-2,m(T0) and SEn-i,m(T0) must not have been more than RATFX=FXn-2,m(T0) / FXn-i,m(T0)<(1-THRES).

[0066] Thus, the execution of one shifting step in x-direction allows to derive ratios RATFX of original light fluxes FXn,m(T0), FXn-i,m(T0) onto neighboring sensor elements SEn,m, SEn-i,m for each pair of such neighboring sensor elements SEn,m, SEn-i,m. Thus light flux FXn,m(T0) onto SEn,m is related to light flux FXn-i,m(T0) onto SEn-i,m and light flux FXn-i,m(T0) onto SEn-i,m is related to light flux FXn-2,m(T0) onto SEn-2,m etc. so that, in the end, all light fluxes FXn,m(T0) can be related to one arbitrary reference value, e.g. FXn,m’(T0) onto an arbitrary reference sensor element SEref=SEn ,m’. For example, SEref might be a sensor element in the central region of the sensor chip 111.

[0067] Therein, the expression “neighboring” and its extension depends on the shifting amount ASEQST(k,s) and direction DIRSEQST(k.s) of a shifting step SEQST(k,s). In the preferred situation, the shifting steps are specified such that the directions are along the rows or columns of sensor elements and the shifting amounts are identical to the respective pixel pitches. Thus, sensor elements which are indeed adjacent to each other are “neighboring” sensor elements. However, in case the shifting amount ASEQST(k,s) is selected to be, for example, PIT>1 pixel pitches in x-direction, sensor elements SEn,m and SEn+piT,m would be neighboring sensor elements.

[0068] The shifting sequence SEQ(k) might include a second shifting step SEQST(k,2), which can be executed in a direction opposite to the first shifting step SEQST(k,1), i.e. in negative x-direction. After the first shifting step SEQST(k,1), the sensor elements SE are potentially experiencing other light fluxes than before the first shifting step SEQST(k,1) so that for each sensor element the absolute threshold might also have changed. Consequently, the event signals SIGn,m(T2) generated by the second shifting step SEQST(k,2) can be, but are not necessarily inverse to the respective event signals SIGn,m(T1) generated by the first shifting step SEQST(k,1).

[0069] However, a first situation is assumed in which the sensor element SEn,m generates an event signal SIGn,m=NEG in response to the second shifting step SEQST(k,2) which is inverse to the event signal SIGn,m=POS of the opposite first shifting step SEQST(k,1). In that case, it can be derived for the point TO in time that the original light flux FXn,m(T0) onto sensor element SEn,mmust have been significantly lower than the original light flux FXn.i,m(T0) onto sensor element SEn-i,m. Consequently, it can be derived for the original light fluxes FX(TO) at the point TO in time that the ratio RATFX between fluxes on SEn.2,m(T0) and SEn,m(T0) must be RATFX=FXn,m(T0) / FXn-i,m(T0)<(1-THRES). This essentially supports the insight from the first shifting step SEQST(k,1).

[0070] Assuming a second situation in which the sensor element SEn,m does not generate an event signal at all in response to the second, opposite shifting step SEQST(k,2), it can only be derived for the point TO in time that the original light flux FXn,m(T0) onto sensor element SEn,m has not been significantly higher or lower than the original light flux FXn-i,m(T0) onto sensor element SEn. i.m, i.e. (1-THRES)<FXn,m(T0) / FXn-i,m(T0)<(1+THRES).

[0071] In any one of the assumed situations, insights from two such shifting steps SEQST(k,1), SEQST(k,2) can be combined by averaging the results from the individual steps. In the first assumed situation, in which the insight from the first shifting step has been confirmed, it can be assumed that FXn-i,m(T0) / FXn,m(T0)>(1+THRES) is applicable and, for example, FXn. i,m(T0)=FXn,m(T0)*(1+THRES). In the second assumed situation of non-consistent event signals, an averaging step might be reasonable, e.g. FXn.i,m(T0)=1X*FXn,m(T0)*(1+THRES). The generation of a 2D image IMA requires that the stage INSPECT also includes shifting steps SEQST(k,s) the direction DIRSEQST(k.s) of which is the y-direction. Such additional shifting steps SEQST(k,s) in y-direction might be included in the same shifting sequence SEQ(k) as described above or they might form elements of an additional shifting sequence SEQ(k’). However, conceptually the same procedure and insights for determination of ratios of the original light fluxes FX(TO) are applicable as for the shifting in x-direction as explained above.

[0072] This allows insights about original light fluxes FXn,m-2(T0), FXn,m-i(T0), FXn,m(T0) etc. onto sensor elements SEn,m-2, SEn.m-i, SEn,m etc. so that, in the end, information about all original light fluxes FXn,m(T0) onto all sensor elements SEn,m of the sensor chip 111 can be derived with n=1 ,... ,N and m=1 ,..,M and with N, M representing the numbers of sensor elements SE of the sensor chip 111 in x- and y- direction.

[0073] Thus, the shifting steps SEQST(k,s) of the one or more sequences SEQ(k) are preferably executed in positive and / or negative x- and y- directions. In that way, application of one or more suitable sequences SEQ(k) of shifting steps SEQST(k,s) allows to determine the ratios RATFX introduced above, from which insights about all original light fluxes FXn,m(T0) for each sensor element SEn,mcan be derived, which are, however, related to a reference sensor element SEref, because, as shown above, the method only provides ratios of light fluxes of neighboring sensor elements.

[0074] Those determined original light fluxes FXn,m(T0) can then be used to determine the pixel values PXVALn.m of the pixels PXn,m of the image IMA, e.g. PXVALn,m=f(FXn,m(T0)) with f preferably being a strictly monotone function, e.g. in the simplest case f(x)=c*x with c being a constant.

[0075] Thus, the application of the one or more shifting sequences SEQ(k) with shifting steps SEQST(k,s) in x- and y-directions allows to generate the full frame image IMA by determining the pixel values PXVALn,m of the pixels PXn,m of the image IMA. Therein and due to the approach to determine ratios of light fluxes, the pixels PXn,m of the image IMA, corresponding to respective sensor elements SEn,m, are assigned values PXVALn,m in units of the threshold THRES or in terms of threshold increments, respectively, and relative to a reference sensor element SEn,m.

[0076] When looking at the whole sensor chip 111 , all sensor elements SE will build up a similar metric with neighboring sensor elements in x-direction and y-direction, respectively, exactly in the same way. This means, that all sensor elements SE in a row will build up a relation based on differences in light fluxes onto neighboring sensor element SEn,mand SEn+i,mor, correspondingly, SEn,mand SEn,m+i. When looking at the shifting steps SEQST(k,s) in the x- and the y-directions in the sequence SEQ(k), the sensor elements SEn,mof the whole sensor chip 111 get related with respect to the photocurrent created by each sensor element SEn,m in terms of threshold THRES increments. This leads to the aspired full field relation of the entirety of sensor elements SE across the entire chip 111.

[0077] Based on the information about the relative light fluxes in terms of the threshold THRES, a full coverage of the entire number of sensor elements SE on the sensor chip 111 can be determined. Thus, the full field 2D intensity distribution or “image” IMA of the original light flux field onto the sensor chip 111 can be reconstructed, in the minimum embodiment from a combination of a single shift SEQST(1 ,1) in x-direction by one x-pixel-pitch dPXx and a second shift SEQST(1 ,2) in y-direction by one y-pixel-pitch dPXy. Preferably, in case the pixel pitches dPXx, dPXy of the sensor chip 111 are different from each other, the shifting amount ASEQST(k,s) of a certain shifting step SEQST(k,s) depends on the respective shifting direction DIRSEQST(k.s) to ensure that each shifting amount ASEQST(k,s) of a certain shifting step SEQST(k,s) corresponds to the pixel pitch in the respective shifting direction DIRSEQST(k.s).

[0078] For example, one full shifting sequence SEQ(1)={+X;-X;+Y;-Y} might be embodied as a motion cycle including a first shifting step SEQST(1 ,1)=”+X” in positive x-direction by an amount ASEQST(1 ,1)=dPXx, a second shifting step SEQST(1 ,2)=”-X” in negative x-direction by an amount ASEQST(1 ,2)=ASEQST(1 ,1)=dPXx, a third shifting step SEQST(1 ,3)=”+Y” in positive y- direction by an amount ASEQST(1 ,3)=dPXy, and a fourth shifting step SEQST(1 ,4)=”-Y” in negative y-direction by an amount ASEQST(1 ,4)=ASEQST(1 ,3)=dPXy. Execution of the shifting sequence SEQ(1) achieves that a particular light flux segment FXn,m which originally, at the point TO in time, hits a certain first sensor element SEn,m travels via SEn+i,m, SEn,m, and SEn,m+i back to SEn,m. Thus, a group of three sensor elements SE is covered by the shifting steps of SEQ(1), namely SEn.m, SEn+i,m, SEn,m+i.

[0079] Another example for a full shift sequence SEQ(2)={+X;+Y;-X;-Y} might include a first shifting step SEQST(2,1)=”+X” in positive x-direction by an amount ASEQST(2,1)=dPXx, a second shifting step SEQST(2,2)=”+Y” in positive y-direction by an amount ASEQST(2,2)=dPXy, a third shifting step SEQST(2,3)=”-X” in negative x-direction by an amount ASEQST(2,3)=ASEQST(2,1)=dPXx, and a fourth shifting step SEQST(2,4)=”-Y” in negative y- direction by an amount ASEQST(2,4)=ASEQST(2,2)=dPXy. This shifting sequence SEQ(2) represents a motion along the edges of a rectangle so that a particular light flux segment FXn,mwhich originally, at the point TO in time, hits a certain first sensor element SEn,m, travels via SEn+i,m, SEn+i,m+1> and SEn,m+i back to SEn,m. Thus, a group of four sensor elements SE is covered by the shifting steps, namely SEn,m, SEn+i,m, SEn+i,m+i, and SEn,m+i, instead of only three sensor elements in the first shift sequence SEQ(1) example. With the second example SEQ(2), execution of the shifting sequence SEQ(2) achieves the coverage of the entire sensor chip 111.

[0080] The design of a shifting sequence SEQ(k) can be used to control the degree of coverage of the sensor chip 111 by the light fluxes and to control the way how imaging data DATn,m of the different sensor elements SEn,m get related. In principle, it is sufficient for a 2D image IMA that the shifting sequence SEQ(k) includes one shifting step in x-direction and one shifting step in y- direction. Thus, regarding the second example SEQ(2), it would be sufficient to have the first half of the sequence SEQ(2)={+X;+Y;-X;-Y} completed, i.e. the steps SEQST(2,1)=”+X” and SEQST(2,2)=”+Y”, to derive a first relation of all sensor elements SE of the sensor chip 111 as described above. However, with the execution and completion of the sequence’s SEQ(2) second half including steps SEQST(2,3)=”-X” and SEQST(2,4)=”-Y”, even more information and imaging data DAT, respectively, is created. Moreover, a closing condition is provided since - after completion of the sequence SEQ(2)- each sensor element SEn,mis again hit by the original light flux segment FXn,mas before the shifting sequence SEQ(2) was executed.

[0081] In the examples SEQ(1), SEQ(2) described above, the image IMA is created from the imaging data DAT generated due to the execution of the shifting steps SEQST(k,s) of the respective shifting sequence SEQ(k). In an embodiment for the creation of a series of images IMA, e.g. a time series, a measurement for capturing the imaging data DAT might include several executions of the same shifting sequence in a row and over time. Thus, sequences SEQ(k1), SEQ(k2), SEQ(k3),... are executed with SEQ(k1)=SEQ(k2)=SEQ(k3). For example, each shifting sequence SEQ(ki) with i=1 ,2,3,... might be set up as SEQ(2), i.e. SEQ(ki)=SEQ(2). The imaging data DAT resulting from the second half of a first cycle k, i.e. SEQST(k,3) and SEQST(k,4), and the first half of a second, subsequent cycle k+1 , i.e. SEQST(k+1 ,1) and SEQST(k+1 ,2), can be combined for the creation of the images IMA. Therein, the term “cycle” k means the one-time execution of the respective shifting sequence SEQ(k). With such “sharing” approach, the number of full frame images IMA can be increased without increasing the number of shifting steps or, as an alternative, the same number of images IMA can be generated with fewer shifting steps. Moreover, it makes the method more stable and allows to detect changes of the observed object 200 over time.

[0082] To increase the intensity resolution, the stage INSPECT might comprise a first shifting sequence SEQ(1) with shifting steps SEQST(1 ,s) in an original order, i.e. SEQST(1 ,1), SEQST(1 ,2), SEQST(1 ,3), and SEQST(1,4), and subsequently a second shifting sequence SEQ(2) with the same shifting steps, but in reversed order, i.e. SEQST(2,1)=SEQST(1 ,4), SEQST(2,2)=SEQST(1,3), SEQST(2,3)=SEQST(1,2), and SEQST(2,4)=SEQST(1,1). Depending on the capabilities of the technology applied for achieving the relative lateral shift of the sensor chip 111 and the projection P200, also other shifting sequences and / or shifting amounts ASEQST larger than the respective pixel pitches dPXx, dPXy can be used to optimize accuracy and resolution of the resulting image IMA.

[0083] In general, the selection and order of shifting steps SEQST(k,s) of the shifting sequence SEQ(k) and, therewith, the resulting trajectory TR(k) along which a particular light flux segment FXn,m travels across the sensor chip 111 can be arbitrary and different from the first and second exemplary sequences SEQ(1), SEQ(2).

[0084] In any case, the entire concept of determining ratios of original light fluxes onto neighboring sensor elements in a first step and calculating representations of the original light fluxes from the ratios for each sensor element in a second step remains the same for any shifting sequence. However, it is required that the applied one or more shifting sequences SEQ(k) of the stage INSPECT and especially the individual shifting steps SEQST(k,s) and their parameters DIRSEQST(k.s) and ASEQST(k,s) are known because that knowledge is processed when the imaging data DAT, generated during the stage INSPECT, are used to reconstruct the light flux ratios in units of threshold THRES increments between neighboring sensor elements SE, as they are directly related to the shifting path TR(k) and to the individual shifting steps SEQST(k,s), respectively. Thus, the stage EVAL processes the entirety of imaging data DAT resulting from execution of the shifting steps SEQST(k,s) of all applied shifting sequences SEQ(k) to generate the image IMA or, possibly, a plurality of images IMA.

[0085] The method METH of generating an image IMA as shown in FIG 2 includes the stage INSPECT and the stage EVAL. In the stage INSPECT, or alternatively in a preliminary stage, the shifting sequences SEQ(k) are specified by defining the steps SEQST(k,s) regarding the respective shifting amounts ASEQST(k,s) and directions DIRSEQST(k.s). The stage INSPECT includes the execution of the specified shifting steps SEQST(k,s), controlled by the control unit 122. At the same time, i.e. during execution of the shifting steps SEQST(k,s) in the stage INSPECT, imaging data DAT generated by the sensor chip 111 are collected and provided to the evaluation unit 123. The evaluation unit 123 processes the received imaging data DAT as well as the knowledge about the corresponding shifting steps SEQST(k,s) of the specified one or more sequences SEQ(k) to generate the full field image IMA. The generation of the image IMA includes, for each pixel PXn,mof the image IMA, the determination of pixel values PXVALn,m- Due to the shift motion, a linear or preferably 2D relation of the photocurrent created by each sensor element SEn,m in response to the impinging light flux FXn,m is created for all the sensor elements. This relation is measured in units of the threshold value THRES. This value THRES can be selected and set by a user in the camera software or in the control unit 122. When one pixel PXn,m and signal element SEn,m, respectively, is selected as the reference pixel, all other pixels can be related to it as introduced above. Neighboring pixels with a change of one increment can be assigned a +1 or -1 , respectively, corresponding to positive or negative polarities POL of the respective events EV. For a higher number t of threshold increments registered for a particular pixel PXn,m’, a +t or -t with t>1 will be attributed to that pixel PXn,m’, again depending on the positive or negative polarities POL of the number of events EV.

[0086] In that way, the full matrix of pixels PX of the image IMA can be filled with these increment numbers as pixel values PXVAL, resulting in a kind of incremental image. The range of increment numbers can be high due to the high dynamic range of the camera of e.g. 120dB, which is a factor of 106between the lowest and highest signal. An increment resolution of, for example, THRES=10% change in photocurrent would lead to (1+THRES)int=1.1int=106and, thus, a number of approx. int=145 as different intensity levels.

[0087] Selection of THRES=20% results in int=76 resolvable different intensity values. Since the scale is nonlinear due to the scaling according to (1+THRES)n, the linear range of 106intensity values of the dynamic range is discretized into 145 or 76 portions of different size, respectively.

[0088] The incremental image IMA generated as described above is a kind of differential incremental image since it contains the respective differences in photocurrent in units of threshold increments THRES. To determine a regular intensity image IMAim, in which the pixel values and the intensities they represent have a linear relationship, the differential incremental image IMA has to be transferred into a linear image IMAim. Due to the relative nature of the threshold increments, it is preferred to first integrate the differential incremental image IMA into an absolute incremental image IMAabs with a common scale for all its pixels. In a subsequent step, this absolute incremental image IMAabs is converted into the image IMAim with linear scaling for intensity. To do so, the nonlinear threshold increments, the amounts of which are exponentially increasing with increasing absolute increment number, need to be converted into linearly scaled values for the photocurrent. The kind of integration of the differential incremental image IMA can be achieved by selecting one pixel as a reference. For example, the central pixel of the image IMA would be suitable to be the reference, while corner or edge pixels are less preferred since they do not have neighboring pixels in all directions and are thus potentially less accurate. A central pixel is preferred, since the integration of noise or deviation is minimal, since the number of steps in each direction in the image IMA is minimal.

[0089] The conversion from the differential image IMA to the linear image IMAim requires to step along a row or column, respectively, of pixels PX of the image IMA. For each step to a neighboring pixel PXn,m, the increment value POLn,m with consideration of its sign “+” or is added to the value of the neighboring pixel as shown above in the concrete example with two shifting steps SEQST(k,1), SEQST(k,2) in x-direction. With that, the differential increments POL can be integrated to absolute increments, with reference to the reference pixel, by traversing over the whole array of sensor elements SE of the sensor chip 111. After finishing the integration, the minimal resulting value PXVAL of all pixels PX can be subtracted, with consideration of its sign, from the pixel values of all pixels in order to get non-negative pixel values only.

[0090] To retrieve the normal intensity image IM A|inwith linear scale from the rescaled absolute incremental image IMAabs is a straight forward approach. The linear value A for each pixel value a is calculated by A=(1+THRES)a. For example, A can be in the range [1 ; max~106] for a 120dB dynamic range of the DVS camera. Any other scaling can be used as offset in the absolute incremental image IMAabs or even for the linear image IMAim.

[0091] Technically, the shift trajectory TR(k) and the respective shifting directions DIRSEQST and amounts ASEQST of the shifting steps forming the trajectory can lead to different increment numbers of threshold values when the light flux pattern shift forth and back on the sensor. This information can be used to improve the resolution in the incremental image retrieved from integer numbers only to real numbers. As the absolute threshold value THRESabs(n) for an event EV(n) is defined relative to the actual photocurrent value PCURR(n-l) of the most recent event EV(n-1), i.e. THRESabs(n)=THRES*PCURR(n-1), an actual absolute threshold value THRESabsfor a certain sensor element depends on the level of the photocurrent PCLIRR of that sensor element. In effect, typically the number of events EV during a shifting sequence can be larger when going from lower to higher light flux FX, i.e. with positive events POS, due to smaller absolute threshold increments THRESabsfrom one event to the next, when compared to going from higher to lower light flux, i.e. with negative events NEG. This difference can be used to calculate for real and non-integer numbers of increments. In a simple approach, the values are simply averaged from the positive and negative increments for each sensor element from the two different shift directions in the shift path. Alternatively, one can stay with integer numbers by simply adding the two values from positive and negative increments and compensating for this in the formula, e.g. A=(1+THRES)a / 2.

[0092] In case more than one event occurs, the resolution of the intensity or related photocurrent can be improved even further: The sequence of events can be fitted and extrapolated with a polynomial function, e.g. linear, quadratic, or cubic, or an exponential function in order to generate interpolated reals numbers of events at the exact time for the beginning of the shift at the exact pixel position and at the end of the shift at the exact position of the shifted exact pixel position. Then the difference of the two extrapolated values of increment number end and increment number beginning will be a real number of increments which can be used for the respective pixel in the incremental image. This method can also be applied, if there are different shift path for the way back and forth. If there is no or just one event during the shift, the interpolating method cannot be applied. Therefore, depending on the number of events, the one or the other method can be applied to create integer or reals increment number for the pixels for the incremental image. The subsequent conversion from the incremental image to the linearly scaled image can be done as described above for the two cases.

[0093] Regarding the general setup of the imaging system, the beam shifting device 130 can be embodied in different ways.

[0094] For example and as shown in FIG 4, the beam shifting device 130 can be embodied as a transparent, plano-parallel plate, which has two parallel surfaces 131 s1 , 131s2 and which is positioned in a z-direction between the object 200 and the sensor chip 111, so that the light flux field LFL emitted by the object 200 passes through the parallel surfaces 131 s1 , 131s2 and through the inner of the plate 131 to reach the sensor chip 111. Lateral shifting is achieved by tilting the plate 130 around the x- and / or y-direction.

[0095] In another example, as shown in FIG 5, the beam shifting device can be embodied as a pair of mirrors 132 which are arranged parallel to each other and, in a default position, under an angle of 45° with regard to the z-direction, such that the light flux field LFL emitted by the object 200 is reflected by a first one of the mirrors 132 to the second one of the mirrors 132 and from the second one of the mirrors 132 towards the sensor chip 111. Lateral shifting is achieved by tilting both mirrors by equal angels ALPHA around the x- and / or y-direction relative to the default position. In a further example, shown in FIG 6, the beam shifting device 130 can be embodied as an electro-optical device which has a plate 133 with two surfaces 134 and which is positioned in a z-direction between the object 200 and the sensor chip 111 , so that the light flux field LFL emitted by the object 200 passes through the surfaces 134 and through the inner of the plate 133 to reach the sensor chip 111. Lateral shifting is achieved by applying a voltage at preferably transparent electrodes arranged at the surfaces 134.

[0096] For each one of those embodiments of the beam shifting device 130, the respective elements might be integrated into the optics section 112. In any case, the beam shifting device 130 would preferably be controlled by the control unit 122.

[0097] Since DVS cameras detect changes in the flux FX per sensor element SE and generate an event if the photocurrent rises or lowers by a pre-defined amount THRES or more, they automatically adapt themselves for each sensor element SE to the actual irradiance level in the operating range of the DVS camera 110. In concrete, but to be understood as an example, this might result in an effective dynamic range of more than 100dB or even 120dB and above. This capability is a result of the DVS sensor chip’s 111 intrinsic ability to adapt locally to the respective signal strengths so that it applies the full well capacity of a sensor element SE with different exposure times where the respective sensor element is monitoring for a change in light exposure. In effect this makes up the effective dynamic range of 100dB and above.

[0098] While the present invention has been described above by reference to various embodiments, it should be understood that many changes and modifications can be made to the described embodiments. It is therefore intended that the foregoing description be regarded as illustrative rather than limiting, and that it be understood that all equivalents and / or combinations of embodiments are intended to be included in this description. Thus, the invention is not restricted to the above illustrated embodiments, but variations can be derived by a person skilled in the art without deviation from the scope of the invention.

Claims

Claims1. Method METH for camera based imaging of an object (200) for generating an image IMA of the object (200) with an imaging system (100), wherein the imaging system (100) comprises the camera (110), which is embodied as a DVS camera (110) with a light sensitive sensor chip (111) with a plurality of sensor elements SEn,m, each of which generates imaging data DATn,m in case of an event EVn,m of change of light flux FX onto that sensor element SEn,m by more than a given threshold THRES, wherein the method METH comprises- a stage INSPECT of observing the object (200) with the DVS camera (110), including execution of one or more shifting sequences SEQ(k), wherein each shifting sequence SEQ(k) comprises one or more shifting steps SEQST(k,s), wherein during each shifting step SEQST(k,s) a relative lateral shifting of a projection P200 of at least a section of the object (200) and the sensor chip (111) is executed in a shifting direction DIRSEQST(k.s) to generate the imaging data DATn,m(k,s),- a stage EVAL of evaluating the generated imaging data DATn,m(k,s), wherein o ratios RATFXn,m,n-i,m of original light fluxes FXn,m(T0), FXn-i,m(T0) onto sensor elements SEn,m, SEn-i,m being neighbored to each other in the shifting direction DIRSEQST(k.s) are determined for each pair of such neighboring sensor elements SEn,m, SEn-i.m, o the ratios RATFXn,m;n-i,m are processed to determine representations f(FXn,m(T0)) of original light fluxes FXn,m(T0) onto each one of the sensor elements, o pixel values PXVALn,m of pixels PXn,m of the image IMA are determined based on the representations f(FXn,m(T0)) of original light fluxes, preferably PXVALn,m=f(FXn,m(T0)).

2. Method according to claim 1 , wherein imaging data DATn,m(k,s) of a sensor element SEn,m, in case generated in response to a shifting step SEQST(k,s), includes an event signal SIGn,m(k,s) which represents a polarity POLn,m(k,s) of the respective event EVn,m(k,s), wherein the ratios RATFXn,m;n-i,m of original light fluxes FXn,m(T0), FXn-i,m(T0) of the neighboring sensor elements SEn,m, SEn-i,m are determined based on the polarities.

3. Method according to any one of claims 1 to 2, wherein, for a shifting step SEQST(k,s) in a direction DIRSEQST(k.s) from a first sensor element SEn.i,mto a second sensor element SEn,m,- in case the event signal SIGn,m(k,s) has positive polarity POS, the ratio RATFXn,m;n-i,m(k,s) between the original light flux FXn-i,m(T0) onto sensor element SEn-i,m and the original light flux FXn,m(T0) onto sensor element SEn,m(T0) is set to be RATFXn,m;n-i,m(k,s)=FXn- i.m(T0) / FXn.m(T0)>(1+THRES),- in case the event signal SIGn,m(k,s) has negative polarity NEG, the ratio RATFXn,m;n-i,m(k,s) between the original light flux FXn.i,m(T0) onto sensor element SEn.i,mand the original light flux FXn,m(T0) onto sensor element SEn,m(T0) is set to be RATFXn,m;n-i,m(k,s)=FXn- i,m(T0) / FXn,m(T0)<(1-THRES).

4. Method according to any one of claims 1 to 3, wherein the shifting sequence SEQ(k) includes a plurality of shifting steps SEQST(k.si) with i=1 , ... ,ST(k) in different directions DIRSEQST(k.si).

5. Method according to claim 4, wherein the direction DIRSEQST(k,s1) of at least one SEQST(k,s1) of the shifting steps SEQST(k,s) extends along a row of sensor elements SEn,m on the sensor chip (111) and the direction DIRSEQST(k,s2) of another one SEQST(k,s2) of the shifting steps SEQST(k,s) extends along a column of sensor elements SEn,m on the sensor chip (111).

6. Method according to any one of claims 4 to 5, wherein ratios RATFXn,m;n-i,m(k,s1), RATFXn,m;n- i.m(k,s2) which are resulting from different shifting steps SEQST(k,s1), SEQST(k,s2) and which refer to the same sensor elements SEn,mand SEn-i,m, are averaged.

7. Method according to any one of claims 4 to 6, wherein the shifting sequence SEQ(k) includes- a first shifting step SEQST(k,1) in positive x-direction by an amount ASEQST(k,1)=dPXx,- a second shifting step SEQST(k,2) in negative x-direction by an amountASEQST(k,2)=ASEQST(k,1)=dPXx,- a third shifting step SEQST(1 ,3) in positive y-direction by an amount ASEQST(k,3)=dPXy, and- a fourth shifting step SEQST(k,4) in negative y-direction by an amountASEQST(k,4)=ASEQST(k,3)=dPXy8. Method according to any one of claims 4 to 6, wherein the shifting sequence SEQ(k) includes- a first shifting step SEQST(k,1) in positive x-direction by an amount ASEQST(k,1)=dPXx,- a second shifting step SEQST(k,2) in positive y-direction by an amount ASEQST(k,2)=dPXy,- a third shifting step SEQST(k,3) in negative x-direction by an amountASEQST(k,3)=ASEQST(k,1)=dPXx, and- a fourth shifting step SEQST(k,4) in negative y-direction by an amountASEQST(k,4)=ASEQST(k,2)=dPXy.

9. Method according to any one of claims 1 to 8, wherein the relative lateral shifting of the projection P200 and the sensor chip (111) is executed with the help of a beam shifting device(130) of the imaging system (100), which is configured to effect the relative lateral shift of the sensor chip (111) and the projection P200 onto the sensor chip (111) by- shifting the projection P200 relative to the sensor chip (111), preferably by controlling an optics section of the DVS camera, and / or- shifting the sensor chip (111) relative to the projection P200.

10. Method according to claim 9, wherein an optics section (112) of the DVS camera (110) comprises one or more integrated optical devices (112’) for generating the projection P200 of the object (200) from the light flux field LFL emitted by the object (200), wherein the beam shifting device (130) is realized by at least one of the optical devices (112’) of the optics section (112), wherein the lateral shifting is achieved by changing optical properties of the at least one optical device (112’), preferably by shifting, tilting, and / or deforming the at least one optical device (112’).

11. Imaging system (100) for camera (110) based imaging of an object (200) for generating an image IMA of the object (200), configured to execute a method METH according to any one of claims 1 to 10 and comprising- the camera (110), which is embodied as a DVS camera (110) with a light sensitive sensor chip (111) with a plurality of sensor elements SE(i), each of which generates imaging data DAT(i) in case of an event EV(i) of change of light flux FX onto that sensor element SE(i) by more than a given threshold THRES,- the beam shifting device (130) for effecting a relative lateral shift of the sensor chip (111) and a projection P200 of at least a section of the object (200) onto the sensor chip (111), and- a control system (120) with o a control unit (122) for controlling the beam shifting device and the sensor chip (111) to generate the imaging data DATn,m(k,s) and o an evaluation unit (123) for executing the stage EVAL for evaluating the imaging data DATn.m(k,s) to generate the image IMA.

12. Imaging system (100) according to claim 11 , wherein the DVS camera (110) comprises an optics section (112) for converting a light flux field LFL from the object (200) into the projection P200 of the object (200) onto the sensor chip (111), wherein the optics section (112) comprises one or more optical devices (112’) for generating the projection P200 from the light flux field LFL, wherein- the beam shifting device (130) is realized by at least one of the optical devices (112’) of the optics section (112),- the lateral shifting is achieved by changing electrical and / or optical properties of the at least one optical device (112’), preferably by shifting, tilting, and / or deforming the at least one optical device (112’).

Images