Autofocus imaging circuit for event camera, autofocus imaging device and method thereof

Abstract

The present disclosure relates generally to auto-focus imaging circuitry configured to obtain a first plurality of brightness change events from a brightness change event detection element for a first adjusted focus, obtain a second plurality of brightness change events from the brightness change event detection element for a second adjusted focus, and determine a focus for auto-focusing on a scene based on the first plurality of brightness change events and the second plurality of brightness change events.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to an autofocus imaging circuit, an autofocus imaging apparatus, and an autofocus imaging method. BACKGROUND

[0002] Generally, it is known to automatically adjust the focus of a lens (stack) of an imaging system (e.g. a camera), also referred to as autofocus. For example, known mirrorless cameras generally perform passive autofocus by using phase detection or contrast detection.

[0003] In phase detection, a dedicated focus pixel is used to determine the focus at a certain frequency, which is separated from the main imaging frequency.

[0004] In contrast detection, a scan is performed over a range of focal distances to determine the best focus.

[0005] Despite the techniques for autofocus, it is generally desirable to provide an autofocus imaging circuit, an autofocus imaging apparatus, and an autofocus imaging method. SUMMARY

[0006] According to a first aspect, the present disclosure provides an autofocus imaging circuit configured to: obtain, for a first adjusted focus, a first plurality of luminance change events from a luminance change event detection element; obtain, for a second adjusted focus, a second plurality of luminance change events from the luminance change event detection element; and determine a focus for autofocusing a scene based on the first and second pluralities of luminance change events.

[0007] According to a second aspect, the present disclosure provides an autofocus imaging apparatus comprising: an imaging device comprising a plurality of luminance change event detection elements and a plurality of imaging elements; and an autofocus imaging circuit configured to, for each luminance change event detection element: obtain, for a first adjusted focus, a first plurality of luminance change events from the luminance change event detection element; obtain, for a second adjusted focus, a second plurality of luminance change events from the luminance change event detection element; and determine a focus for autofocusing a scene based on the first and second pluralities of luminance change events.

[0008] According to a third aspect, the present disclosure provides an autofocus imaging method comprising: obtaining, for a first adjusted focus, a first plurality of luminance change events from a luminance change event detection element; obtaining, for a second adjusted focus, a second plurality of luminance change events from the luminance change event detection element; and determining a focus for autofocusing a scene based on the first and second pluralities of luminance change events.

[0009] Further aspects are set forth in the dependent claims, the following description, and the accompanying drawings. Attached Figure Description

[0010] The embodiments are explained by way of example with reference to the accompanying drawings, in which:

[0011] Figure 1 A block diagram of an autofocus imaging apparatus according to the present disclosure is depicted;

[0012] Figure 2 A block diagram depicting another embodiment of an autofocus imaging apparatus according to the present disclosure is shown;

[0013] Figure 3 An embodiment of a probability filter for determining focus based on event density is described;

[0014] Figure 4 A schematic diagram of a hybrid sensor according to this disclosure is depicted;

[0015] Figure 5 An autofocus imaging method according to this disclosure is described;

[0016] Figure 6 Another embodiment of the autofocus imaging method according to the present disclosure is described;

[0017] Figure 7 Another embodiment of the autofocus imaging method according to this disclosure is described; and

[0018] Figure 8 Another embodiment of the autofocus imaging method according to this disclosure is described. Detailed Implementation

[0019] Provide a reference Figure 1 Before a detailed description of the embodiments, a general description will be given.

[0020] As mentioned at the beginning, known imaging systems or cameras can use either phase detection or contrast detection for autofocus.

[0021] However, phase detection requires lenses with relatively high numerical apertures. Therefore, phase detection is used in large imaging systems (e.g., microscopes, DSLR cameras) but not in small devices such as mobile phones.

[0022] On the other hand, contrast detection can generally be used regardless of the lens's numerical aperture. However, for autofocus, the entire focal length range of the imaging system used is typically scanned. Therefore, contrast detection is limited by the measurement rate of the image sensor used and the speed and accuracy of the motor used, which adjusts the focal length by changing the position of the focusing lens. Thus, contrast detection can be considered time-inefficient.

[0023] Furthermore, phase detection as well as contrast detection require image processing in order to focus on a moving object (often referred to as focus tracking or AI (Artificial Intelligence) servo). However, the image processing in known systems is thus limited by the hardware used for the image processing. For example, in small devices (e.g. mobile phones), battery power is limited, so for example users of the device want to save battery power, the processing power usually needs to remain small, or, conversely, if the processing is performed on the device, a large battery is needed, which is usually not desired.

[0024] For these reasons, known systems can not be able to focus correctly, especially in scenes with (fast) moving objects. This is also the case for difficult lighting conditions (e.g. low ambient light, night, dark room, etc.) or for scenes with low contrast (e.g. similar colors of different objects).

[0025] Therefore, some embodiments relate to an autofocus imaging circuit configured to obtain, for a first adjusted focus point, a first plurality of brightness change events from a brightness change event detection element, obtain, for a second adjusted focus point, a second plurality of brightness change events from the brightness change event detection element, and determine a focus point based on the first and second plurality of brightness change events for autofocusing a scene.

[0026] The autofocus imaging circuit according to the present disclosure can comprise any known circuit, e.g. one or more processors (e.g. CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), neuromorphic computing platform, etc. Furthermore, it is well known that the autofocus imaging circuit can comprise a computer, a server, etc. system.

[0027] Such an autofocus imaging circuit can be configured to obtain, for a first (and / or second) adjusted focus point, a first (and / or second) plurality of brightness change events from a brightness change event detection element.

[0028] For example, it is well known that an autofocus imaging circuit can be employed in a camera system comprising a lens system for focusing incoming light. It is well known that the lens system can be adjusted to focus on a first (or second) focus point (or focal plane, focal point, etc.) such that light reflected on or originating from the adjusted focal plane can be collected, while light from other focal planes is not collected.

[0029] If the amount of light in the focal plane changes, the brightness change event detection element can detect this change if the change is above a predetermined threshold of the amount of light.

[0030] For example, in a first adjusted focus, an amount of light can be incident on the intensity variation event detection element, thereby generating a first plurality of intensity variation events, while in a second adjusted focus, adjusted according to a feedback loop based on the first plurality of intensity variation events, another amount of light can be incident on the intensity variation event detection element, thereby generating a second plurality of intensity variation events.

[0031] The focus can be determined (and auto-focusing can be achieved) based on the second plurality of intensity variation events, which in turn is based on the second adjusted focus, which is based on the first plurality of intensity variation events, such that the focus is determined based on the first plurality of intensity variation events and the second plurality of intensity variation events, in the aggregate. However, in some embodiments, the second plurality of intensity variation events is fed to the auto-focusing imaging circuit in a feedback loop, thereby establishing an iterative method to further refine the determination of the focus.

[0032] The amount of light detected by the (any) imaging element can be expressed as intensity. Thus, if a change in intensity above a predetermined threshold is detected in an adjusted focus, the intensity variation event detection element can generate an event indicating whether a positive change in intensity (e.g., brighter) or a negative change in intensity (e.g., dimmer) is detected, although the present disclosure is not limited to this, as in some embodiments, only the absolute value of the change in intensity can be detected.

[0033] Such an intensity variation event detection element can be configured by (pixels of) event cameras, neuromorphic cameras, silicon retina, dynamic vision sensors (DVS), etc.

[0034] For example, the intensity variation event detection element can be part of an optical sensor (e.g., a DVS), i.e., its pixels or groups of pixels.

[0035] Each (group of) pixel of a DVS can independently, asynchronously produce sparse, time-resolved measurements of perceived intensity variations.

[0036] When a pixel measures a change in intensity greater than a predetermined threshold, an event is generated, which includes the (absolute or relative) time at which the change was detected (e.g., in microsecond resolution) and the polarity of the change (positive (i.e., brighter) or negative (i.e., dimmer)). However, in some embodiments, it is not necessary to determine the polarity of the event. In some embodiments, the intensity variation event detection element can be further configured to determine the absolute intensity of the light.

[0037] Such a DVS can have a dynamic range exceeding 120 decibels, e.g., a sensitivity of less than 15% relative to a minimum contrast and a latency of less than 100 microseconds, without limiting the present disclosure to any of these attributes.

[0038] However, for predetermined lighting scenarios, this configuration can maximize speed while reducing redundant data.

[0039] Because the redundant information is minimized, the processing power can also be minimized. Thus, for example, focus tracking can be implemented in a time-efficient manner, making it suitable for devices with limited computing power (e.g., mobile phones)

[0040] According to the present disclosure, if the lighting conditions (e.g., light intensity) are substantially constant, a (temporal) variation of the contrast within the scene can be identified. Based thereon, motion estimation, object tracking, video data compression, gesture recognition, etc. can be performed.

[0041] As already discussed, the first and second focus can be adjusted, and thus, according to the present disclosure, by evaluating the data provided (directly or indirectly) from the brightness variation event detection element, a first plurality of brightness variation events for the first adjusted focus (however, typically no or only one brightness variation event can be detected at all) and a second plurality of brightness variation events for the second adjusted focus (or none or only one) can be obtained by the autofocus imaging circuit.

[0042] Based on the first plurality of brightness variation events and the second plurality of brightness variation events, the autofocus imaging circuit can be configured to determine the focus.

[0043] For example, if a scene (e.g., an object) is to be imaged, the focal plane in which the scene lies can be found by comparing the number of brightness variation events of the first plurality of brightness variation events and the second plurality of brightness variation events.

[0044] It can be envisaged that a lower number of brightness variation events indicates a better focal plane (the present disclosure is not limited thereto), such that the least events can indicate the focus corresponding to the first or second adjusted focus.

[0045] Further, it can be envisaged that a focus scan is performed, i.e. a plurality of focuses is adjusted and for each adjusted focus a plurality of brightness variation events is obtained, in order to generate a brightness variation curve from which the ideal focus can be determined by determining a minimum of the curve, etc.

[0046] Thus, the autofocus imaging circuit according to the present disclosure is capable of determining an autofocus based on the events sensed in different adjusted focuses, such that the scene (e.g., the (object of interest)) can be automatically focused on based on the determined focus.

[0047] In some embodiments, the first plurality of brightness variation events and the second plurality of brightness variation events are indicative of an event density.

[0048] That is, for each adjusted focus point, a corresponding brightness change event can be determined, such that the brightness change events can be expressed as a function of the adjusted focus points, resulting in an event density, which in some embodiments can be determined by the autofocus imaging circuit in order to determine the focus point, for example, based on a minimum of the event density, without the disclosure being limited thereto.

[0049] However, in some embodiments, the focus point can be known as a function of time, and the event density can also be known as a function of time, such that both graphs can be mapped in order to determine the focus point.

[0050] For example, if a moving object (e.g., a bird, a football, etc.) is to be tracked, a probabilistic filter (e.g., an extended Kalman filter (EKF)) representing the event density can be used to estimate the motion of the object. In this case, the focus scan can be limited, for example, to a range smaller than the entire focus range, and centered on an estimated trajectory of the tracked object, such that the speed of determining the focus point can be improved.

[0051] In some embodiments, the autofocus imaging circuit is further configured to determine a contrast based on the determined event density.

[0052] Since the contrast can be intertwined with the brightness, it is known according to the disclosure that determining the contrast based on the event density can be sufficient without performing brightness imaging, but rather event sensing with lower computational complexity.

[0053] In some embodiments, a focus scan is performed to determine the contrast of each focus point, and the contrast of the image projected on the image sensor is detected by measuring the rate of change of the event rate or by the rate of change of the spatial density of the events. Because the events are generated asynchronously, detecting the contrast according to the disclosure can be performed faster (e.g., by an order of magnitude) than in known systems, because the events are generated asynchronously, as described above.

[0054] Furthermore, in some embodiments, an autofocus imaging device is provided that includes a high-bandwidth lens (e.g., a liquid lens), such that low-latency measurements can be performed, which can enable closed-loop autofocus. However, the disclosure is not limited to the case of a high-bandwidth lens, since any lens with an arbitrary bandwidth can be used.

[0055] Generally, if a video of a (fast) moving object is captured, autofocus is required that adapts its speed to the speed at which the object propagates through different focus planes.

[0056] Therefore, in some embodiments, the autofocus imaging circuit is further configured to track an object based on the determined focus point.

[0057] The object can be any object, whose position relative to the camera can change due to movement of the object and / or the camera.

[0058] The tracking can be performed after determining the focus using known tracking techniques, or according to the present disclosure by re-determining the focus (in order to find the object).

[0059] Furthermore, if DVS pixels are used, such re-determination of the focus (i.e. refocusing) can be performed between frames of (color) imaging pixels, such that the refocusing can not be visible (or rarely visible) in the captured image (or consecutive images).

[0060] In general, the present disclosure is not limited to finding (determining) the focus based on the above-described methods (e.g. based on event density), as in some embodiments, the determination of the focus is also based on a neural network processing the first plurality of luminance change events and the second plurality of luminance change events.

[0061] Thus, in some embodiments, the determination of the focus and / or the tracking can be performed with a neural network or any other artificial intelligence, which can utilize known algorithms.

[0062] In some embodiments, the determination of the focus and at least one of the tracking is based on a training of the artificial intelligence.

[0063] The artificial intelligence (AI) can use a machine learning based approach or an explicit feature based approach, e.g. shape matching, e.g. by edge detection, histogram based approaches, template matching based approaches, color matching based approaches, etc. In some embodiments, a machine learning algorithm can be used to perform object recognition, e.g. for comparing a detected predefined object with a recognized object to increase the correctness of the detection, which can be based on at least one of: scale-invariant feature transform (SIFT), gray level co-occurrence matrix (GLCM), Gabor features, Tubeness, etc. Furthermore, the machine learning algorithm can be based on a classifier technique, wherein such machine learning algorithm can be based on at least one of: random forest; support vector machine; neural network, Bayesian network, etc. Furthermore, the machine learning algorithm can apply a deep learning technique, wherein such deep learning technique can be based on at least one of: autoencoder, generative adversarial network, weakly supervised learning, guidance, etc.

[0064] The supervised learning can also be based on a regression algorithm, a perception algorithm, a Bayesian classification, a Naiver Bayer classification, a next neighbor classification, an artificial neural network, etc.

[0065] In such embodiments, the artificial intelligence can be fed ground truth data, which can correspond to or be based on predefined objects, such that the artificial intelligence can learn to determine the focus point.

[0066] However, the neural network can also employ semi-supervised or unsupervised learning.

[0067] For example, for semi-supervised or unsupervised learning, an existing autofocus method can be used, which can be a high-performance method, but which can determine the autofocus rather slowly. However, with such a method, (raw) data can be captured, which can allow properties of the (slow) autofocus method according to the present disclosure to be transferred to the (faster) method. In case of training the autofocus method according to the existing method, such a method can be considered as a kind of semi-supervised learning, or in case of manually implementing the autofocus, such a method can be considered as unsupervised learning.

[0068] In some embodiments, the neural network is based on or comprises a spiking neural network.

[0069] The spiking neural network can employ analog, digital or hybrid neuron models or any related or unrelated asynchronous computing hardware or software.

[0070] In some embodiments, the intensity variation event detection element can produce sparse time-resolved data asynchronously, which can be processed by an asynchronous and / or sparse computing platform, e.g. a spiking neural network, without limitation.

[0071] The spiking neural network can be implemented in (neuromorphic hardware), which can be implemented in an imaging system or the like. However, the spiking neural network can derive an algorithm for determining the focus point and / or for tracking an object externally, and the found algorithm can be implemented in an autofocus imaging circuit according to the present disclosure.

[0072] Such a configuration can result in low-power and low-latency control of the autofocus imaging circuit, such that a fast and efficient autofocus determination can be performed.

[0073] In some embodiments, the autofocus imaging circuit is further configured to obtain at least one of the first plurality of intensity variation events and the second plurality of intensity variation events based on detection of a reflection of the structured emission pattern.

[0074] For example, a light source (e.g. of a camera system), which can comprise a plurality of lasers, laser diodes, VCSELs (vertical cavity surface emitting lasers) or the like, can emit structured light, e.g. a plurality of dots, a grid or the like.

[0075] Based on the reflection of the emission pattern (and based on the first and / or second adjusted focus), the autofocus imaging circuit can determine the focus, e.g. by determining the event density based on the structured emission pattern. For example, after the first (and only) adjusted focus, the focus can have been determined, as the degradation or blurring of the structure can indicate the focus event density.

[0076] However, generally, as discussed herein, the first and second focus can be adjusted so as to determine the focus based on the structured light.

[0077] Further, the emission pattern can comprise one or different wavelengths, wherein the present disclosure is not limited to the emission of visible light, as any suitable electromagnetic frequency can be utilized, e.g. infrared light.

[0078] For example, the pixels of the dynamic vision sensor can be responsive in the infrared light range as well as in the visible light range, whereas the present disclosure is not limited thereto.

[0079] Thereby, the focus can be (rapidly) determined in dark areas, i.e. areas where the ambient light is below a predetermined threshold, e.g. (dark) rooms, laboratories, etc.

[0080] In some embodiments, the determination of the focus is based on a feedback loop from the first plurality of luminance change events and the second plurality of luminance change events.

[0081] For example, it can be desirable to find the focus with the least absolute number of events based on the plurality of events. In such an example, the number of events in the second adjusted focus can be compared to the first adjusted focus. If the number of events of the second adjusted focus is higher (or lower) than the number of events of the first adjusted focus, the other focus corresponding to the direction of focus at which a lower number of events is expected can be adjusted.

[0082] Feedback loops are well-known, and thus unnecessary explanations are omitted this time. According to the present disclosure, the feedback loop can be envisaged, as the luminance change event detection elements (e.g. DVS pixels) do not perform imaging, such that a fast feedback loop can be had compared to known autofocus methods such as contrast scanning, phase detection, etc.

[0083] Some embodiments relate to an autofocus imaging device comprising: an imaging arrangement comprising a plurality of luminance change event detection elements and a plurality of imaging elements; and an autofocus imaging circuit configured to, for each luminance change event detection element: obtain, for a first adjusted focus, a first plurality of luminance change events from the luminance change event detection element; obtain, for a second adjusted focus, a second plurality of luminance change events from the luminance change event detection element; and determine, based on the first plurality of luminance change events and the second plurality of luminance change events, a focus for autofocusing on a scene, as discussed herein.

[0084] The autofocus imaging device can be implemented in, or correspond to, any device having a variable focus lens, e.g. a mobile phone, a digital (still or video) camera, a portable computer, a laptop, etc., a smart television, etc.

[0085] The autofocus imaging device can comprise a hybrid imaging device, e.g. a hybrid DVS (dynamic vision sensor) camera, which can comprise an image sensor having both conventional imaging elements as well as luminance change event detection elements.

[0086] Such image sensor can be based on known image sensor technology, e.g. CMOS (complementary metal-oxide-semiconductor), CCD (charge-coupled device), etc. It is also known that the imaging elements of the sensor can be based on known technology, e.g. CAPD (current- assisted photodiode), SPAD (single-photon avalanche diode) or any other diode technology.

[0087] The number of imaging elements can be higher than the number of luminance change event detection elements (e.g. a ratio of three to one), since if a high resolution image is desired, it can not be necessary to have a large number of luminance change event detection elements. On the other hand, if a more accurate focus determination is desired, the ratio can change in favor of the luminance change event detection elements.

[0088] Furthermore, the arrangement of imaging elements and luminance change event detection elements can be any arrangement that the skilled person can envisage and can depend on the application. In some embodiments, the arrangement can be based on a predetermined pattern of luminance change event detection elements and imaging elements, e.g. a checkerboard pattern, a simplified checkerboard pattern (i.e. regular, but with individual elements differing in number), an irregular pattern, etc. Furthermore, the predetermined pattern can be based on a uniform distribution of luminance change event detection elements in order to find a sufficiently average focus of the entire scene, but with “blind” pixels between the imaging elements. On the other hand, in some embodiments, if a good scene average focus is not important, but if having a maximum number of connected image points (i.e. pixels) is important, the luminance change event detection elements can be arranged in one (or more) row(s) (or column(s)) of the sensor. The present disclosure is not limited to these two embodiments of image sensor arrangement, since any (also arbitrary or random) arrangement can be envisaged.

[0089] However, in some embodiments, the imaging device can be based on a dual sensor, i.e. an image sensor and a luminance change event detection sensor having a plurality of luminance change event detection elements. In such embodiments, a beam splitter can be employed to direct the incoming light onto each sensor.

[0090] In some embodiments, the autofocus imaging circuit is further configured to determine, for each luminance change event detection element: a luminance change event density, for determining a spatial luminance change event density of the plurality of luminance change event detection elements.

[0091] In such embodiments, for each luminance change event detection element, an event density can be determined. This results in a plurality of event densities distributed over the two-dimensional plane of the image sensor, i.e. from which a spatial luminance change event density can be inferred, in order to determine a focus point that is suitable for the two-dimensional nature of the image sensor in a sufficient manner.

[0092] In some embodiments, the imaging apparatus has a hybrid sensor with luminance change event detection elements and a predetermined pattern of a plurality of imaging elements, as discussed herein.

[0093] Some embodiments relate to an autofocus imaging method comprising: obtaining, for a first adjusted focus point, a first plurality of luminance change events from luminance change event detection elements; obtaining, for a second adjusted focus point, a second plurality of luminance change events from the luminance change event detection elements; and determining a focus point based on the first plurality of luminance change events and the second plurality of luminance change events, for autofocusing on a scene, as discussed herein.

[0094] As discussed herein, the autofocus imaging method according to the present disclosure can be performed with an autofocus imaging circuit, an autofocus imaging device, etc.

[0095] In some embodiments, the autofocus imaging method further comprises determining an event density based on the first plurality of luminance change events and the second plurality of luminance change events, as discussed herein. In some embodiments, the autofocus imaging method further comprises determining a contrast based on the determined event density, as discussed herein. In some embodiments, the autofocus imaging method further comprises tracking an object based on the determined focus point. In some embodiments, the determination of the focus point is further based on a neural network processing the first plurality of luminance change events and the second plurality of luminance change events, as discussed herein. In some embodiments, the neural network comprises a spiking neural network, as discussed herein. In some embodiments, the autofocus imaging method further comprises obtaining at least one of the first plurality of luminance change events and the second plurality of luminance change events based on a detection of a reflection of a structured emission pattern, as discussed herein. In some embodiments, the determination of the focus point is based on a feedback loop from the first plurality of luminance change events and the second plurality of luminance change events, as discussed herein.

[0096] The methods described herein are also implemented in some embodiments as a computer program that, when executed on a computer and / or processor, causes the computer and / or processor to perform the method. In some embodiments, there is also provided a non-transitory computer-readable recording medium in which a computer program product is stored, which, when executed by a processor (e.g., the processor described above), causes the method described herein to be performed.

[0097] Returning Figure 1 depicts a block diagram of an embodiment of an autofocus imaging device 1 according to the present disclosure. Such an imaging device can be implemented in a mirrorless camera.

[0098] The autofocus imaging device 1 has a lens stack 2 configured to focus incident light onto a hybrid sensor 3 according to the present disclosure, which comprises imaging elements and brightness change event detection elements. In some embodiments, the hybrid sensor 3 is also implemented as a hybrid DVS (Dynamic Vision Sensor) camera.

[0099] The lens stack 2 has a motor for adjusting the distance between the lenses of the lens stack 2, so that the focus can be adjusted.

[0100] The hybrid sensor 3 is configured to generate a plurality of brightness change events in response to changes in the intensity of the incident light.

[0101] According to the present disclosure, the brightness change event data is transmitted to an autofocus imaging circuit 4, which is implemented as a neuromorphic multi-core processor, which is a specialized processor for event data.

[0102] The autofocus imaging circuit 4 causes the lens stack to adjust a second focus distance after obtaining a first plurality of events in a first adjusted focus distance, so that the autofocus imaging circuit 4 generates and obtains a second plurality of brightness change events. As discussed herein, the autofocus imaging circuit 4 is configured to determine a focus point based on the first plurality of brightness change events and the second plurality of brightness change events. This focus point is then transmitted to the lens stack 2 and adjusted therein.

[0103] In this configuration, an autofocus feedback loop is constituted, so that each successively adjusted focus point is adjusted based on the number of events of the current iteration. This means that the present disclosure is not limited to the case of exactly finding two pluralities of brightness change events, and the feedback loop can be repeated (a predetermined) number of times until the optimal focus point is determined.

[0104] Figure 2 Another embodiment of an autofocus imaging device 1’ according to the present disclosure is depicted. Such an imaging device can be implemented in a mirror camera.

[0105] With Figure 1instead of the hybrid sensor 3, the arrangement of beam splitter 5, the image sensor 6 with "regular" (color) imaging pixels and the intensity change event detection sensor 7.

[0106] In this embodiment, the lens stack 2 focuses the incoming light onto the beam splitter 5, which distributes the light onto the image sensor 6 and the intensity change event detection sensor 7.

[0107] As discussed above, the autofocus imaging circuit 4 obtains a plurality of intensity change events from the intensity change event detector 7.

[0108] The skilled person will appreciate that the present disclosure is not limited to the optical path described herein.

[0109] For example, the beam splitter 5 can be replaced by an adjustable mirror configuration, such that the incoming light can first be transmitted to the intensity change event detection sensor 7 in order to determine the focus, and after the focus has been found, the light can be directed to the image sensor 6.

[0110] In other embodiments, one of the two optical paths to either the image sensor 6 or the intensity change event detection sensor 7 can also be blocked when determining the focus or when imaging the scene, respectively.

[0111] Figure 3 An embodiment of a probability filter 20 for determining the focus based on the event density is depicted.

[0112] The following figure depicts the event density (symbolic curve) which is generated based on a focus scan (i.e. for a predetermined number of focal points, a corresponding number of a plurality of intensity change events is obtained).

[0113] In this embodiment, the minimum of the event density is searched for in order to find the best focus, since if the number of events is low, it can be concluded that also the amount of scattered and diffracted light is low, so that from the minimum of the event density the focus in the plane of interest (e.g. the object plane) is inferred.

[0114] From Figure 3 It can be seen that the abscissa of both figures is time, i.e. both figures are focal length versus time (top) and event density versus time (bottom). In this embodiment, it is assumed that the adjusted focal length (of the focus scan) has a direct influence on the event density, and considering the respective delay, the event density can be mapped onto the same time axis as the focal length is mapped onto.

[0115] Thus, the focus can be inferred from the respective event density minimum.

[0116] Figure 4 A schematic diagram of a hybrid sensor 30 according to the present disclosure is depicted.

[0117] As is known, the hybrid sensor 30 has luminance change event detection elements 31 and color imaging elements 32.

[0118] As discussed herein, the hybrid sensor 30 has a predetermined pattern of luminance change event detection elements 31 and color imaging elements 32. Every second row of the hybrid sensor has only color imaging elements 32, wherein in the remaining rows, the rows with one luminance change event detection element 31 in the middle of the row alternate with rows with two luminance change event detection elements in the second column from the left and from the right, respectively.

[0119] In other words, the present hybrid sensor 30 has an exemplary 7x7 pixels (of course, the present disclosure is not limited thereto). In the first row, from the left side, there are three color imaging elements 32, then a luminance change event detection element 31, then again three color imaging elements 32. In the second row, there are seven color imaging elements. In the fourth row, again there are seven color imaging elements 32. The fifth to seventh rows are a repetition of the first three rows.

[0120] As indicated, it should be appreciated that, as is known, the present disclosure is not limited to a hybrid sensor with 7x7 pixels, as generally any number of pixels can be provided (e.g., in the range of megapixels), and the embodiment of the hybrid sensor 30 can be interpreted as a simplification of such a case.

[0121] Figure 5 A block diagram of an autofocus imaging method 40 according to the present disclosure is depicted.

[0122] The autofocus imaging method 40 can be performed by an autofocus imaging circuit 4 according to Figure 1 and 2 as discussed herein.

[0123] In 41, a first plurality of luminance change events is obtained from luminance change event detection elements, as discussed herein.

[0124] In 42, a second plurality of luminance change events is obtained from luminance change event detection elements, as discussed herein.

[0125] In 43, a focus point is determined based on the first plurality of luminance change events and the second plurality of luminance change events, as discussed herein.

[0126] Figure 6 Another embodiment of an autofocus imaging method 50 according to the present disclosure is depicted. The autofocus imaging method 50 differs from the autofocus imaging method 40 in that a density of events is determined, on the basis of which a contrast is determined, for determining a focus point, as discussed herein.

[0127] The autofocus imaging method 50 can be performed by the autofocus imaging circuit 4 according to Figure 1 and 2 as discussed herein.

[0128] In 51, a first plurality of brightness change events is obtained from the brightness change event detection element as discussed herein.

[0129] In 52, a second plurality of brightness change events is obtained from the brightness change event detection element as discussed herein.

[0130] In 53, a focus point is determined based on the first plurality of brightness change events and the second plurality of brightness change events as discussed herein.

[0131] In 54, a contrast is determined as a derivative of the event density based on the event density.

[0132] In 55, the focus point is determined by evaluating a zero point of the contrast based on the determined contrast.

[0133] Figure 7 Another embodiment of an autofocus imaging method 60 according to the present disclosure is depicted. In this embodiment, in addition to the autofocus imaging method 40 according to Figure 5 an object is identified and tracked based on a spiking neural network as discussed herein.

[0134] The autofocus imaging method 60 can be performed by the autofocus imaging circuit 4 according to Figure 1 and 2 as discussed herein.

[0135] In 61, a first plurality of brightness change events is obtained from the brightness change event detection element as discussed herein.

[0136] In 62, a second plurality of brightness change events is obtained from the brightness change event detection element as discussed herein.

[0137] In 63, a focus point is determined based on the first plurality of brightness change events and the second plurality of brightness change events as discussed herein.

[0138] In 64, an object is identified by a spiking neural network and the object is tracked. The focus point is then readjusted based on a motion of the object and based on an algorithm developed by the spiking neural network.

[0139] Figure 8 Another embodiment of an autofocus imaging method 70 according to the present disclosure is depicted.

[0140] In 71, a luminance change event is obtained. Arrow 72 indicates that the luminance change event is obtained based on a feedback loop, i.e. a first plurality of luminance change events is obtained in a first adjusted focus point, based on which a second focus point is adjusted, and a second plurality of luminance change events is obtained. The feedback loop can have a predetermined number of iterations for determining the focus point (e.g. more than the first and second adjusted focus points), or the feedback loop can run as long as it is able to determine a focus point above a predetermined threshold value.

[0141] In 73, the focus point is determined based on the obtained luminance change events.

[0142] In other words, the focus point is determined based on a closed loop to drive the lens of a camera system having an adjustable focus lens.

[0143] It will be appreciated that the embodiments describe a method having an exemplary ordering of method steps. However, the particular order of the method steps is given merely for illustrative purposes and should not be construed as having a binding force. For example, Figure 5 The order 41 and 42 in the embodiments of FIG. 4 can be interchanged. Furthermore, Figure 6 The order 51 and 52 in the embodiments of FIG. 5 can be interchanged. Furthermore, Figure 7 The order 61 and 62 in the embodiments of FIG. 6 can also be interchanged. Other variations of the order of the method steps will be apparent to the skilled person.

[0144] All units and entities described in the present specification and claimed in the appended claims can be implemented as integrated circuit logic, e.g. on a chip, if not stated otherwise, and the functionality provided by such units and entities can be implemented by software, if not stated otherwise.

[0145] Insofar as the above disclosed embodiments are implemented at least in part using a software controlled data processing device, it will be appreciated that the provision of such software controlled computer programs, and the provision of transmission, storage or other media for such computer programs, are envisaged as aspects of the present disclosure.

[0146] Note that the present technology can also be configured as described below.

[0147] (1) An auto-focusing imaging circuit configured to:

[0148] obtain a first plurality of luminance change events from the luminance change event detection element for a first adjusted focus point;

[0149] obtain a second plurality of luminance change events from the luminance change event detection element for a second adjusted focus point; and

[0150] determine a focus point based on the first and second plurality of luminance variation events for auto-focusing on a scene.

[0151] (2) The auto-focusing imaging circuit according to (1), wherein the first and second plurality of luminance variation events represent event density.

[0152] (3) The auto-focusing imaging circuit according to (2), further configured to:

[0153] determine event density.

[0154] (4) The auto-focusing imaging circuit according to (3), further configured to:

[0155] determine contrast based on the determined event density.

[0156] (5) The auto-focusing imaging circuit according to any one of (1) to (4), further configured to:

[0157] track the focus point based on the determined focus point.

[0158] (6) The auto-focusing imaging circuit according to any one of (1) to (5), wherein the determination of the focus point is further based on a neural network processing the first and second plurality of luminance variation events.

[0159] (7) The auto-focusing imaging circuit according to (6), wherein the neural network comprises a spiking neural network.

[0160] (8) The auto-focusing imaging circuit according to any one of (1) to (7), further configured to:

[0161] obtain at least one of the first and second plurality of luminance variation events based on detection of reflection of a structured emission pattern.

[0162] (9) The auto-focusing imaging circuit according to any one of (1) to (8), wherein the determination of the focus point is based on a feedback loop from the first and second plurality of luminance variation events.

[0163] (10) An auto-focusing imaging device comprising:

[0164] an imaging arrangement comprising a plurality of luminance variation event detection elements and a plurality of imaging elements; and

[0165] an auto-focusing imaging circuit configured to, for each luminance variation event detection element:

[0166] for the first adjusted focus, obtaining a first plurality of intensity variation events from the intensity variation event detection elements;

[0167] for the second adjusted focus, obtaining a second plurality of intensity variation events from the intensity variation event detection elements; and

[0168] determining a focus for auto-focusing the scene based on the first and second pluralities of intensity variation events.

[0169] (11) The auto-focusing imaging device of (10), the auto-focusing imaging circuitry is further configured to, for each intensity variation event detection element:

[0170] determining an intensity variation event density for determining a spatial intensity variation event density of the plurality of intensity variation event elements.

[0171] (12) The auto-focusing imaging device of any one of (10) and (11), wherein the imaging apparatus has a hybrid sensor having the intensity variation event detection elements and a predetermined pattern of a plurality of imaging elements.

[0172] (13) An auto-focusing imaging method, comprising:

[0173] for the first adjusted focus, obtaining a first plurality of intensity variation events from the intensity variation event detection elements;

[0174] for the second adjusted focus, obtaining a second plurality of intensity variation events from the intensity variation event detection elements; and

[0175] determining a focus for auto-focusing the scene based on the first and second pluralities of intensity variation events.

[0176] (14) The auto-focusing imaging method of (13), further comprising:

[0177] determining an event density based on the first and second pluralities of intensity variation events.

[0178] (15) The auto-focusing imaging method of (14), further comprising:

[0179] determining a contrast based on the determined event density.

[0180] (16) The auto-focusing imaging method of any one of (13) to (15), further comprising:

[0181] tracking an object based on the determined focus.

[0182] (17) The autofocus imaging method of any one of (13) to (16), wherein the determination of the focus point is further based on a neural network processing the first and second plurality of brightness change events.

[0183] (18) The autofocus imaging method of (17), wherein the neural network comprises a spiking neural network.

[0184] (19) The autofocus imaging method of any one of (13) to (18), further comprising:

[0185] obtaining at least one of the first and second plurality of brightness change events based on detection of a reflection of a structured emission pattern.

[0186] (20) The autofocus imaging method of any one of (13) to (19), wherein the determination of the focus point is based on a feedback loop from the first and second plurality of brightness change events.

[0187] (21) A computer program comprising program code which, when executed on a computer, causes the computer to perform the method of any one of (11) to (20).

[0188] (22) A non-transitory computer-readable recording medium storing therein a computer program product which, when executed by a processor, causes the method of any one of (11) to (20) to be performed.

Claims

1. An autofocus imaging circuit, configured as follows: For the first adjusted focus, a first plurality of brightness change events are obtained from the brightness change event detection element; For the second adjustment focus, a second plurality of brightness change events are obtained from the brightness change event detection element; as well as The focus is determined based on the first plurality of brightness change events and the second plurality of brightness change events, and is used for automatic focusing of the scene. In response to autofocusing on the scene, light is directed to multiple imaging elements to image the scene, wherein the brightness change event detection element and the multiple imaging elements are arranged in a predetermined pattern in a hybrid sensor.

2. The autofocus imaging circuit according to claim 1, wherein, The first plurality of brightness change events and the second plurality of brightness change events represent the event density.

3. The autofocus imaging circuit according to claim 2 is further configured as follows: Determine the event density.

4. The autofocus imaging circuit according to claim 3 is further configured as follows: Contrast is determined based on the established event density.

5. The autofocus imaging circuit according to claim 1 is further configured as follows: Based on the identified focus tracking object.

6. The autofocus imaging circuit according to claim 1, wherein, The determination of the focus is further based on a neural network that processes the first plurality of brightness change events and the second plurality of brightness change events.

7. The autofocus imaging circuit according to claim 6, wherein, The neural network includes a spiking neural network.

8. The autofocus imaging circuit according to claim 1 is further configured as follows: Based on the detection of reflections from the structured emission pattern, at least one of the first plurality of brightness change events and the second plurality of brightness change events is obtained.

9. The autofocus imaging circuit according to claim 1, wherein, The focus is determined based on a feedback loop of the first plurality of brightness change events and the second plurality of brightness change events.

10. An autofocus imaging device, comprising: The imaging device includes multiple brightness change event detection elements and multiple imaging elements; as well as The autofocus imaging circuit is configured to detect elements for each brightness change event: For the first adjusted focus, a first plurality of brightness change events are obtained from the brightness change event detection element; For the second adjustment focus, a second plurality of brightness change events are obtained from the brightness change event detection element; as well as The focus is determined based on the first plurality of brightness change events and the second plurality of brightness change events, and is used for automatic focusing of the scene. In response to autofocusing the scene, light is directed to the plurality of imaging elements to image the scene through the plurality of imaging elements, wherein the imaging device has a hybrid sensor, the brightness change event detection element and the plurality of imaging elements are arranged in the hybrid sensor in a predetermined pattern.

11. The autofocus imaging device of claim 10, wherein the autofocus imaging circuit is further configured to detect an element for each brightness change event: Determine the brightness change event density, which is used to determine the spatial brightness change event density of the plurality of brightness change event elements.

12. An autofocus imaging method, comprising: For the first adjusted focus, a first plurality of brightness change events are obtained from the brightness change event detection element; For the second adjustment focus, a second plurality of brightness change events are obtained from the brightness change event detection element; as well as The focus is determined based on the first plurality of brightness change events and the second plurality of brightness change events, and is used for automatic focusing of the scene. In response to autofocusing on the scene, light is directed to multiple imaging elements to image the scene, wherein the brightness change event detection element and the multiple imaging elements are arranged in a predetermined pattern in a hybrid sensor.

13. The autofocus imaging method according to claim 12, further comprising: The event density is determined based on the first plurality of brightness change events and the second plurality of brightness change events.

14. The autofocus imaging method according to claim 13, further comprising: Contrast is determined based on the established event density.

15. The autofocus imaging method according to claim 12, further comprising: Based on the identified focus tracking object.

16. The autofocus imaging method according to claim 12, wherein, The determination of the focus is further based on a neural network that processes the first plurality of brightness change events and the second plurality of brightness change events.

17. The autofocus imaging method according to claim 16, wherein, The neural network includes a spiking neural network.

18. The autofocus imaging method according to claim 12, further comprising: Based on the detection of reflections from the structured emission pattern, at least one of the first plurality of brightness change events and the second plurality of brightness change events is obtained.

19. The autofocus imaging method according to claim 12, wherein, The focus is determined based on a feedback loop of the first plurality of brightness change events and the second plurality of brightness change events.

Images