Electronic device and apparatus

Abstract

An electronic device comprising circuitry configured to determine a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and to perform focusing of an optical assembly based on the spot profile.

Description

[0001] ELECTRONIC DEVICE AND APPARATUS

[0002] TECHNICAL FIELD

[0003] The present disclosure generally pertains to the field of imaging devices, in particular to an electronic device, method or computer program that provides autofocusing for an imaging assembly and / or digital camera.

[0004] TECHNICAL BACKGROUND

[0005] Known devices for providing autofocusing for cameras rely on methods such as phase detection autofocus (PDAF) and contrast detection autofocus (CDAF). CDAF is based on capturing a small portion of the image and evaluating the contrast within that area. The lens position is then adjusted to maximize the contrast, higher contrast indicating better focus. This process is iteratively repeated until the desired focus is achieved. PDAF is an autofocusing technique based on measuring phase differences in light signals. Using the phase difference in the light signals, in PDAF, a lens position that is optimal for getting an in-focus image is determined.

[0006] There are also systems that rely on Time-of-Flight techniques to provide autofocus. Time of Flight (ToF) imaging is a technology used to capture depth. ToF imaging operates by measuring the time it takes for light to travel from the camera to the subject and back. A light signal is emitted and the time it takes to return is analyzed, yielding a depth map.

[0007] A first type of ToF-device are direct time-of-flight (dToF) devices in which typically SPADs (Single Photon Avalanche Diode) are used to detect light events and the time-of-arrival is stored in a histogram which is read out. Based on the speed of light, the distance to objects can be calculated.

[0008] A second type of ToF-device are indirect time-of-flight (iToF) devices, in which a phase shift of between light emission and light detection is used to obtain distance information. In iToF, the light detection is based on a demodulation signal typically applied to CAPD (Current Assisted Photonic Demodulator) pixels.

[0009] After determining the distance between the camera and the subject, autofocus techniques relying in ToF calculate a focus position of the camera based on the distance between the camera and the subject.

[0010] The present disclosure provides devices, methods and computer programs that provide autofocusing while improving known techniques. SUMMARY

[0011] According to a first aspect the disclosure provides an electronic device according to independent claim 1.

[0012] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0013] BRIEF DESCRIPTION OF THE FIGURES

[0014] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0015] Fig. 1 is a schematic illustration of a configuration and operation of a camera with autofocusing functionality that is based on a spot profile generated by a laser array; and

[0016] Fig. 2 shows, in a cross-sectional schematic view, an example of a self-registered and singleaperture RGB+SPAD sensor; and

[0017] Fig. 3 schematically shows an exploded view of a self-registered and single-aperture RGB+SPAD sensor; and

[0018] Fig. 4 is a schematic illustration of an alternative configuration and operation of a camera that provides autofocusing functionality that is based on a spot profile generated by a laser array; and

[0019] Fig. 5A shows a schematic illustration of a spot in a focused state and a spot in a defocused state; and

[0020] Fig. 5B is an illustration of a relation between the lens position and a spot profile; and

[0021] Fig. 6 is schematic illustration of a process for providing autofocusing of a camera based on a spot profile; and

[0022] Fig. 7 is a more detailed flowchart of a coarse adjustment step to determine the optimal lens position; and

[0023] Fig. 8 is a flowchart of a first determination step; and

[0024] Fig. 9 is a flowchart of a fine adjustment step; and

[0025] Fig. 10 is a flowchart of a second determination step; and

[0026] Fig. 11 is a schematic illustration of an alternative process for providing autofocusing of a camera based on a spot profile; and

[0027] Fig. 12 is a more detailed schematic illustration of a process for providing autofocusing of a camera based on a spot profile; and Fig. 13 schematically describes an embodiment of a device that implements the electronic device for focusing an optical assembly as described in the embodiments.

[0028] DETAILED DESCRIPTION OF EMBODIMENTS

[0029] Before a detailed description of the embodiments under reference of Fig. 1 is given, general explanations are made.

[0030] Some embodiments according to the present disclosure provide an electronic device comprising circuitry configured to determine a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and to perform focusing of an optical assembly based on the spot profile.

[0031] The electronic device may be a camera or a component of a camera.

[0032] The spots may for example be areas of illumination of limited size and / or area generated by radiation incident on a surface. The surface can be a photosensitive surface of a imaging sensors such as a charge coupled device (CCD), Complementary Metal-Oxide-Semiconductor (CMOS) sensor or a Time-of-Flight (ToF) sensor. The photosensitive surface of known imaging sensors comprises pixels. Pixels subtended by a spot on the surface may be called spot pixels. The surface can also be the surface of an object. The spots may be areas of illumination with a circular shape, though shapes other than circular shapes may be provided in accordance with the present disclosure.

[0033] Image data generated by known ToF image sensors comprises two distinct types of image data. A first type of image data generated by ToF image sensors, in particular image sensors working according to a indirect ToF (iToF)-principle, is phase image data, which can be called depth image data. Alternatively to phase image data, timing image data can be generated by image sensors working according to a direct ToF (dToF)-principle. Timing image data can likewise which can be called depth image data. Any of dToF or iToF-based sensors may be used according to the present disclosure. In the following, unless dToF or iToF are mentioned specifically, any mention of ToF should be understood to comprise either of dToF or iToF. Embodiments of the present disclosure that generate depth information based on spots may therefore be said to be based on a spot-ToF (sToF) principle. A second type of image data generated by ToF image sensors is intensity image data, which can be called confidence image data. The depth image data is known to allow calculation of the distance of an object whose image is contained in the depth image data to the ToF image sensor. Intensity image data is known to be based on an intensity of the incoming radiation. The ToF image sensor within the meaning of the present disclosure is configured to detect infrared radiation. The infrared radiation can be called IR radiation.

[0034] An IR image sensor can be a CCD sensor or a CMOS sensor configured to detect IR radiation.

[0035] The optical assembly may be an arrangement of lenses and / or mirrors intended to transmit and focus incoming radiation, specifically on the photosensitive surface of an image sensor such as, generally, a CCD device, a CMOS device, or specifically, an RGB-IR sensor, a ToF sensor or a RGB-ToF sensor. The optical assembly may be composed of a single lens.

[0036] Infrared radiation may be electromagnetic radiation or light in a part of the known electromagnetic spectrum usually classified as infrared. Infrared may thus be electromagnetic radiation with a wavelength in the range of 700 nm to 1 mm.

[0037] The focusing may be called autofocusing.

[0038] As the focusing of the embodiments relies on spot profile, the autofocusing of the embodiments provides efficient results even in lighting situations where PDAF -based or CDAF-based techniques are not able to confidently provide autofocusing. The autofocusing of the embodiments may in particular determine a focus position on low-contrast situations.

[0039] The spot profile can be a good focus measure in a defined system with known parameters, e.g., a pixel size, an illumination pattern, optics and distance Alternatively, the spot profile can be used as a guide for contrast detection autofocus.

[0040] By focusing the optical assembly based on intensity image data obtained from a ToF image sensor or based on intensity image data obtained from an IR image sensor, the focusing can rapidly be performed in low contrast environments.

[0041] According to some embodiments, focusing comprises determining an optimal configuration of the optical assembly.

[0042] A configuration of the optical assembly is a configuration of components of the optical assembly, such as lenses, with respect to each other and with respect to the image sensor, such as an RGB-ToF sensor or an RGB-IR sensor. The configuration may comprise a lens position.

[0043] The optimal configuration of the optical assembly is a configuration of the optical assembly that causes light transmitted through the optical assembly to be focused or sharply focused on the photosensitive surface of the ToF sensor or the IR image sensor.

[0044] According to some embodiments, focusing comprises determining an optimal lens position. The lens position may be a spatial distance of the lens from the image sensor or an angular orientation of a plane of the lens to a plane, such as the photosensitive surface, of the image sensor.

[0045] The optical assembly may be configured by moving components of the optical assembly to other positions with respect to each other, using a known electric or electronic actuation mechanism.

[0046] The optical assembly may also be configured by moving a lens using a known electric or electronic actuation mechanism.

[0047] According to some embodiments, the circuitry is configured to obtain image data from an image sensor representingvisible light after the focusing of the optical assembly.

[0048] Visible light is a part of the electromagnetic spectrum that comprises light in a wavelength that is accessible to human eyesight. The visible light in accordance with the present disclosure is therefore light in a range of wavelengths comprising wavelengths of 780 nm to 375 nm. Image data in visible light may be a digital photograph, video information or the like. Image data in visible light may be image data in RGB and / or grey scale. The image sensor used to obtain image data from visible light may, for example, be an RGB-ToF sensor or an RGB-IR sensor

[0049] According to some embodiments, the electronic device according to the present disclosure provides for the circuitry to be configured to determine a spot contrast from the spot profile.

[0050] The spot contrast can be determined from the spot profile by comparing a spot profile in a focused state with a spot profile in a defocused state. By determining the spot contrast from the spot profile, the spot contrast can be maximized.

[0051] According to some embodiments, determining the spot contrast is based on detecting a minimal spot size.

[0052] The spot profile can be a size of spots, the size of spots can be measured as a radius of the spots or as an area of the spots. The spot contrast, in some embodiments, can be maximal if the spot size is a minimal spot size. A minimal spot size can, for example, be detected by, using the intensity image data and / or the intensity image data, calculating the area on the photosensitive surface of the imaging sensor providing the intensity image data and / or the intensity image data being illuminated by the spots and testing whether this area is minimal.

[0053] The area covered by the spot may be minimal, in a further illustrative example hereinbelow, in a focused state as indicated in Fig. 5B. Detection of a minimal area may, for example, be accomplished by detecting whether any change of the lens position results in a increase of the size of the spot profile. If a change of the lens position in any direction causes a decrease of the area, the area of the spot is not minimal.

[0054] Alternatively, for example, when the spot size is minimal, an intensity of the spot may be constant or substantially constant at any position of the area covered by the spot. In other words, if the area covered by the spot is not minimal, the intensity of the spot may be of a lower intensity close to an edge of the area covered by the spot compared to the intensity close to a center of the area covered by the spot.

[0055] According to some embodiments, the focusing comprises performing a CDAF algorithm based on the intensity image data.

[0056] CDAF algorithms form part of a known class of autofocus algorithms. Known algorithms use radiation in the visible to perform autofocusing. Performing autofocus in the visible can lead to difficulty in determining the optimal lens position. CDAF algorithms, in accordance with the present disclosure, generally comprise moving the lens to the optimal lens position such that, with the lens in the optimal lens position, the spot contrast is maximal. Using the intensity image data and / or the intensity image data in determining the spot profile can improve CDAF algorithms in conditions where known CDAF algorithms yield other than optimal results.

[0057] In some embodiments, the electronic device further comprises a laser emission assembly providing a spot pattern.

[0058] Laser, in accordance with the present disclosure, is electromagnetic radiation emitted coherently by a laser emission assembly. The laser emission assembly is any electronic device capable of emitting laser radiation. The laser emission assembly may be any one of a number of known laser emission assemblies, including, but not limited to: a laser diode, a gas laser, a fiber laser, a solid state laser. The laser emission assembly may be configured to provide a single laser beam or a plurality of laser beams. The laser emission assembly may be configured to provide a laser beam with an IR wavelength. The laser beam may therefore be called an IR laser beam. A spot pattern comprises one or more spots. The spots are generated by the laser beam emitted by the laser emission assembly illuminating a surface, forming spots in the illuminated areas. The radiation reflected from the surface may be detected by the ToF sensor or the IR sensor generating intensity image data. The laser emission assembly may be a laser array. The laser beam can be an IR laser beam. Using the electronic device according to the present disclosure, a fast and accurate autofocusing system using active light spots as a guide can be provided. Active light spots can be spots detected by the ToF sensor or the IR image sensor.

[0059] In some embodiments, the laser emission assembly is a vertical cavity surface-emitting laser (VCSEL) array.

[0060] A VCSEL array is an array of VCSEL emitters. Known VCSEL arrays can be used to provide the spot pattern. The VCSEL array can provide one or more laser beams and the spot pattern comprising one or more spots. The VCSEL array can provide the laser beam in the IR wavelengths.

[0061] In some embodiments, the spot pattern comprises a plurality of spots. The spot pattern comprising a plurality of spots may be generated by a VCSEL array.

[0062] According to some embodiments, the circuitry is configured to determine an initial configuration of the optical assembly based on ToF depth image data obtained from the ToF image sensor.

[0063] The initial configuration of the optical assembly is a configuration of the optical assembly that is not the optimal configuration of the optical assembly, but the initial configuration of the optical assembly is a configuration of the optical assembly similar to the optimal lens position. The initial configuration of the optical assembly is determined by calculating, using known relations, the configuration of the optical assembly, based on the ToF depth image data. The initial configuration of the optical assembly may be an initial lens position.

[0064] According to some embodiments, the circuitry is configured to determine a focus distance based on the depth image data, and to determine the initial configuration of the optical assembly based on the focus distance.

[0065] The focus distance may be a distance of a ToF sensor or an RGB-ToF sensor to a spot or a plurality of spots. The initial configuration of the optical assembly may be determined based on a calculation of a spot located at the focus distance to be in focus. The initial configuration of the optical assembly may be calculated as the initial lens position based on known principles of optics. Alternatively, the initial configuration of the optical assembly may be calculated according to specific parameters of the optical assembly.

[0066] According to some embodiments, the circuitry is configured to perform a fine tuning of the configuration of the optical assembly based on the spot profile.

[0067] The fine tuning can comprise capturing, with the optical assembly in a current configuration, intensity image data and / or intensity image data using the ToF sensor and / or the IR sensor. Based on the intensity image data and / or intensity image data, the spot contrast can be determined. If the spot contrast is not maximal, the optical assembly is configured so that its configuration is more similar to the optimal configuration of the optical assembly.

[0068] According to some embodiments, the circuitry is configured to perform a fine tuning of the configuration of the optical assembly by configuring the configuration of the optical assembly such that the spot contrast is maximal.

[0069] According to some embodiments, the circuitry is configured to perform the fine tuning by iteratively capturing intensity image data and / or IR image data and configuring the optical assembly until the optimal configuration of the optical assembly is determined.

[0070] The configuring can comprise calculating a direction of configuration in a parameter space parameterizing the configuration that points towards the optimal lens position. The direction pointing to the optimal configuration of the optical assembly can be calculated by determining a gradient of the spot profile with respect to the position of the optical assembly in the parameter space. The configuring can comprise moving a lens in a direction toward the optimal lens position. The optimal lens position can be a position in which the gradient of the spot profile is close to zero or zero, as dictated by known mathematical methods and / or numerical methods of finding extreme values of functions. It should be noted that the intensity image data and / or the intensity image data used in determining the optimal configuration of the optical assembly is required to be provided by an image sensor, which can be the ToF image sensor or the IR image sensor, that has received the light through the optical assembly.

[0071] In some embodiments, determining the initial configuration of the optical assembly precedes the fine tuning.

[0072] By determining the initial configuration of the optical assembly before the fine tuning, the optical assembly can be configured to the initial configuration of the. Since initial configuration of the optical assembly is similar to the optimal configuration of the optical assembly, the fine tuning is augmented. The fine tuning thus can be rendered more efficient.

[0073] According to some embodiments, the circuitry is configured to perform a direct estimation of the optimal configuration of the optical assembly based on the spot profile.

[0074] The direct estimation may be performed in situations exhibiting a low or medium blurriness of the spot profile.

[0075] According to some embodiments, the circuitry is configured to construct a relation between a configuration of the optical assembly and the spot profile. A relation between the configuration of the optical assembly, such as the lens position, and the spot profile may be constructed. The construction of the relation between the configuration of the optical assembly and the spot profile can be based on the surface parameters of the surface reflecting the spot profile. The surface parameters are based on the parameters of different surfaces. For example, according to the surface, appropriate reflective parameters and appropriate reflective parameters can be used to construct the relation between the configuration of the optical assembly and the spot profile. Different amounts of scattering may be taken into account. Scattering and defocusing may be taken into account.

[0076] The relation may be constructed in situations exhibiting the low or medium blurriness of the spot profile.

[0077] Constructing the relation between lens position and spot profile may comprise multiple acquisition depth image data and / or intensity image data associated with multiple lens positions. According to some embodiments, the circuitry is configured to determine an optimal configuration of the optical assembly based on the relation between the configuration of the optical assembly and the spot profile.

[0078] The optimal lens position can be determined by calculating the configuration of the optical assembly where the relation between the configuration of the optical assembly and the spot profile is minimal.

[0079] In some embodiments, the ToF image sensor is part of a coaxial RGB-ToF image sensor configured to obtain the intensity image data and / or depth image data.

[0080] The RGB-component and the ToF-component of the coaxial RGB-ToF image sensor are not required to be congruent in shape and size. Instead, the ToF-component may comprise a smaller size than the size of the RGB-component or vice-versa. Additionally, the ToF-component may be located close to an axis of the RGB-component coaxial RGB-ToF image sensor or vice-versa. The electronic device can be configured such that the coaxial RGB-ToF image sensor is illuminated by light that has first passed through the lens or the optical assembly, comprising the lens. The coaxial RGB-ToF image sensor can be configured as described with respect to Fig. 2 and Fig. 3. The coaxial RGB-ToF image sensor can be configured to detect IR radiation and visible light. The coaxial RGB-ToF image sensor can provide the intensity image data and / or the intensity image data and the depth image data. The coaxial RGB-ToF image sensor can further provide the RGB image data and / or the grey scale image data.

[0081] In some embodiments, the IR image sensor is part of an RGB-IR sensor configured to obtain the intensity image data. The electronic device can be configured such that the RGB-IR sensor is illuminated by light that has first passed through the lens or the optical assembly, comprising the lens. The electronic device can further be configured to comprise, in addition to the RGB-IR image sensor, a ToF image sensor. The RGB-IR image sensor can be configured to detect IR radiation and visible light. The ToF image sensor can be configured to detect IR radiation. The coaxial RGB-IR image sensor can provide the intensity image data and the RGB image data and / or the grey scale image data.

[0082] Some embodiments according to the present disclosure provide an imaging apparatus comprising the electronic device according to the present disclosure.

[0083] The imaging apparatus can be a camera, an imaging unit to be included in another electronic device, a video camera, a handheld electronic device such as a mobile phone, a smartphone, a tablet computer, a personal computer, a webcam, a PTZ pan-tilt-zoom camera, a wearable electronic device, such as augmented reality glasses, an augmented reality device, extended reality glasses, an extended reality device, virtual reality glasses, a virtual reality headset. Other devices may, in accordance with the present disclosure, be the imaging apparatus.

[0084] The imaging apparatus comprising the electronic device may provide fast and reliable autofocus in scenes that are challenging for PDAF are difficult for CDAF because of low contrast, scene darkness and low-textured or non-textured surfaces.

[0085] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer- readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0086] Fig. 1 is a schematic illustration of a configuration and operation of a camera with autofocusing functionality that is based on a spot profile generated by a laser array. The camera comprises a laser array 40 (e.g. a VCSEL array), a lens 50 and a coaxial RGB-ToF sensor 61. The laser array 40 may for example be arranged in close proximity to the coaxial RGB-ToF sensor 61. The RGB-ToF sensor 61 may for example be a self-registered and single-aperture RGB+ToF sensor of an all-in-one sensor module as described in more detail in Figs. 2 and 3. The RGB component of the RGB-ToF sensor 61 is provided to capture RGB images in visible light. The ToF component of the RGB-ToF sensor 61 is provided to assist in the autofocusing process for capturing RGB images and operates based on known principles of Time of Flight (ToF) by providing depth information (e g. phase information or time-of-arrival information) and / or intensity information (e.g. an intensity image).

[0087] The laser array 40 emits a plurality of IR laser beams 41 at an infrared wavelength. The IR laser beams 41, if intersecting a surface of an object E, generate a spot pattern comprising a plurality of spots. Reflected IR light 42 is received by the ToF component of the coaxial RGB-ToF sensor 61, providing depth information and / or an intensity image of the spot pattern. Based on the depth information and / or a spot profile of one or more spots in the spot pattern (see Figs. 5A, B for more details), an optimal lens position is determined such that the reflected light 42 is optimally focused on a photosensitive surface of the coaxial RGB-ToF sensor 61. Reflected light may be called optimally focused if the spot pattern is in the sharpest possible focus on the photosensitive surface of the coaxial RGB-ToF sensor 61 or if a contrast of the spot pattern is as sharp as possible. The lens 50 is then positioned in the optimal lens position. With the lens 50 in the optimal lens position, an image of the object E taken in RGB light by the coaxial RGB-ToF sensor 61 is likewise in focus or at least in better focus.

[0088] The lens 50 is an example of an optical assembly. The lens position in an example of a configuration of the optical assembly. The optimal lens position is an example of an optimal configuration of the optical assembly.

[0089] In the example of Fig. 1, the RGB-ToF sensor 61 comprises a RGB component. In alternative embodiments, the RGB component may be any component that is capable of generating image data, for example in grey scale and / or RGB color scale from incoming visible light.

[0090] The lens 50 of the camera of Fig. 1 may be an optical assembly comprising one or more lenses or other optical elements such as a mirror configured to focus incident radiation on the coaxial RGB-ToF sensor 61. That is, the lens 50 may comprise one or more refractive elements or reflective elements.

[0091] In order to provide focusing of the imaging assembly, the lens 50 can be positioned in different arrangements, for example at different distances, with respect to the coaxial RGB-ToF sensor 61. The positioning of the lens 50 can, for example, comprise positioning at least one lens in relation to at least one other lens and / or in relation to the coaxial RGB-ToF sensor 61.

[0092] To be positioned, the lens 50 may for example be provided with an appropriate adjustment mechanism (not pictured in Fig. 1 above). Positioning of the lens 50 may for example comprise moving the lens 50 axially with respect to the imaging sensor array 61 in a range of distances from the coaxial RGB-ToF sensor 61. The adjustment mechanism may be an automatic mechanism able to move the lens with respect to the image sensor assembly. The adjustment mechanism may be configured to process electronic signals containing positioning information provided by the circuitry according to the present disclosure and position the lens according to the positioning information.

[0093] The laser array 40 of Fig. 1 may be any laser emission assembly that is configured to emit IR laser beams 41 in an infrared wavelength. The infrared wavelength of the IR laser beam 41 emitted by the laser array 40 may be chosen such that the reflected IR light 42 is detectable by the ToF component of coaxial RGB-ToF sensor 61.

[0094] In the example of Fig. 1 above, the laser emission assembly 40 is arranged codirectionally with respect to the coaxial RGB-ToF sensor 61. A codirection arrangement is, for example, an arrangement or orientation of the array 40 with respect to the coaxial RGB-ToF sensor 61 such that a direction of the IR laser beams 41 emitted by the laser array 40 are parallel or largely parallel to an axis of the coaxial RGB-ToF sensor 61. The axis of the coaxial RGB-ToF sensor 61 is, for example, an axis that is perpendicular to a front facing surface of the coaxial RGB-ToF sensor 61.

[0095] ToF Depth Image

[0096] It has been described above with reference to Fig. 1 that the autofocusing process is based on depth information (e.g. phase information or timing information) and / or intensity information (e g. a intensity image). Depth information may be provided using known Time-of-Flight techniques.

[0097] In sensors working according to the iToF-principle, depth information is obtained by determining a phase angle c|) of incident light in pixels of the ToF sensor as compared to a modulation signal. Each pixel of a ToF sensor may typically include two distinct taps (e g. tap A and tap B), each tap providing gain information (e.g. GA for tap A) in a it!2 phase offset as well as phase-dependent intensity information (e.g. S(0) for the a phase of zero and S(TI / 2) for a phase of 7t / 2). Using the gain information and the phase-dependent intensity, variables I and Q can be calculated using the relations

[0098] I = 2(GA- GA) (0) (1) and

[0099] Q = 2(GA- GA)S(tt / 2). (2)

[0100] The phase angle for each pixel comprising the ToF sensor can then be calculated using

[0101] 4> = — atan2 Q, I). (3) Executing this calculation for each pixel comprising the ToF sensor yields a phase image associating each pixel with phase information.

[0102] Depth information is calculated from the phase information by correlating the phase information with phase information of emitted laser beams, such as the IR laser beams 41 generating the spot pattern. The correlation of the phase information detected by the pixel with phase information of emitted IR laser beams 41 yields a phase offset A0 between the emitted laser beams and the incident radiation sensed by the pixel. A distance D of the pixel to the surface reflecting the emitted laser beams can then be calculated using where c is the speed of light in an atmosphere and f is a frequency of the emitted light.

[0103] For example, if the emitted light has a frequency of / =20 MHz and the calculated phase offset A0 is 22°, the distance to the surface is about 0.45m. The distance is depth information.

[0104] Thus, by correlating the phase angle of the incident radiation with the phase angle of the spot pattern, the distance of the pixel to the spot can be calculated

[0105] In sensors working according to the dToF-principle, a photon counting (PC) technique pulse method is used.

[0106] Photon counting ToF systems like the dToF system record a photon histogram. DToF systems may, for example, use single photon avalanche diodes (SPADs) as detectors. In dToF, depth information is obtained, for example, by measuring the time a signal pulse of a defined duration, emitted by the device using dToF, takes to reach an object, be reflected on a reflection surface, and return to the device to be sensed by the dToF sensor. The signals are then correlated to obtain the distance.

[0107] A pixel of a dToF-type sensor may comprise two distinct switches SA and SB and two memory elements MA and MB. The switches SA and SB rout an incident signal, which is the reflection of the emitted signal, received by the pixel to the corresponding memory element MA and MB. The switches are controlled by a control signal cadenced to coincide with the emitted signal pulse. During exposure, the incident signal is first routed to the first memory element, for example MA After a time of the length of the emitted signal pulse has elapsed, the incident signal is routed to the second first memory element, for example MB. The ratio of a number of photons of the incident signal routed to either MA or MB is related to the length of time it took the signal to travel to the reflection surface and back. The number of photons define an intensity, with, for example, IA or IB corresponding to the signals routed to either MA or MB.

[0108] The distance of the reflection surface can then be calculated where c is the speed of light in an atmosphere and to is the duration of the emitted pulse.

[0109] For example, if the emitted pulse has a duration of 25 ns and the ratio of intensities is IA / IB = 2 / 3, then the distance to the surface is about 3.75 m. The distance is depth information.

[0110] Calculating a distance to an object requires a choice of which spots detected by the ToF sensor to use to calculate the depth information. A description of the choice of which spots detected by the ToF sensor to use to calculate the depth information is presented hereinbelow.

[0111] ToF Intensity Image

[0112] It has been described above with reference to Fig. 1 that the autofocusing process may also be based on intensity information (e.g. an intensity image).

[0113] Intensity information (which in the context of ToF methods is also called confidence information) may for example be calculated from the variables I and Q defined in equations (1) and (2) using the relation

[0114] C = Vl2+Q2, (6)

[0115] Confidence, or intensity, C is a measure of the amplitude of a signal received by a ToF sensor on, for example, taps A and B as mentioned above.

[0116] In dToF, intensity image data for a single pixel may, for example, be calculated based on the sum of photons, or the intensity, stored by each memory element.

[0117] C = IA+ IB (7)

[0118] All-in-one RGB / ToF sensor module

[0119] Fig. 2 shows, in a cross-sectional schematic view, an example of a self-registered and singleaperture RGB+SPAD sensor (110 in Fig. 2) of an all-in-one sensor module that may act as the coaxial RGB-ToF sensor 61 described in Fig. 1 above. The description of an RGB+SPAD sensor 110 is only one possible sensor that may act as the coaxial RGB-ToF sensor 61 and should be understood as non-limiting. Use of the RGB+SPAD sensor may be considered if the depth image data is to be generated based on timing image data, as in the dToF. method. However, any ToF-component of the coaxial RGB-ToF sensor 61 capable of generating depth image data can be considered according to the present disclosure.

[0120] For example, a device as described in patent document WO2021117642, specifically as shown in Fig. 12a therein, may be provided. Alternatively, for example, a single-chip RGB-ToF camera, as described in non-patent literature SHINMURA et al. Estimation of Human Orientation using Coaxial RGB-Depth Images. In: Proceedings of the 10th International Conference on Computer Vision Theory and Applications (VISAPP-2015), pages 113-120, ISBN: 978-989-758-090-1, especially section 2, may be provided.

[0121] The sensor 110 may be configured according to a triple wafer stacking technique with dual sensing layer 400. A dual band pass filter (DBPF) 401 for wavelengths of interest selection is arranged above the dual sensing layer 400. The dual sensing layer 400 comprises a thin RGB sensitive layer 404 (photodiode layer, PD) with no or little NIR / SWIR absorption. This RGB sensitive layer 404 may for example comprise Silicon photodiodes or organic photodiodes. The dual sensing layer 400 further comprises a visible-cut filter 405 for residual visible light removal. The dual sensing layer 400 further comprises a Silicon (NIR) or Germanium (SWIR) -based SPAD layer 406 for indirect Time of Flight (iToF), direct Time of Flight (dToF) and / or Photon Counting (PC). The SPAD layer 406 may for example comprise Germanium on Silicon or Germanium-based Single-photon avalanche diodes. A readout layer 407 in CMOS technology is arranged below the dual sensing layer 400. The readout layer 407 may for example be realized as a CMOS readout and processing layer. The connectivity for the SPADs may for example be realized as a high density wafer-to-wafer connectivity (e.g. by hybrid bonding).

[0122] The self-registered and single-aperture RGB+SPAD sensor 110 is configured to sense RGB radiation 402, which can be visible light 402, as well as IR radiation 403, which can for example, in embodiments corresponding to Fig. 1, be the reflected IR light 42 or the infrared light 42.

[0123] In the self-registered and single-aperture RGB+SPAD sensor of Fig. 2, Event-based Vision Sensor (EVS) functionality may be added at RGB wafer level or into readout wafer level.

[0124] Fig. 3 schematically shows an exploded view of a self-registered and single-aperture RGB+SPAD sensor. An RGB sensitive layer 404 comprising red, green and blue pixels is stacked above a SPAD layer 406. A readout layer 407 is arranged below the RGB sensitive layer 404 and the SPAD layer 406.

[0125] The RGB+SPAD sensor may provide the advantage that RGB, EVS and SPAD data / images are self-registered and consistent. That is, a 1-to-l spatial and temporal correspondence in the image of intensity, color, and distance data may be possible. Still further, the image data may also be temporarily registered. For example, the sensor module may provide full parallel simultaneous acquisition of both RGB / EVS and SPAD imaging.

[0126] Fig. 4 is a schematic illustration of an alternative configuration and operation of a camera that provides autofocusing functionality that is based on a spot profile generated by a laser array.

[0127] In the alternative configuration, the camera does not comprise the coaxial RGB-ToF sensor (61 in Fig. 1), but instead comprises a combined RGB-IR sensor and a ToF sensor. Moreover, the ToF sensor is configured to not receive IR light through the lens 50. Note that the RGB-IR sensor is provided in this embodiment as any intensity image data provided by the ToF sensor is independent of the lens 50 and can therefore not assist in an autofocus process based on the spot profile.

[0128] The camera comprises a laser array 40 (e.g. a VCSEL array), a lens 50, a combined RGB-IR sensor 62 and a ToF sensor 63. The laser emission assembly 40 may for example be arranged in close proximity to the combined RGB-IR sensor 62 and the ToF sensor 63. The ToF sensor 63 is provided to capture depth image data and may be any known ToF sensor capable of providing depth image data. The RGB-IR sensor 62 is provided to capture RGB images in visible light and IR image data in IR light. The IR image data is intensity image data. The IR image data captured by the RGB-IR sensor 62 assists in the autofocusing process for capturing RGB images. The ToF sensor 63 is likewise provided to assist in the autofocusing process for capturing RGB images and operates based on known principles of Time of Flight (ToF) by providing depth information (e g. phase information).

[0129] The laser array 40 emits a plurality of IR laser beams 41 at an infrared wavelength. The IR laser beams 41, if intersecting a surface of an object E, generate a spot pattern comprising a plurality of spots. Reflected IR light 42 is received by the ToF sensor 63, providing depth information. Reflected IR light 42 is likewise received by the RGB-IR sensor 62, providing an intensity image of the spot pattern. Based on the depth information and / or a spot profile of one or more spots in the spot pattern (see Figs. 5A, B for more details), an optimal lens position of the lens 50 is determined such that the reflected light 42 is optimally focused on a photosensitive surface of the coaxial RGB-IR sensor 62. The lens 50 is then positioned in the optimal lens position. With the lens 50 in the optimal lens position, an image of the object E taken in RGB light by the coaxial RGB-IR sensor 62 is likewise in focus.

[0130] In the example of Fig. 4, the RGB-IR sensor 62 comprises a RGB component. In alternative embodiments, the RGB component may be any component that is capable of generating image data, for example in grey scale and / or RGB color scale from incoming visible light. The combined RGB-IR sensor of Fig. 4 may for example be implemented as a Charge-Coupled- Device (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) sensor. Sensitivity to both visible light in the RGB color scale and IR light can be provided by including a suitable Bayer-filter arrangement that is selectively transparent to red, green, blue and infrared components of incident radiation and is configured to transmit red, green, blue and infrared components to an underlying pixel array.

[0131] The laser array 40 of Fig. 4 may be any laser emission assembly that is configured to emit IR laser beams 41 in an infrared wavelength. The infrared wavelength of the IR laser beam 41 emitted by the laser emission assembly 40 may be chosen such that the reflected IR light 42 is detectable by the IR-component of the RGB-IR sensor 62.

[0132] In the example of Fig. 4 above, the laser emission assembly 40 is arranged codirectionally with respect to the RGB-IR sensor 62 and the ToF sensor 63. A codirectional arrangement is, for example, an arrangement or orientation of the laser emission assembly 40 with respect to the RGB-IR sensor 62 and the ToF sensor 63 such that a direction of the IR laser beams 41 emitted by the laser emission assembly 40 are parallel or largely parallel to axes of the RGB-IR sensor 62 and the ToF sensor 63. The axes of the RGB-IR sensor 62 and the ToF sensor 63 are, for example, axes that are perpendicular to a front facing surfaces of the RGB-IR sensor 62 and the ToF sensor 63.

[0133] Fig. 5 A shows a schematic illustration of a spot DI in a focused state Fl and a spot D2 in a defocused state F2. The size of the spot D2 in a focused state Fl is indicated by the dashed circle. The size of the spot D2 in a focused state Fl is the same as the size of the spot DI in a focused state. The sizes of spots may, according to the present disclosure, be measured on the photosensitive surface of the coaxial RGB-ToF sensor 61 or the RGB-IR sensor 62 to provide an intensity image of the spot pattern. The sizes of spots are an example of the spot profile. Note that according to the present disclosure, single spots may be used to determine the spot profile, but a plurality of spots may be used as well. For example, an intensity image of a plurality of spots may be sensed, in the embodiment shown in Fig. 1 above, by the ToF-component of the coaxial RGB-ToF sensor 61 or, in the embodiment shown in Fig. 4 above, the IR-component of the RGB-IR sensor 62. The plurality of spots is not required to have equal or similar sizes either in the focused state Fl or in the defocused state F2. Autofocusing is provided by determining the spot profile from an intensity image.

[0134] Fig. 5B is an illustration of the relation R between the lens position and the spot profile. The optimal position is a position of the lens 50 wherein the spot profile is minimal. Positions of the lens 50 that are not the optimal position are positions wherein the spot DI, D2 is in a defocused state F2. If a plurality of spots is used to determine the spot profile, as described hereinabove, then the focused state Fl is a state wherein the spot profile is minimal. Moreover, if a plurality of spots is used to determine the spot profile, as described hereinabove, then any state that is not the focused state Fl is the defocused state F2. Autofocusing is provided according to one embodiment, by iteratively determining the optimal lens position, as described in detail with reference to Figs. 8 and 9 below, or by constructing the relation R and then directly moving the lens to the optimal lens position, as described with reference to Figs. 10 and 11 below.

[0135] Fig. 6 is schematic illustration of a process for providing autofocusing of a camera based on a spot profile. The autofocusing uses a coarse adjustment S2 followed by a fine adjustment S3. In the coarse adjustment S2, an object distance is determined based on depth image data as provided by the ToF component of coaxial RGB-ToF sensor (61 in Fig. 1) or ToF sensor (63 in Fig. 4) of a camera such as described in Fig. 1 or 4 above. Based on the object distance an initial lens position is calculated. The lens 50 is then positioned in the initial lens position. Once the lens 50 is in the initial lens position, the coarse adjustment S2 is concluded, and followed by the fine adjustment S3. Starting in the initial lens position, during fine adjustment, the optimal lens position of the lens 50 is determined in an iterative process as described in more detail in Fig. 7.

[0136] Fig. 7 is a more detailed flowchart of the coarse adjustment step S2 to determine the optimal lens position according to Fig. 6. The process comprises an acquisition step S21, a focus distance determination step S22, a calculation step S23 and a movement step S24.

[0137] In the acquisition step S21, depth image data is acquired by the ToF component of the coaxial RGB-ToF sensor (61 in Fig. 1) or the ToF sensor (63 in Fig. 4) of a camera such as described in Fig. 1 or 4 above. In the focus distance determination step S22, a focus distance is determined. The focus distance is a distance of a spot pattern from the camera. In the calculation step S23, the initial lens position is calculated. In the movement step S24, the lens is moved to the initial lens position. The focus distance determination step is discussed in greater detail in Fig. 8.

[0138] The initial lens position can be calculated in the calculation step S23 based on the focus distance according to known optical relations. The initial lens position can, for example, be calculated according to known principles of optics, for example using the known equation (6)

[0139] Wherein L is the focal length of the lens 50, u is the focus distance and v is the distance of the lens 50 to the photosensitive surface of the coaxial RGB-ToF sensor 61 or the ToF sensor 63. The lens position can, for example, be based on the quantity v. The parameter L is a known quantity related to the structure and type of the lens 50. The parameter u can be determined based on the focus distance. The equation (6) is, in this example, solved for the parameter v.

[0140] The initial lens position may have the same numerical value as the optimal lens position, but this circumstance is likely accidental. More likely, the initial lens position is a position close, but not identical in value, to the optimal lens position. In most cases, a fine adjustment S3 is necessary to move the lens 50 to the optimal lens position starting from the initial lens position.

[0141] In the movement step S24 the lens 50 is moved to the initial lens position. After the movement step S24, the coarse adjustment S2, in some embodiments, is concluded, to be followed, in some embodiments, by the fine adjustment S3.

[0142] Fig. 8 is a flowchart of the determination step (S22 in Fig. 7). In a detection step S220, spots in the depth image data are detected, in a selection step S221, spots detected in the depth image data are selected, in a determination step S222, respective focus distances of the selected spots are determined and in a profile determination step S223, the spot profile is determined based on the respective focus distances of the selected spots.

[0143] The detection step may comprise transforming the depth image data from a pixel domain into a spot domain. In the pixel domain, a spot is defined by a spot pixel region which includes a plurality of pixels related to the spot. In the spot domain, a set of sparse IQ values or intensity values describes a respective spot each.

[0144] Spots may, for example, be detected in S330 by detecting local intensity maxima in the intensity image data, with each local maximum indicating a location of a spot. Detection of local maxima may comprise detecting a local maximum by searching a 7px x 7px kernel, i.e. a region of the photosensitive surface of an RGB-ToF sensor or a ToF sensor or a RGB-IR sensor comprising 7px x 7px, wherein such local maximum search scans the raw intensity image data while iteratively shifting the kernel by 1 pixel until the entire image area has been scanned.

[0145] If the ToF-component is intended to work according to the iToF principle, IQ values of a pixel of the raw image (dense data) may be displayed in a coordinate system having the variable I (cf. equation 1) on the horizontal axis and the variable Q (cf. equation 2) on the vertical axis.

[0146] In order to detect the spots, alternatively, background spot detection filtering may be performed. In background spot detection filtering, prefixed spot positions are input from a database so that filtering is performed on dense IQ values based on the prefixed spot positions. In this manner, a mask image is created to limit the local-maxima search. The pixels, i.e. spots, that are not found by background spot detection filtering may be searched and found by spot noise reduction filtering, and in particular by spot finding, to obtain the sparse IQ data.

[0147] The spots that are not found by background spot detection filtering may be searched and found by spot noise reduction filtering, and in particular by spot finding, to obtain the sparse IQ data.

[0148] For the spot noise reduction filtering, for example DGS filtering may be used. DGS filtering can be performed based on spot finding, wherein each spot position obtained by the spot finding represents the center of a spot, i.e. the center of a spot pixel region including a plurality of pixels.

[0149] Each pixel value of a spot pixel region, i.e. the pixel region subtended by a spot, has a different phase, which is given by the angle of the arrow from the origin of coordinates to the respective IQ value, even though they belong to the same spot pixel region. A pixel that is in the vicinity of the spot pixel region but still outside of the spot pixel region, that is a valley pixel region value, is displayed in the coordinate system.

[0150] The phase of the spot pixel region IQ values may be corrected (or accuracy may be improved) by subtracting the valley pixel region IQ value from the spot pixel region IQ values, thereby corrected spot pixel region IQ values are obtained. The pixels inside the corrected spot pixel region 602 may then have the same phase. Because the spot peak pixel is included in the spot pixel region, by applying the DGS to I and Q values of each spot pixel region corrected I and Q value for the spot peak pixel of each spot is also obtained.

[0151] In some embodiments, only a single spot may be provided. If only a single spot is provided, the single spot is selected. If multiple spots are included in the depth image data, a single spot may still be selected. For example, a spot located closest to a geometric center of the photosensitive surface of the coaxial RGB-ToF sensor 61 or the ToF sensor 63 is selected.

[0152] Alternatively, a plurality of spots may be selected. For example, all spots located within a radius surrounding the geometric center of the photosensitive surface of the coaxial RGB-ToF sensor 61 or the ToF sensor 63 is selected. Spots may, for example, be detected by detecting intensity maxima and intensity minima in intensity image data corresponding to the depth image data. Selection of the spots may be based on other criteria as well, as long as one or more specific spots are selected. Spots may optionally be selected based on whether they were generated by laser beams emitted by the laser array intersecting a surface of a specific object. Spots may also be selected based on an algorithm or a selection of a machine learning algorithm or deep neural network. Each spot may encompass multiple pixels of the photosensitive surface of the coaxial RGB-ToF sensor 61 or the ToF sensor 63. The depth information of each spot detected by each pixel describes the distance of that pixel to the spot. A focus distance of each individual selected spot may be calculated based on an average of the distances of the pixels detecting that spot.

[0153] The focus distance may be calculated based on the average distance of each selected spot.

[0154] Returning to Fig. 8, based on the focus distance, the initial lens position is calculated. Subsequently the lens is moved directly to the initial lens position. By moving the lens directly to the initial lens position, the fine adjustment step can be executed with minimal delay.

[0155] Fig. 9 is a flowchart describing the fine adjustment step (S3 in Fig. 6 above). The fine adjustment S3 comprises a plurality of steps with the goal of iteratively moving the lens 50 to the optimal lens position. The fine adjustment requires the lens to be in the initial lens position. In an intensity image acquisition step S32, intensity image data of the spot pattern is generated by the ToF component of the coaxial RGB-ToF sensor 61 or the RGB-IR sensor 62 with the lens in the current lens position. Note that, in the first iteration, the current lens position is the initial lens position. Based on the intensity image of the spot pattern, the spot profile is determined in a determination step S33. In a confirmation step S34, the determined spot profile is confirmed to be minimal - “yes” - or not minimal - “no”. Note that, if a current iteration is a first iteration, the outcome of the confirmation step S34 can always be “no” in order for at least one iteration to be iterated. The determination step S33 is discussed in greater detail in Fig. 10.

[0156] If the spot profile is confirmed to be not minimal, a movement step S35 follows the confirmation step S34. In the movement step S35, the lens 50 moved, by a small amount, in a direction that the optimal lens position is expected to be located. The direction can be calculated by determining a gradient of the spot profile with respect to the lens position. The direction can be a direction in which the gradient is negative, such that, with each movement, the spot profile decreases. The optimal lens position can be a position in which the gradient of the spot profile is close to zero or zero, as dictated by known mathematical methods of finding extreme values of functions.

[0157] Following the movement step S35, the acquisition step S32 is repeated in a next iteration. Iterations repeat until the lens 50 has been moved to the optimal lens position.

[0158] Note that the fine adjustment step S3 can also comprise a data storage step (not shown) wherein the intensity image data or the lens profile of past iterations is stored. Accordingly, the confirmation step S34 can also be based on the intensity image data or the spot profile of a current iteration and the intensity image data or the spot profile of the past iteration. The iteration comprising the steps S32, S33, S34 and S35 is repeated until the spot profile is confirmed to be minimal.

[0159] Note that some embodiments may provide for the iteration comprising the steps S32, S33, S34 and S35 to be terminated once a maximum number of iterations, set at production of the camera have been performed or once a maximum time, set at production of the camera, has elapsed or once another criterion has been fulfilled.

[0160] If the spot profile is confirmed to be minimal in the confirmation step S34, with the spot profile being minimal, the lens is in the optimal lens position 30. With the optimal lens position having been determined, the fined adjustment step S3 is concluded. The fine adjustment S3 concludes the autofocus procedure. Following the autofocus procedure, RGB image data using the RGB- component of the coaxial RGB-ToF sensor 61 or the RGB-component of the RGB-IR sensor 62 may be acquired.

[0161] The fine adjustment S3 can be called a CDAF procedure or a CDAF algorithm.

[0162] Fig. 10 is a flowchart of the determination step (S33 in Fig. 9). In a detection step S330, spots in the intensity image data are detected, in a selection step S331, spots detected in the intensity image data are selected, in a determination step S332, respective spot profiles of the selected spots are determined and in a profile determination step S333, the spot profile is determined based on the respective spot profiles of the selected spots.

[0163] Spots may, for example, be detected in S330 by detecting local intensity maxima in the intensity image data, with each local maximum indicating a location of a spot. Detection of local maxima may comprise detecting a local maximum by searching a 7px x 7px kernel, i.e. a region of the photosensitive surface of an RGB-ToF sensor or a ToF sensor or a RGB-IR sensor comprising 7px x 7px, wherein such local maximum search scans the raw intensity image data while iteratively shifting the kernel by 1 pixel until the entire image area has been scanned.

[0164] If CDAF is used, only a region of interest (Rol) needs to be specified, the Rol being a region of the photosensitive surface of an RGB-ToF sensor or a ToF sensor or a RGB-IR sensor such that at least some spots are present in the Rol. A CDAF method may use known focus measure functions to improve the contrast of the specified Rol. Because of the presence of the spots in the Rol, the focus measure functions can reliably assess whether spots in the Rol are in focus.

[0165] In some embodiments, only a single spot may be provided. If only a single spot is provided, the single spot is selected. If multiple spots are included in the intensity image data, a single spot may still be selected. For example, a spot located closest to a geometric center of the photosensitive surface of the ToF component coaxial RGB-ToF sensor 61 or the RGB-IR sensor 62 is selected.

[0166] Alternatively, a plurality of spots may be selected. For example, all spots located within a radius surrounding the geometric center of the photosensitive surface of the coaxial RGB-ToF sensor 61 or the RGB-IR sensor 62 is selected. Selection of the spots may be based on other criteria as well, as long as one or more specific spots are selected. For example, the same spots as are selected in the spot selection step in the depth image data (S221 in Fig. 9 above) may be selected. Spots may optionally be selected based on whether they were generated by laser beams emitted by the laser array intersecting a surface of a specific object. Spots may also be selected based on an algorithm or a selection of a machine learning algorithm or deep neural network.

[0167] Each spot may encompass multiple pixels of the photosensitive surface of the coaxial RGB-ToF sensor 61 or the RGB-IR sensor 62. The number of pixels encompassed by each selected spot are a measure of the size of that spot. The size of the spot may be used as spot profile of that spot.

[0168] The spot profile may be calculated based on the average size of all selected spots. Alternatively, the spot profile may be determined on a basis of an arithmetic mean of the sizes of all selected spots. Alternatively, the spot profile may be determined on a basis of a sum of the areas all selected spots.

[0169] Fig. 11 is a schematic illustration of an alternative process for providing autofocusing of a camera based on a spot profile. The autofocusing uses a direct adjustment S5. During fine adjustment, the optimal lens position of the lens 50 is determined by constructing a relation between the spot profile and the lens position, as described in more detail in Fig. 12.

[0170] The direct adjustment may comprise a direct estimation of the lens profile. The direct adjustment may be performed in situations exhibiting a low or medium blurriness of the spot profile.

[0171] Fig. 12 is a more detailed schematic illustration of a process for providing autofocusing of a camera based on a spot profile. The autofocusing uses a direct adjustment S5. The process comprises an acquisition step S51, a determination step S52, a lens position determination step S53, a construction step S54, an calculation step S55 and a movement step S56.

[0172] In the acquisition step S51, intensity image data is acquired by the ToF component of the coaxial RGB-ToF sensor (61 in Fig. 1) or the RGB-IR sensor (62 in Fig. 4) of a camera such as described in Fig. 1 or 4 above. Based on the intensity image of the spot pattern, the spot profile is determined in a determination step S52. The determination step S52 can follow the same procedure as in Fig. 10. Based on the spot profile, a relation between the spot profile and the lens position is constructed in the construction step S54. The relation between spot profile and the optimal lens position can be the relation R shown in Fig. 5B. The relation R can, for example, be constructed by acquiring a plurality of intensity images of the spot pattern with the lens 50 at different positions.

[0173] However, construction of the relation R may only be performed if the intensity images exhibit a low or medium blurriness.

[0174] From each intensity image, the spot profile is calculated. Each spot profile value with associated lens position defines a data point of the relation R. The relation R can then be constructed by fitting a suitable mathematical function to the data points.

[0175] The mathematical function can include a specific scattering parameter and / or other specific attributes of the surface that the spot pattern is generated on. From the relation R, the optimal lens position determined by calculating the lens position where the relation R is minimal in a calculation step S55. Determining the optimal lens position in step S55 may be called a direct estimation of the lens position. The lens position where the relation R is minimal is the optimal lens position. In a movement step S56 the lens 50 is moved to the optimal lens position. Following the determination if the optimal lens position and the movement step S56, the direct adjustment S5, and thus the autofocus procedure, is concluded. Following the autofocus procedure, RGB image data using the RGB-component of the coaxial RGB-ToF sensor 61 or the RGB-component of the RGB-IR sensor 62 may be acquired.

[0176] Fig. 13 schematically describes an embodiment of a device that implements the electronic device for focusing an optical assembly as described in the embodiments. The electronic device 1200 may implement all processes related to the embodiments. The electronic device 1200 comprises a CPU 1201 as processor. The electronic device 1200 further comprises an image sensor assembly 1210 connected to the processor 1201. The processor 1201 may implement any processes related to code-based runtime estimation as described above. Image sensor assembly 1210 may comprise any or all of a coaxial RGB-ToF sensor (61 in Fig. 1 above), a RGB-IR sensor (62 in Fig. 4 above) and a ToF sensor (63 in Fig. 4 above) and may generate intensity image data, depth image data, RGB image data or grey scale image data. The data generated by the image sensor assembly 1210 may be processed by the processor. The electronic device further comprises an optical assembly adjustment unit 1211, which, upon receiving instructions from the processor, may mechanically configure the optical assembly and / or move a lens (50 in Figs 1 and 4 above) from a first position to a second position. The electronic device further comprises a laser array 1213 (e.g. 40 in Figs. 1 and 4 above), which, upon receiving instructions from the processor, emits laser beams (41 in Figs. 1 and 4 above).

[0177] The electronic device 1200 may further comprise a user interface 1212 that is connected to the processor 1201. This user interface 1212 acts as a man-machine interface and enables a dialogue between a user and the electronic device. For example, the user may make configurations to the system using this user interface 1212. The electronic device 1200 may further comprise a Bluetooth interface 1204 and a WLAN interface 1205. These units 1204, 1205 may act as I / O interfaces for data communication with external devices. For example, other devices with WLAN or Bluetooth connection may be coupled to the processor 1201 via these interfaces 1204 and 1205. The electronic device 1200 further comprises a data storage 1202, and a data memory 1203 (here a RAM). The data storage 1202 is arranged as a long-term storage, e g. for storing image data, like depth image data, intensity image data, RGB image data, or user preference information or the like. The data memory 1203 is arranged to temporarily store or cache data or computer instructions for processing by the processor 1201

[0178] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the CPU described above, causes the methods described herein to be performed.

[0179] It should be recognized that the embodiments may describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. For example, the ordering of S52 and S53 in the embodiment of Fig. 8 may be exchanged. Other changes of the ordering of method steps may be apparent to the skilled person.

[0180] All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0181] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure. Note that the present technology can also be configured as described below.

[0182] (1) An electronic device comprising circuitry configured to determine a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and to perform focusing of an optical assembly based on the spot profile.

[0183] (2) The electronic device according to (1), wherein the focusing comprises determining an optimal configuration of the optical assembly.

[0184] (3) The electronic device according to any of (1) or (2), wherein the focusing comprises determining an optimal lens position.

[0185] (4) The electronic device according to any of (1) to (3), wherein the circuitry is configured to obtain image data from an image sensor representing visible light after the focusing of the optical assembly.

[0186] (5) The electronic device according to any of (1) to (4), wherein the circuitry is configured to determine a spot contrast from the spot profile.

[0187] (6) The electronic device according to any of (1) to (5), wherein determining the spot contrast is based on detecting a minimal spot size.

[0188] (7) The electronic device according to any of (1) to (6), wherein the focusing comprises performing a CDAF algorithm based on the intensity image data.

[0189] (8) The electronic device according to any of (1) to (7), wherein the circuitry further comprises a laser emission assembly providing a spot pattern.

[0190] (9) The electronic device according to any of (1) to (8), wherein the laser emission assembly is a VCSEL array.

[0191] (10) The electronic device according to any of (8) to (9), wherein the spot pattern comprises a plurality of spots.

[0192] (11) The electronic device according to any of (1) to (10), wherein the circuitry is configured to determine an initial configuration of the optical assembly based on ToF depth image data obtained from the ToF image sensor.

[0193] (12) The electronic device according to any of (1) to (11), wherein the circuitry is configured to determine a focus distance based on the depth image data, and to determine the initial configuration of the optical assembly based on the focus distance.

[0194] (13) The electronic device according to any of (1) to (12), wherein the circuitry is configured to perform a fine tuning of the configuration of the optical assembly based on the spot profile. (14) The electronic device according to any of (1) to (13), wherein the circuitry is configured to perform the fine tuning by iteratively capturing intensity image data and / or IR image data and configuring the optical assembly until the optimal configuration of the optical assembly is determined.

[0195] (15) The electronic device according to any of (1) to (14), wherein the circuitry is configured to perform a fine tuning of the configuration of the optical assembly by configuring the configuration of the optical assembly such that the spot contrast is maximal.

[0196] (16) The electronic device according to any of (11) to (15), wherein determining the initial configuration of the optical assembly precedes the fine tuning.

[0197] (17) The electronic device according to any of (1) to (16), wherein the circuitry is configured to perform a direct estimation of the optimal configuration of the optical assembly based on the spot profile.

[0198] (18) The electronic device according to any of (1) to (17), wherein the circuitry is configured to construct a relation between a configuration of the optical assembly and the spot profile.

[0199] (19) The electronic device according to any of (1) to (18), wherein the circuitry is configured to determine an optimal configuration of the optical assembly based on the relation between the configuration of the optical assembly and the spot profile.

[0200] (20) The electronic device according to any of (1) to (19), wherein the ToF image sensor is part of a coaxial RGB-ToF image sensor configured to obtain the intensity image data and / or depth image data.

[0201] (21) The electronic device according to any of (1) to (20), wherein the IR image sensor is part of an RGB-IR sensor configured to obtain the intensity image data.

[0202] (22) Imaging apparatus comprising the electronic device according to any of (1) to (21).

[0203] (23) A method comprising determining a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and performing focusing of an optical assembly based on the spot profile.

[0204] (24) A computer program that, if executed by a computer, causes the computer to determine a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and to perform focusing of an optical assembly based on the spot profile. LIST OF REFERENCE SIGNS

[0205] E Object

[0206] R Relation

[0207] Fl Focused state

[0208] F2 Defocused state

[0209] 1 Electronic device (first embodiment)

[0210] 2 Electronic device (second embodiment)

[0211] 3-39 Unassigned

[0212] 40 Laser emission assembly

[0213] 41 IR laser beams

[0214] 42 reflected IR light

[0215] 43-49 Unassigned

[0216] 50 Lens

[0217] 51-59 Unassigned

[0218] 61 Coaxial RGB-ToF sensor

[0219] 62 RGB-IR sensor

[0220] 63 ToF sensor

[0221] 64-109 Unassigned

[0222] 110 Single-aperture RGB+SPAD sensor

[0223] 111-399 Unassigned

[0224] 400 Dual sensing layer

[0225] 401 Dual band pass filter (DBPF)

[0226] 402 Visible light / RGB light

[0227] 403 infrared light

[0228] 404 Thin RGB sensitive layer

[0229] 405 Visible-cut filter 06 SPAD layer 07 Readout layer

[0230] 1200 electronic device

[0231] 1201 CPU 1202 Storage

[0232] 1203 RAM

[0233] 1204 Bluetooth interface

[0234] 1205 WLAN interface

[0235] 1210 Image sensor assembly 1211 Optical assembly adjustment unit

[0236] 1212 User interface

[0237] 1213 Laser array

Claims

CLAIMS1. An electronic device comprising circuitry configured to determine a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and to perform focusing of an optical assembly based on the spot profile.

2. The electronic device of claim 1, wherein the focusing comprises determining an optimal configuration of the optical assembly.

3. The electronic device of claim 1, wherein the focusing comprises determining an optimal lens position.

4. The electronic device of claim 1, wherein the circuitry is configured to obtain image data from an image sensor representing visible light after the focusing of the optical assembly.

5. The electronic device of claim 1, wherein the circuitry is configured to determine a spot contrast from the spot profile.

6. The electronic device of claim 5, wherein determining the spot contrast is based on detecting a minimal spot size.

7. The electronic device of claim 1, wherein the focusing comprises performing a CDAF algorithm based on the intensity image data.

8. The electronic device of claim 1, further comprising a laser emission assembly providing a spot pattern.

9. The electronic device of claim 8, wherein the laser emission assembly is a VCSEL array.

10. The electronic device according to claim 8, wherein the spot pattern comprises a plurality of spots.

11. The electronic device of claim 2, wherein the circuitry is configured to determine an initial configuration of the optical assembly based on ToF depth image data obtained from the ToF image sensor.

12. The electronic device of claim 11, wherein the circuitry is configured to determine a focus distance based on the depth image data, and to determine the initial configuration of the optical assembly based on the focus distance.

13. The electronic device of claim 11, wherein the circuitry is configured to perform a fine tuning of the configuration of the optical assembly based on the spot profile.

14. The electronic device of claim 13, wherein the circuitry is configured to perform the fine tuning by iteratively capturing intensity image data and / or IR image data and configuring the optical assembly until the optimal configuration of the optical assembly is determined.

15. The electronic device of claim 5, wherein the circuitry is configured to perform a fine tuning of the configuration of the optical assembly by configuring the configuration of the optical assembly such that the spot contrast is maximal.

16. The electronic device according to claim 13, wherein determining the initial configuration of the optical assembly precedes the fine tuning.

17. The electronic device of claim 2, wherein the circuitry is configured to perform a direct estimation of the optimal configuration of the optical assembly based on the spot profile.

18. The electronic device according to claim 1, wherein the circuitry is configured to construct a relation between a configuration of the optical assembly and the spot profile.

19. The electronic device according to claim 18, wherein the circuitry is configured to determine an optimal configuration of the optical assembly based on the relation between the configuration of the optical assembly and the spot profile.

20. The electronic device of claim 1, wherein the ToF image sensor is part of a coaxial RGB-ToF image sensor configured to obtain the intensity image data and / or depth image data.

21. The electronic device of claim 1, wherein the IR image sensor is part of an RGB-IR sensor configured to obtain the intensity image data.

22. A method comprising determining a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and performing focusing of an optical assembly based on the spot profile.

23. A computer program, that, if executed by a computer, causes the computer to determine a spot profile based on an intensity image obtained from a ToF image sensor or from an IR image sensor; and to perform focusing of an optical assembly based on the spot profile.

Images