Sensor arrangement

The sensor arrangement aligns field-of-views of image and depth sensors using a dichroic mirror and micro-scan devices, effectively addressing calibration complexities and parallax issues, enabling efficient and accurate depth-to-color mapping for optical sensor systems.

WO2024240509A1PCT designated stage expired Publication Date: 2024-11-28CHIRP LIGHT AG
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/062842
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-22
Filing Date
2024-05-08
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing optical sensor arrangements face challenges in merging depth and image data due to calibration complexities, particularly in large-scale production, hardware complexity, power consumption, and inaccuracies arising from differing viewpoints and field-of-view differences between sensors, which are exacerbated by the limitations of sparse depth sensors and parallax effects.

Method used

A sensor arrangement comprising an array of image sensors and sparse range sensors with a lens system to align field-of-views and a processor to associate depth pixels with image pixels, utilizing a dichroic mirror and micro-scan devices like MEMS, VCM, or OPA to achieve parallax-free imaging and efficient data mapping.

Benefits of technology

This solution enables straightforward and accurate mapping of depth to color information, avoiding dead spots and parallax effects, and is applicable to high-resolution image sensors, reducing computational complexity and hardware requirements, making it suitable for mass production.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024062842_28112024_PF_FP_ABST
    Figure EP2024062842_28112024_PF_FP_ABST
Patent Text Reader

Abstract

1. A sensor arrangement (1) for mapping depth information to colour information, comprising: - an image sensor (2) configured to capture colour information of a scene; - a depth sensor (3) configured to capture depth information of the scene; and - a lens system (4,5) configured to position the field of views of the depth sensor (3) and the image sensor (2) to substantially overlap; and - a processor (6) configured to associate a depth pixel with a corresponding image pixel.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] SENSOR ARRANGEMENT

[0002] DESCRIPTION

[0003] TECHNICAL FIELD

[0004] The present invention relates to a sensor arrangement for mapping depth information to colour information and a method for mapping depth information to colour information with such sensor arrangement .

[0005] BACKGROUND

[0006] Optical sensor arrangements , like time-of- f light ( ToF) sensors , face a challenge in merging data from image and depth sensors due to their intrinsic and extrinsic calibration . This calibration helps convert depth pixels into image pixels for mapping purposes . Kinect RGB-D sensor is an example of this process . In some cases , stereo calibration is used by capturing a series of images with a known pattern . Calibration can become a bottleneck for large-scale production and requires repetition for each sensor . Autocalibration algorithms are necessary as calibration may change during sensor lifetime . Joint calibration of sensors is done in a simultaneous locali zation and mapping ( SLAM) framework, but this adds hardware complexity and power consumption . Mapping dead spots can arise from di f ferences in the baseline and f ield-of-view ( FoV) of the sensors . Additionally, there can be inaccuracies due to di f fering viewpoints of obj ects between the sensors . High-resolution RGB-D sensors are not always practical , so machine learning is used to merge data from low-resolution depth imagers and high-resolution RGB sensors .

[0007] The classic way to solve the registration of depth and texture or image information used in most of the consumer RGB-D sensors from the last decade (e.g., Kinect or RealSense) is to treat it as a stereo calibration problem.

[0008] For example, the document "A Flexible New Technique for Camera Calibration" (Zhang, 1998) discloses the acquisition of a known planar calibration pattern in multiple poses viewed by both sensors. Non-linear optimization of a pro j ection / depro j ection model based on the detected common features on both cameras allow for the desired mapping. This technique is limited by the ability to capture an intensity image of a suitable quality from the depth sensor. For sparse dToF or iToF depth sensors it can become challenging and end up in suboptimal solutions. For frequency-modulated SMI signals, it is not even clear how to obtain such intensity images. Moreover, the method implies a parallax between the cameras, which is desired for stereo imaging, but detrimental for colour mapping.

[0009] Loveday et al. ("On the Impact of Parallax Free Colour and Infrared Image Co-Registration to Fused Illumination Invariant Adaptive Background Modelling"; Loveday, M; Breckon TP; CVPR workshop, CVPR Workshop 2018) proposes a method to fuse RGB and thermal cameras using a dichroic filter, however, the provided image registration is not applicable to sparse range sensors and the intensity image capture of the depth sensor stays insufficient.

[0010] The document US 8, 698,939 B2 describes a setup for multi- spectral imaging. However, in this document two images are formed at different and non-overlapping locations of the imaging array depending on the wavelength of light and it is not clear how two sub-images may be registered.

[0011] The document US 9,210,288 B2 discloses the registration of a point cloud and a colour image in a parallax-free strategy with a depth scanner being an amplitude-modulated laser scanner with rotary mirror and base and a line-scan colour camera .

[0012] SUMMARY

[0013] One obj ective of the present invention is to provide an improved sparse range sensor arrangement for mapping depth information to colour information .

[0014] With respect to the device the obj ective is met by a sensor arrangement according to the present invention for mapping depth information to colour information . Such sensor arrangement comprises at least one image sensor, preferably an array of image sensors , configured to capture colour information of a scene ; at least one sparse range sensor as depth sensor, preferably an array of depth sensors , configured to capture depth information of the scene ; and a lens system configured to position the field of views of the depth sensor and the image sensor to substantially overlap ; and a processor configured to associate a depth pixel with a corresponding image pixel .

[0015] The present invention thus provides a micro-scan concept for an optically overlaid high-resolution image sensor and a sparse depth sensor . The present invention thereby achieves a straightforward mapping between depth sensor data and image sensor data ( texture ) , particularly avoiding parallax or dead spots . Particularly a dense from sparse depth array is achieved with smaller deflection angles due to micro-scan .

[0016] As compared to standard RGB-D techniques to colour mapping, the present invention solves multiple inherent problems via an optical setup that aligns the field of views of depth and image sensors , particularly in embodiments comprising a dichroic mirror . While both sensors usually suf fer from parallax ef fect , the present invention advantageously removes parallax .

[0017] The present inventors thereby surprisingly found that although the idea of parallax- free imaging is not new, it can be advantageously achieved by registering a depth and a texture image .

[0018] Furthermore , although the applicability of methods known in the art is often limited to at least VGA sensors , the present invention is also applicable to 2D focal plane arrays . Particularly, a high-resolution image sensor and a potentially sparse depth sensor . Thereby, the present invention further provides a micro-scan concept combining it with the optical overlay of sensors . Lastly, the proposed calibration procedure is unique and can be practically implemented for mass production .

[0019] Preferably, the image sensor is a colour camera, for example , a regular CCD / CMOS array .

[0020] The term " sparse range sensor" as used herein, preferably refers to a type of range sensor that operates by capturing distance measurements at a limited number of discrete points in a scene . Unlike dense range sensors that capture a depth map with continuous measurements at each pixel , sparse range sensors preferably capture only a small number of depth measurements , resulting in a lower overall data density . The sparse range sensor may use various techniques to capture these distance measurements , such as time-of- f light or structured light .

[0021] Preferably, the image sensor and / or the depth sensor comprises a light source and a light detector .

[0022] Preferably, the sparse range sensor emits and receives light in the form of laser beams . Preferably, the image sensor is a high-resolution image sensor .

[0023] Preferably, the sensor arrangement of the present invention comprises an array of depth sensors .

[0024] More preferably, the depth sensor of the present invention emits frequency-modulated light with sel f-mixing interferometry or pulse-based or amplitude-modulated ToF .

[0025] In a preferred embodiment of the present sensor arrangement , the image sensor and the depth sensor are arranged such that the field of views of the depth sensor and image sensor are aligned .

[0026] In a further preferred embodiment of the sensor arrangement according to the present invention, the depth sensor and the image sensor are positioned orthogonally to each other . In other words , the main direction of emittance and thus the direction of light beams emitted by the image sensor and the depth sensor, respectively, are perpendicular each other .

[0027] In a further preferred embodiment of the sensor arrangement according to the present invention, the depth sensor and the image sensor are positioned in a plane side-by-side to each other . In other words , the main direction of emittance and thus the direction of light beams emitted by the image sensor and the depth sensor, respectively, are parallel to each other .

[0028] In a further preferred embodiment of the sensor arrangement according to the present invention, the sensor arrangement further comprises a mirror arrangement . Such mirror arrangement preferably comprises at least one first mirror .

[0029] Alternatively, the present sensor arrangement may comprise more than one mirror, particularly two mirrors , namely a first mirror and a second mirror . In the mirror arrangement the mirrors are arranged such that the field of views of the depth sensor and the image sensor are aligned .

[0030] In a further preferred embodiment of the sensor arrangement according to the present invention, the arrangement comprises at least a first mirror, wherein said first mirror is a dichroic mirror .

[0031] The term "dichroic mirror" as used herein, preferably refers to an optical component that selectively reflects and transmits light of di f ferent wavelengths or colours . A dichroic mirror as known in the art may comprise a thin coating with optical properties that allows reflection of certain wavelengths of light while transmitting others . Such dichroic mirrors are usually made of glass or other transparent material and coated with a thin layer of metal or dielectric material that has di f ferent reflective properties for di f ferent wavelengths of light .

[0032] In the context of the present invention a dichroic mirror can be advantageously arranged such that the field of views of the depth sensor and the image sensor are aligned .

[0033] Particularly, the depth sensor and the image sensor may be positioned in an angle , or particularly orthogonally, to each other . In such embodiment , the dichroic mirror may be arranged accordingly to align the field of views of the depth sensor and the image sensor .

[0034] In a further preferred embodiment of the sensor arrangement according to the present invention, the arrangement further comprises a micro-scan device . Such micro-scan device is advantageously selected from the group comprising a micro- electro-mechanical system (MEMS ) , a voice coil motor (VCM) , a steerable lens comprising a metalens and a MEMS , or an optical phased array ( OPA) .

[0035] Preferably, the micro-scan device comprises a micro-electro- mechanical system (MEMS ) configured to actuate one of the mirrors . Preferably, a MEMS is arranged at the first and / or the second mirror such that it can actuate the first and / or the second mirror, respectively . A person skilled in the art will immediately understand that a MEMS may be arranged to actuate a dichroic mirror or any other mirror .

[0036] Depending on the particular mirror arrangement the mirrors of the sensor arrangement of the present invention can be arranged in various embodiments such that the field of views of the depth sensor and image sensor are aligned .

[0037] In the context of this invention the term "MEMS" is preferably understood to be a microelectromechanical system . It may particularly refer to a technology of miniaturi zed mechanical and electro-mechanical elements that are fabricated using microfabrication techniques . MEMS devices are typically made of components that are smaller than 100 micrometres , and they are commonly used in a wide range of applications , such as sensors , actuators , and microfluidic systems . It is known in the art that MEMS technology enables the creation of highly precise and ef ficient devices that can perform complex functions with high reliability and accuracy .

[0038] For example , in the document "MEMS Mirrors for LiDAR : A Review" ( 2020 ) , Wang & al . provide an overview of LiDAR scanning techniques , with a focus on MEMS scanners . In general , there is a single laser whose beam is steered via MEMS or OPA. Unfortunately, this requires large scanning angles for reaching adequate field of views for typical automotive applications . In contrast thereto , the present invention is advantageous , as small scanning angles can be applied and the depth information from the dept sensor can be more directly combined with texture data from the image sensor .

[0039] In a further preferred embodiment of the sensor arrangement according to the present invention, the micro-scan device comprises a voice coil motor (VCM) configured to translate the lens system .

[0040] In the context of the present invention the term "VCM" or "voice coil motor" preferably refers to a type of electromagnetic actuator capable to control the position of components such as lenses or mirrors . VCMs are used in the art for various electronic devices , including smartphones and cameras . The VCM may particularly comprise a coil that moves inside a magnetic field, generating a force that is used to move the component . By controlling the amount of current supplied to the coil , the position of the component can be accurately controlled . VCMs are known for their precision, speed, and ability to provide high force with a small form factor .

[0041] In a further preferred embodiment of the sensor arrangement according to the present invention, the micro-scan device comprises a metalens and a MEMS configured to create a steerable lens .

[0042] As used herein, the term "metalens" preferably refers to a type of lens made of an array of nanostructures . Such metalens is typically made of metals or dielectrics arranged in a periodic pattern . These structures may be designed such that the phase , amplitude , and / or polari zation of the light may be altered, allowing the metalens to focus or shape light in speci fic ways . Unlike traditional lenses , metalenses may be made extremely thin and lightweight , making them an ideal technology in the context of the present invention for a wide range of applications , such as high-resolution imaging, optical communications , and augmented reality .

[0043] In a further preferred embodiment of the sensor arrangement according to the present invention, the micro-scan device comprises an optical phased array ( OPA) . The OPA may be configured to directly steer the laser beams emitted by the depth sensor . As used herein, the term "OPA" or "optical phased array" preferably refers to a device that uses multiple small , individually controllable light sources to produce a focused beam of light . Each light source produces a small portion of the overall beam, and by controlling the phase of each source , the individual beams can be made to interfere constructively, resulting in a focused beam that can be steered in di f ferent directions . In the context of the present invention, OPAs can be used in a variety of applications of the inventive sensor arrangement comprising lidar systems for autonomous vehicles , optical communications , and laser machining .

[0044] In a further preferred embodiment of the sensor arrangement according to the present invention, the depth sensor is a sparse range sensor emitting IR light . This is advantageous , particularly when using a dichroic filter arranged such that the field of views of the depth sensor and image sensor are aligned .

[0045] In a further preferred embodiment of the sensor arrangement according to the present invention, the image sensor comprises a VIS and / or IR sensor .

[0046] The person skilled in the art will acknowledge that the terms "VIS" and " IR" are commonly used abbreviations for di f ferent regions of the electromagnetic spectrum .

[0047] The term "VIS" stands for "visible" and refers to the range of wavelengths of electromagnetic radiation that are visible to the human eye . The visible spectrum refers to the spectrum between about 400 nanometres and about 700 nanometres . This includes the colours of the rainbow, from violet to red .

[0048] The term " IR" stands for " infrared" and refers to the range of wavelengths of electromagnetic radiation that are longer than those of visible light . The spectrum of " IR" refers to the spectrum between about 700 nanometres and 1 millimetre . This type of radiation is not visible to the human eye , but can be felt as heat , and is commonly used in various imaging applications .

[0049] In a further preferred embodiment of the sensor arrangement according to the present invention, the depth sensor is a sparse range sensor comprising a sensor array selected from the group comprising a Vertical-Cavity Surface-Emitting Laser (VCSEL ) and / or a PD / AS IC arrangement , a SPAD array and a flood or dot proj ector .

[0050] The term "VCSEL" or "Vertical-Cavity Surface-Emitting Laser" as used herein preferably refers to a type of semiconductor laser that emits light perpendicular to the surface of a chip . Usually, it is called "vertical cavity" because the active region of the laser is situated between two distributed Bragg reflector mirrors , which are perpendicular to the chip surface . VCSELs are known to have several advantages over traditional edge-emitting lasers , including low power consumption, high ef ficiency, and the ability to be fabricated in large arrays . They are used in a variety of applications , including optical communication, sensing, and 3D sensing in mobile devices .

[0051] The term "PD" refers to " Photodiode" and the term "AS IC" stands for "Application-Speci fic Integrated Circuit" .

[0052] As used herein, the term "photodiode" or "PD" as used herein, preferably refers to a semiconductor device that converts light into an electrical current . It works by absorbing photons from incoming light and generating a flow of electrons in response . Photodiodes are commonly used in applications such as optical communications , sensing, and imaging .

[0053] The term "application-speci fic integrated circuit" or "AS IC" as used herein, preferably refers to a type of integrated circuit ( IC ) , more preferably designed for the speci fic application or function according to the present invention . In the context of the present invention, the AS IC is preferably used for processing the electrical signals generated by the photodiode and controlling the operation of the VCSEL .

[0054] In a further preferred embodiment of the sensor arrangement according to the present invention, , the sensor arrangement further comprises a dot detector .

[0055] The term "dot detector" as used herein preferably refers to an algorithm capable of detecting the position of laser dots or spots on an image taken by the image sensor . It preferably works by identi fying local maxima in the intensity of the image and using quadratic interpolation to estimate the precise location of the laser dot . The dot detector preferably is used to extract the xy coordinates of the laser spots in the image and store them in a table for each mechanical position of the mirror or lens . This information preferably is then used at runtime to generate a 3D point cloud of the scene . The dot detector thus may play a crucial role in the proposed hardware solution according to the present invention as it enables the association of depth information with image information for dense RGB-D reconstruction from a sparse depth sensor in the sensor arrangement according to the present invention .

[0056] Hence , a dot detector in sensor arrangement according to the present invention preferably is configured to extract the locations of laser spots on the image sensor .

[0057] Further, a dot detector in sensor arrangement according to the present invention preferably is for extracting the xy proj ections of the laser spots onto the high-resolution imager into one table per mechanical position of a mirror or lens .

[0058] Still further, a dot detector in sensor arrangement according to the present invention preferably is based on local maxima and quadratic interpolation . In a further preferred embodiment of the sensor arrangement according to the present invention, the processor is configured to generate a 3D point cloud of the scene .

[0059] In such embodiment a 3D point cloud of the scene is preferably generated by the processor by tracing rays from the xy stored locations through the optical centre of the image sensor until the estimated depth values , thereby producing a 3D point cloud of the scene .

[0060] In a further preferred embodiment of the sensor arrangement according to the present invention, the processor is configured to attach an intensity or colour to each 3D point . Thereby, the processor may take either the value of the closest corresponding image pixel and / or a linear combination of neighbouring pixels weighted by their distance to the xy location .

[0061] In a further preferred embodiment of the sensor arrangement according to the present invention, the sensor arrangement according to the present invention may further comprise an RGB image sensor . Such RGB Image sensor advantageously may be configured to enable auto-calibration of the setup by allowing continuous visibility of the dots . Preferably, the processor is configured to perform dense RGB-D reconstruction from the sparse depth sensor . Further, the processor may be configured to require only the geometric camera calibration of the high-resolution image sensor for camera calibration . Additionally, or alternatively, the processor may be configured to store the association between depth sensor data and image sensor data in look-up tables , preferably thereby reducing computational complexity .

[0062] The above obj ectives are also solved in a second aspect by a method for mapping depth information to colour information with a sensor arrangement according to the present invention .

[0063] It shall be noted, that the steps given below do not necessarily have to be carried out in the given order . The provided steps may be carried out in any other suitable order .

[0064] However, the order as given in the following may apply for particular variants of the method .

[0065] It will be immediately acknowledged by a person skilled in the art that a feature , embodiment , ef fect or advantage described herein in connection with the inventive sensor arrangement , may also be a feature , embodiment , ef fect or advantage of the inventive method, respectively, and vice versa .

[0066] Such method according to the present invention comprises at least the following steps : a step of capturing depth information of a scene using at least one sparse range sensor as depth sensor ; a step of capturing colour information of the scene using at least one image sensor ; optionally, a step of aligning the field of views of the depth sensor and the image sensor using a dichroic mirror and a lens system to substantially overlap ; and a step of associating a depth pixel with a corresponding image pixel using a processor .

[0067] Thereby, preferably the sensor arrangement is the sensor arrangement according to the present invention .

[0068] In a preferred embodiment of the method according to the present invention, the method further comprises a step of extracting the locations of the laser spots on the image sensor using a dot detector . Typical fields of application for a sensor arrangement and / or the method according to the invention comprise but are not limited to :

[0069] • 3D scanning

[0070] • SLAM

[0071] • predictive motion

[0072] • 4d ranging with VCSEL array based on sel f-mixing interferometry ( SMI )

[0073] • ToF sensor

[0074] • automotive sensors

[0075] • AR / VR sensors

[0076] • Robotic sensors

[0077] • surveillance sensors

[0078] These and other features , embodiments , obj ectives , and advantages of the invention will become apparent from the subsequent description .

[0079] BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Exemplary embodiments of the invention are discussed below with reference to the accompanying drawings .

[0081] FIG . 1 illustrates schematically a sensor arrangement and respective field of illumination and a field of view according to a first embodiment of the present invention, wherein the depth sensor and the image sensor are positioned orthogonally to each other ;

[0082] FIG . 2 illustrates schematically a sensor arrangement and respective field of illumination and a field of view according to a second embodiment of the present invention, wherein the depth sensor and the image sensor are positioned in a plane side-by-side to each other ;

[0083] FIG . 3 A and 3 B illustrate schematically a sensor arrangement and respective field of illumination and a field of view according to a third and fourth embodiment of the present invention, wherein the sensor arrangement comprises a micro-scan device in the form of a micro-electro-mechanical system (MEMS ) ;

[0084] FIG . 4 illustrate schematically a sensor arrangement and respective field of illumination and a field of view according to a fi fth embodiment of the present invention, wherein the sensor arrangement comprises a micro-scan device in the form of a VCM;

[0085] FIG . 5 illustrate schematically a sensor arrangement and respective field of illumination and a field of view according to a sixth embodiment of the present invention, wherein the sensor arrangement comprises a micro-scan device in the form of a steerable metalens ;

[0086] FIG . 6 illustrate schematically a sensor arrangement and respective field of illumination and a field of view according to a sixth embodiment of the present invention, wherein the sensor arrangement comprises a micro-scan device in the form of an OPA;

[0087] FIG . 7 illustrates schematically a sensor arrangement and respective field of illumination and a field of view according to a seventh embodiment of the present invention, wherein the depth sensor and the image sensor are positioned in a plane side-by-side to each other and the field of views of the sensors do not overlap .

[0088] DETAILED DESCRIPTION

[0089] The sensor arrangement 1 according to the present invention is suitable for mapping depth information to colour information, and comprises an image sensor 2 configured to capture colour information of a scene and a sparse range sensor as a depth sensor 3 that is configured to capture depth information of the scene . In addition, a lens system 4 , 5 is provided that is configured to position the field of views of the depth sensor 3 and the image sensor 2 , respectively, to substantially overlap . For this reason the image sensor 2 and the first lens 4 are arranged adj acent to each other, such that the light emitted by the image sensor 2 passes lens 4 . The depth sensor 3 is arranged adj acent to a second lens 5 , such that the light emitted by the depth sensor 3 passes second lens 5 .

[0090] In the present sensor arrangement 1 , additionally a processor 6 is provided that is configured to associate a depth pixel as sensor data from the depth sensor 3 with a corresponding image pixel as sensor data from the image sensor 2 .

[0091] In FIG . 1 a sensor arrangement and respective field of illumination and a field of view according to a first embodiment of the present invention is schematically shown .

[0092] In the shown embodiment , a the depth sensor 3 and the image sensor 2 are positioned orthogonally to each other .

[0093] Thereby, the depth sensor 3 and the image sensor 2 are arranged such that the field of views of the depth sensor 3 and image sensor 2 are aligned . This is , as the sensor arrangement 1 further comprises a mirror arrangement in the form of the first mirror 7 that is a dichroic mirror arranged to align the field of views of the depth sensor 3 and the image sensor 4 . Notably, the light of image sensor 2 passes mirror 7 , while the light emitted by depth sensor 3 is reflected by the dichroic mirror 7 . In such way, the dichroic mirror 7 aligns the field of views of the depth sensor 3 and the image sensor 4 .

[0094] Accordingly, the image sensor' s 2 and depth sensor' s 3 receiver planes are placed orthogonal to each other and are then aligned by the single dichroic mirror 7 ( see FIG . 1 ) .

[0095] This is , as infrared laser beams as emitted by the depth sensor 3 are nearly fully reflected by the dichroic mirror 7 , while visible light is nearly fully transmitted to the image sensor 2 . A person skilled in the art will understand that , on the one hand, the field of views of the image sensor 2 and depth sensor 3 shall be as close as possible by a suitable choice of lenses 4 and 5, respectively. On the other hand, the mirror 7 shall be mounted in such a way that the field of views of both sensors overlay (approx. 45 degrees) . Using this setup, the present invention advantageously provides a unique association of a depth to an image pixel, regardless of the object distances. Furthermore, it is provided that there is always a matching image pixel to a depth pixel, with no dead spots in the field of view. Finally, the calibration process is facilitated. Indeed, the geometric camera calibration of the high-resolution image sensor 2 directly provides the mapping between ID range measurements and a 3D point cloud by deprojection of the pixel rays.

[0096] In the particular embodiment of the present invention as shown in FIG. 1 a high-resolution color sensor (e.g., a 3264x2448 CMOS image sensor) may be used as image sensor 2 associated with a corresponding lens 4, and a sparse VCSEL array (e.g., 15x20 elements) is used as depth sensor 3, associated with its corresponding lens 5, and further comprising a sparse photodiode array 3 (e.g., 15x20 elements) . As filter 7 a dichroic filter (e.g., with a transmission band between 400 and 730nm, and a reflection band between 770 and llOOnm) is provided in a 45 degree angle relative to the emission main direction of the image sensor 2 and the depth sensor 3, respectively. The VCSEL / PD arrays 3 are placed at an angle of 90 degrees with respect to the image sensor 2. The dichroic filter 7 is placed in front of the image sensor lens 4 with an angle of 45 degrees. The image sensor 2 may be a monochrome or infrared camera with lower resolutions, e.g., VGA. The VCSEL / PD arrays 3 implement self-mixing interferometry (SMI) with frequency modulation to measure absolute distances and Doppler speeds. The photodiode array 3 may be integrated directly into the VCSELs 3 or may not be needed in case the SMI signal is read from the junction voltages. The VCSEL / PD arrays 3 may be replaced by a SPAD array 3 and a flood or dot projector 3. In this case, the depth sensor 3 estimates directly the time of flight . Additionally or alternatively, in an embodiment of the invention an amplitude-modulation may be used and the di f ference of phases ( indirect ToF) may be measured .

[0097] In FIG . 2 a sensor arrangement 1 and respective field of illumination and a field of view according to a second embodiment of the present invention is shown . Compared to the embodiment shown in FIG . 1 the depth sensor 3 and the image sensor 2 are positioned in a plane side-by-side to each other . Accordingly, the image sensor and the depth sensors receiver planes are arranged next to each other and combined with two respective mirrors 7 and 8 , respectively ( see FIG . 2 ) . In other words , the sensor arrangement 1 comprises a mirror arrangement 7 , 8 , with one first mirror 7 , and one second mirror 8 , both arranged to align the field of views of the depth sensor 3 and the image sensor 4 .

[0098] The first mirror 8 reflects the laser beams emitted by the depth sensor 3 into a second dichroic mirror 7 like in the embodiment shown in FIG . 1 . This second mirror 7 may act in transmission or reflection depending on the wavelength . Like in the first embodiment of FIG . 1 , the mirrors 7 and 8 are positioned in such a way that the field of views of both sensors 2 and 3 match, respectively . In this setup, the second mirror 8 may be a fully reflective mirror 8 that reflects the laser beams emitted by the depth sensor 3 into the dichroic filter 7 . This setup may be of particular advantage for deployment in mobile or AR / VR use cases .

[0099] A third and fourth embodiment of the present invention, is shown in FIG . 3 A and 3 B, respectively . Here , the sensor arrangement 1 comprises a micro-scan device 9 in the form of a micro-electro-mechanical system (MEMS ) . In both embodiments , the basic setup is based on the second embodiment shown in FIG . 2 comprising one first mirror 8 that reflects the laser beams emitted by the depth sensor 3 into a second dichroic mirror 7 . In addition to this embodiment , and to overcome the inherent sparsity of current depth sensor arrays as compared to high-resolution colour imager, the embodiments shown in Figs . 3A and 3B comprise a micro- electro-mechanical system (MEMS ) 9 to actuate either the first mirror 8 ( see FIG . 3A) or the second mirror 7 ( see FIG . 3B ) . Of course , it is within the scope of the invention that both and / or also any further mirror is provided with such micro-electro-mechanical system (MEMS ) 9 .

[0100] As compared to other LIDAR solutions based on MEMS mirrors , the shown embodiments of FIG . 3A and 3B necessitates only small deflection angles ( couple of degrees as compared to approximatively 20 degrees ) , which puts less stress on the mechanical design .

[0101] The MEMS actuator 9 steer the respective mirror 7 and / or 8 up to 3 . 2 degrees with steps of 0 . 1 degree along two orthogonal axes . With this modi fication, the original depth resolution maybe extended, for example , of 15x20 may up to 480x640 pixels at a rate of 30 fps using a capture time of 33 microseconds per pixel array .

[0102] In another embodiment of the present invention, as shown in FIG . 4 , according to a fi fth embodiment of the present invention, the sensor arrangement 1 comprises a micro-scan device 9 in the form of a VCM 9 . Such voice coil motor (VCM) is disposed on the lens 5 . In such setup, the VCM may generate a force that is used to move the lens 5 which may be thus accurately controlled .

[0103] As a further embodiment of the sensor arrangement 1 of the present invention as shown in FIG . 5 a steerable metalens 9 is provided as the micro-scan device 9 , preferably alternatively or in addition to a MEMS micro-scan device 9 . The metalens may be provided in addition to or, preferably as a replacement of lens 5 .

[0104] As a further embodiment of the sensor arrangement 1 of the present invention as shown in FIG . 6 an OPA 9 is provided as the micro-scan device 9 . Such optical phased array ( OPA) 9 advantageously allows to directly steer the laser beams ( FIG . 6 ) .

[0105] In all embodiments as shown in Figs 1 , 2 , 3A, 3B, 4 , 5 and 6 , a sensor array may be used, which advantageously may improve the depth resolution at low cost by small and configurable mechanical motions . Provided an intrinsic camera matrix for the high-resolution sensor, obtained by standard calibration methods , the present invention allows for a straightforward mapping of depth to texture pixels and hence to create a coloured 3D point cloud .

[0106] The mapping may also be created in an of fline phase by pointing the RGB-D device to a wall made of material sensitive to infrared light , such as Thorlabs VRC5 , with 700- 1400nm and with a peak relative sensitivity of 1 . 0 pm wavelength .

[0107] In FIG . 7 a sensor arrangement 1 and respective field of illumination and a field of view according to a seventh embodiment of the present invention is schematically shown . Thereby, the optical principles may apply as depicted in FIG . 2 of the article Freundlich et al . , "Controlling a Robotic Stereo Camera Under Image Quanti zation Noise" in The International Journal of Robotics Research published in June 2017 .

[0108] The depth sensor 3 and the image sensor 2 are positioned in a plane side-by-side to each other and the field of views of the sensors do not overlap . In this particular embodiment an alternative way of solving the problem using a stereo-like setup is shown within the scope of the present invention . It is immediately apparent from FIG . 7 that in such embodiment the field of views of the sensors 2 and 3 do not overlap . Also without further mechanical elements , such as mirrors , a single image is suf ficient . Whenever MEMS or VCM are used, one image per angle step is required . As the laser beams will appear as spots on the texture image , at fixed locations at any distances for a given setup, their locations may be advantageously extracted using a dot detector based on local maxima and quadratic interpolation . As a result , the xy proj ections of the laser spots are stored onto the high- resolution imager into one table per mechanical position of the mirror or lens . At runtime , rays are advantageously simple traced from the xy stored locations through the camera optical centre of the image sensor 2 until the estimated depth values . This will generate a 3D point cloud of the scene . Additionally, an intensity or colour may be attached to each 3D point by taking either the value of the closest corresponding image pixel or a linear combination of neighbouring pixels weighted by their distance to the xy location .

[0109] Auto-calibration of the setup could be easily achieved by using an RGB image sensor 2 without infrared filter, allowing the continuous visibility of the dots . However, the image would need to be post-processed to remove them before outputting it to the user .

[0110] All embodiments of the present invention as described above in the claims and description and in relation to the Figures here , solve the basic computer vision problem of associating depth to texture . Particularly, the present invention provides a relatively simple hardware solution advantageous compared to state-of-the art , in that :

[0111] - The present invention provides a lower computational complexity in that the association between depth and texture remains fixed and can be stored in look-up tables , in contrast to classic solutions requiring depro ection / pro ection;

[0112] - The present invention provides a full matching of image and depth sensor field of views , particularly without providing dead spots , i . e . all depth pixels can be associated to a texture pixel ; - The present invention provides a rather simple camera calibration, in that the inventive method requires only the geometric camera calibration of the high-resolution image sensor - dense RGB-D reconstruction from sparse depth sensor

[0113] - The present invention is suitable also for small deflection angles for steerable mirror or lens .

[0114] - The present invention is in need of only a small degree of computational power for depth reconstruction .

[0115] The embodiments in the figures may relate to preferred embodiments , while all elements and features described in connection with embodiments may be used, as far as appropriate , in combination with any other embodiment and feature as discussed herein, in particular related to any other embodiment discussed further above .

[0116] LIST OF REFERENCE SIGNS

[0117] 1 sensor arrangement

[0118] 2 image sensor 3 depth sensor

[0119] 4 first lens

[0120] 5 second lens

[0121] 6 processor

[0122] 7 dichroic mirror 8 mirror

[0123] 9 micro-scan device

Claims

CLAIMS1. A sensor arrangement (1) for mapping depth information to colour information, comprising: at least one image sensor (2) configured to capture colour information of a scene; at least one sparse range sensor as depth sensor (3) configured to capture depth information of the scene; and - a lens system (4,5) configured to position the field of views of the depth sensor (3) and the image sensor (2) to substantially overlap; and a processor (6) configured to associate a depth pixel with a corresponding image pixel.

2. The sensor arrangement (1) according to claim 1, wherein the image sensor (2) and the depth sensor (3) are arranged such that the field of views of the depth sensor (3) and image sensor (2) are aligned.

3. The sensor arrangement (1) according to claim 1, wherein the depth sensor (3) and the image sensor (4) are positioned orthogonally to each other.

4. The sensor arrangement (1) according to claim 1, wherein the depth sensor (3) and the image sensor (4) are positioned in a plane side-by-side to each other.

5. The sensor arrangement (1) according to any one of claims 1 to 3, further comprising a mirror arrangement (7, 8) , preferably comprising at least one first mirror (7) , more preferably two mirrors (7, 8) , namely a first mirror (7) and a second mirror (8) , the mirror arrangement (7, 8) arranged to align the field of views of the depth sensor (3) and the image sensor (4) .

6. The sensor arrangement (1) according to claim 5, wherein at least the first mirror (7) is a dichroic mirror arranged to align the field of views of the depth sensor (3) and the image sensor (4) .

7. The sensor arrangement (1) according to any one of claims 1 to 5, further comprising a micro-scan device (9) , wherein the micro-scan device is selected from the group comprising a micro-electro-mechanical system (MEMS) , a voice coil motor (VCM) , a steerable lens comprising a metalens and a MEMS, or an optical phased array (ORA) .

8. The sensor arrangement (1) according to any one of claims 1 to 7, wherein the image sensor (2) comprises a VIS and / or IR sensor.

9. The sensor arrangement (1) according to any one of claims 1 to 8, wherein the depth sensor (3) is a sparse range sensor emitting IR light.

10. The sensor arrangement (1) according to any one of claims 1 to 9, wherein the sensor arrangement (1) further comprises a dot detector.

11. The sensor arrangement (1) according to any one of claims 1 to 10, wherein the processor (6) is configured to generate a 3D point cloud of the scene.

12. The sensor arrangement (1) according to claim 11, wherein the processor (6) is configured to attach an intensity or colour to each 3D point.

13. The system of claim 1, further comprising an RGB image sensor (4) configured to enable auto-calibration of the setup by allowing continuous visibility of the dots.

14. A method for mapping depth information to colour information with a sensor arrangement, comprising at least the following steps:a step of capturing depth information of a scene using at least one sparse range sensor as depth sensor (3) ; a step of capturing colour information of the scene using at least one image sensor (4) ; optionally, a step of aligning the field of views of the depth sensor (3) and the image sensor (4) using a dichroic mirror and a lens system to substantially overlap; and a step of associating a depth pixel with a corresponding image pixel using a processor (6) , wherein preferably the sensor arrangement is the sensor arrangement (1) according to any one of claims 1 to 13.

15. The method according to claim 16, further comprising a step of extracting the locations of the laser spots on the image sensor (4) using a dot detector.

Citation Information

Patent Citations

  • Dual beam optic with dichroic filter

    US8698939B2

  • Three-dimensional scanner with dichroic beam splitters to capture a variety of signals

    US9210288B2

  • Device for environmental sensing and method for its operation

    DE102017222614A1

  • Detection device, control method, fusion detection system, and terminal

    EP4276495A1

  • Calibration of a depth sensing array using color image data

    US20210208262A1