Sensing system

Abstract

The invention relates to a sensing system (1) for sensing distance and speed comprising at least one laser SMI (self- mixing interferometry) illuminator (2) configured to emit a laser beam onto a scene, a modulation unit configured to modulate the emitted laser beam, an event-based pipeline for SMI (self-mixing interferometry) detection configured to process the modulated laser beam to detect events indicative of changes in the SMI signal, a neuromorphic sensor (4) configured to capture images of the scene in response to the detected events, and a processor configured to analyze the detected events and the captured images to determine characteristics of the scene. Furthermore, a corresponding method for sensing distance and speed is disclosed.

Description

[0001] 2023PF01790 - 1 - SENSING SYSTEM DESCRIPTION TECHNICAL FIELD The present invention relates to a sensing system. BACKGROUND LIDAR (Light Detection and Ranging) technology, based on pulse light measurements, has been widely used for distance estimation, object detection and ranging. It finds applications in various domains such as automotive safety, mobile applications (e.g. for face recognition), home entertainment (e.g. gesture capture for video games), and augmented reality. LIDAR systems determine the distance to an object by emitting laser pulses onto a scene. These pulses reflect off the object, and a receiver of the system measures the time it takes for photons to travel to the object and return to the LIDAR receiver. They further analyze detected signals to identify reflected signal pulses amidst background light. By measuring the time it takes for a pulse to travel from transmission to reception of the corresponding reflected pulse using a time-of-flight detection, LIDAR calculates the distance to an object. Furthermore, they can create a “point cloud” by collecting distance measurements from sequentially emitted laser pulses. 2023PF01790 - 2 - These point clouds can then be visualized as 3D images or used for various purposes, such as detecting obstacles. Depending on the number of vertical layers a LIDAR system has, it can be a 2D or 3D LIDAR that creates a point cloud of the environment. In recent years, however, a new technology called "4D LIDAR" has emerged. Unlike traditional 3D LIDAR systems that map static environments, 4D LIDAR can capture the movement of objects over time, adding an extra dimension, which is the time dimension. This makes 4D LIDAR particularly useful for applications such as autonomous vehicles, where it is important to accurately track moving objects. Traditional LIDAR systems based on the Time-of-Flight (ToF) method create 3D maps by measuring the time it takes for a beam of light to travel to an object and return. FMCW-LIDAR, on the other hand, measures not only the distance to objects, but also their speed. This is achieved by continuously changing the frequency of the emitted light. When the reflected light returns, the difference between the current frequency of the emitted light and the frequency of the reflected light is measured. This frequency difference can then be used to calculate the speed of the object. Therefore, an FMCW LIDAR measurement provides the radial velocity for each pixel, effectively providing a 4D image. The fourth dimension in "4D LIDAR" refers then to the velocity (or change in position over time) of the detected objects. A Self-mixing interferometry 4D ranger system uses self- mixing or back-injection laser interferometry, which involves redirecting some of the light reflected from a moving object 2023PF01790 - 3 - back into the laser cavity. This leads to alterations in the intensity and frequency of the emitted laser beam, allowing it to measure the distance travelled by the reflected beam. This enables the laser to act as a sensor for distance, speed, or vibration. Yet, both systems for detecting simultaneously speed and distance, that is 4D, have several drawbacks concerning the architecture, as well as the proper calibration. Both systems rely on an external camera or calibration method to correctly position the rays onto the real-world scene. This can make said systems sensitive to environmental conditions. This could lead to inaccuracies in the system’s output and therefore requires frequent recalibration, which can be time-consuming and inconvenient. Furthermore, the need for an external camera or calibration method increases its complexity and cost. Furthermore, FMCW LIDAR requires photonic integrated circuits, which have low coupling and require active alignment of several pieces. Additionally, both systems require additional blocks for detection, even in the presence of cameras in the system, resulting in a fairly high-power consumption, extra components, and a complex architecture. There is also an additional need for the use of calibration in both systems then, which requires computing time, and readout circuits for the SMI-Based 4D ranger. These typically include two amplifiers, filters, an Analog-to-Digital Converter (ADC), and data transmission, which is one of the major contributors to the high-power consumption in these systems. 2023PF01790 - 4 - In navigation systems based on active illumination as for example it is the case with time-of-flight or the above-named architectures, there may be tensions on the parts that may displace the beam with respect to the ego-system calibration constituting a dynamic error. This might happen if they are very dynamical and with large baselines, as it can be the case with robots, drones, and the like. Also, slow piece displacements, constituting an offset error may contribute to a drift in the calibration that will remain undetected until a new calibration is performed, which is of course unsatisfying. In the context of using active illumination, particularly in complex scenes with a limited number of dots, it can become challenging to identify individual dots due to geometric factors and displacement caused by the baseline, which is also a drawback in said systems and additional calibration steps are needed. SUMMARY   The object of the present invention is therefore to provide a sensing system with less power consumption and a less complex architecture. According to the invention, the object is met by a device specified in claim 1 and by a method with the steps specified in claim 14. Therefore, a sensing system for sensing distance and speed is proposed by the invention, comprising: at least one laser SMI (self-mixing interferometry) illuminator configured to emit a laser beam onto a scene, a modulation unit configured to modulate the emitted laser beam, an event-based pipeline for 2023PF01790 - 5 - SMI (self-mixing interferometry) detection configured to process the modulated laser beam to detect events indicative of changes in the SMI signal, a neuromorphic sensor configured to capture images of the scene in response to the detected events, and a processor configured to analyze the detected events and the captured images to determine characteristics of the scene. The combination of a neuromorphic sensor and in field-of-view coherent laser illumination that, via self-mixing interferometry is modulated in a way that it permits the extraction of the distance and the speed via the neuromorphic sensor readout, reduces greatly the power consumption, as it does need further components. In this implementation, the SMI based illuminator is much simplified, as it makes use of the event-based readout pipeline existing on the neuromorphic sensor already, which simplifies the architecture of the sensing system. Generally, it is desirable to employ a frequency modulation, which is quite sensitive, and which is less interference- prone for disruptions from e.g. sunlight or other (LIDAR) signals. In this regard it can be considered that in an embodiment of the invention the modulation may preferably be a FMCW (Frequency-Modulated Continuous-Wave) modulation. FMCW LIDAR uses frequency modulation to measure the distance and velocity of an object. Unlike traditional ToF LIDAR systems, which measure the time it takes for a light pulse to travel to an object and back, FMCW LiDAR measures the change in frequency of the returned signal. It provides a high accuracy of range measurement, while it is less prone to interferences and much more sensitive than ToF 2023PF01790 - 6 - sensors. Furthermore, it can simultaneously detect the position and velocity of an object in a single measurement step through deploying the Doppler frequency shift. In order to further lower the power consumption of the sensing system, the neuromorphic sensor may be an event-based camera, which consumes relatively little power. A neuromorphic sensor is a type of sensor that is designed to mimic the structure and function of biological sensors. Neuromorphic sensors are in general energy-efficient and have a low latency. Furthermore, they can process large amounts of data in real time. An event-based camera is an intrinsic intensity-change detection device, and it can achieve a high rate of frames per second to measure the Doppler frequencies created by the SMI effect on the laser beam. An event-based camera responds to local changes in brightness. Each pixel inside an event-based camera operates independently and asynchronously, reporting changes in brightness as they occur. For this, each pixel in an event- based camera stores a reference brightness level. Continuously, it compares this reference level to the current brightness level. When the difference in brightness exceeds a threshold, the pixel will reset its reference level and generate a so-called event. Thus, event cameras output an asynchronous stream of events triggered by changes in scene illumination. On top of that, event-based cameras have a high dynamic range, which making them suitable for a range of lightning conditions. 2023PF01790 - 7 - In another embodiment of the invention, the at least one SMI illuminator may be a VCSEL (Vertical Cavity Surface Emitting Laser) or an EEL (Edge Emitting Laser). VCSELs offer the advantage of a high resolution, which leads to a high object detection accuracy. On top of that, they are not prone for electromagnetic interference. Whereas EELs have a high stability and a low relative intensity noise, which improves the signal-to-noise ratio. On top of that EELs have a narrow linewidth and a long coherence length. In a further embodiment the sensing system may comprise a plurality of laser SMI (self-mixing interferometry) illuminators, wherein at least two laser SMI (self-mixing interferometry) illuminators out of the plurality of laser SMI illuminators may have a different carrier frequency. With this, the problem of identifying the dots with parallax or in adversary systems is solved, as the sensing system will look for these preestablished carrier frequencies on the image. Each dot is self-localized via its own frequency of events and self-calibrated accordingly. Therefore, no calibration for the dot position is thus needed. In some systems based on active illumination, e.g. navigation systems, if they are very dynamical and with large baselines (robots, drones, etc.), there may be tensions on the parts that may displace the laser beam with respect to the ego- system calibration constituting a dynamic error. Also, slow piece displacements, constituting an offset error, may contribute to a drift in the calibration that will remain undetected until a new calibration is performed. This can be avoided with said arrangement. 2023PF01790 - 8 - To further simplify the detection of the pixels on the scene, which can be linked spatially and in a time domain, the sensing system may further comprise a reference neuromorphic sensor. To reduce the power consumption, this reference neuromorphic sensor preferably may be an event-based camera. In general, it is desirable to have a good signal to noise ratio. A further possibility to reach that is that the sensing system may further comprise a frequency comparison and flagging block. A frequency comparison and flagging block further enables the detection of very close-range targets, which is not possible with other systems and helps to render the system immune to electromagnetic interference or noise. Furthermore, since the sensing system transmits continuously, the frequency and comparison block simplifies the process of information extraction. Furthermore, it is in many cases desirable to build the sensing systems as compact as possible. For this, in another embodiment of the sensing system, the at least one laser SMI (self-mixing interferometry) illuminator or the plurality of laser SMI illuminators and the neuromorphic sensor may be arranged on the same ASIC (Application-Specific Integrated Circuit). Also, the energy efficiency and data processing speed can be improved by this arrangement, as the distances for raw data transmission are minimized. On the other hand, it might make sense in some other cases to instead opt for more modularity and flexibility: in another embodiment of the invention the at least one laser SMI 2023PF01790 - 9 - illuminator or the plurality of laser SMI illuminators and the neuromorphic sensor may be arranged on different ASICs. Using such an arrangement, the system is more adaptable to changes due to its modularity and each component might be optimized independently without affecting the other. Besides, it provides more flexibility in terms of system design and integration. With that, each ASIC can be optimized for its specific task. Preferably, the at least one laser SMI illuminator or the plurality of laser SMI illuminators and the neuromorphic sensor may be arranged on the same substrate. Again, compactness is a reason for choosing this arrangement and one of its advantages. Moreover, this arrangement is more cost- effective than arranging both on separate substrates and the assembly is less complex. In another embodiment, the at least one laser SMI (self- mixing interferometry) illuminator or the plurality of laser SMI illuminators and the neuromorphic sensor may be arranged on different substrates. With this configuration, the sensing system offers increased flexibility in system design and integration. In this context, the components of each substrate can be fine-tuned for their particular function. Furthermore, the system gains adaptability to modifications thanks to its modular nature, and it allows for the independent optimization of each component without mutual interference. Moreover, the invention proposes a method for sensing distance and speed, which comprises at least the steps of: emitting a laser beam onto the scene using at least one laser self-mixing interferometry (SMI) illuminator, modulating the 2023PF01790 - 10 - emitted laser beam with a modulation unit, processing the modulated laser beam with an event-based pipeline for SMI detection to detect events indicative of changes in the SMI signal, capturing images of the scene in response to the detected events using a neuromorphic sensor, and analyzing the detected events and the captured images with a processor to determine characteristics of the scene. What has been said with respect to the device may analogously be applied to the method and therefore need not be repeated there. Method embodiments and details have a counterpart in the device and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS In the following, the invention will be described in further detail with reference to the accompanying drawings, wherein: FIG. 1 depicts an embodiment of the sensing system in a sectional view with the neuromorphic sensor and the laser SMI illuminator on one ASIC and one substrate, FIG. 2 depicts an embodiment of the sensing system in a sectional view with the neuromorphic sensor and the laser SMI illuminator on one ASIC and one substrate, with a reference neuromorphic sensor, FIG. 3 depicts an embodiment of the sensing system in a sectional view with the neuromorphic sensor and the laser SMI illuminator on separate ASICs and one substrate, 2023PF01790 - 11 - FIG. 4 depicts an embodiment of the sensing system in a sectional view with the neuromorphic sensor and the laser SMI illuminator one substrate, while the laser SMI illuminator is arranged directly on the substrate, FIG. 5 depicts an embodiment of the sensing system in a sectional view with the neuromorphic sensor and the laser SMI illuminator on separate ASICs and one substrate having each a cover around it, FIG. 6 depicts an embodiment of the sensing system in a sectional view with the neuromorphic sensor and the laser SMI illuminator on separate ASICs and substrates having each a cover around it. Identical or likewise parts are labelled by the same reference signs throughout the drawings. DETAILED DESCRIPTION In FIG. 1 an embodiment of the present invention is shown. In this embodiment, a sensing system 1 for sensing distance and speed is depicted comprising a laser SMI (self-mixing interferometry) illuminator 2, which is configured to emit a laser beam onto a scene, and a neuromorphic sensor 4, which is configured to capture images of the scene in response to the detected events. The neuromorphic sensor 4 is in this embodiment an event- based camera but can be any other dynamic vision sensor suitable for this embodiment. 2023PF01790 - 12 - With regard to the laser SMI illuminator 2, this can be a VCSEL (Vertical Cavity Surface Emitting Laser) or an EEL (Edge Emitting Laser) or the like. The laser beam, emitted by the laser SMI illuminator 2 is modulated by a modulation unit (not depicted in the figures) configured to modulate the emitted laser beam. The modulation can be a FMCW (Frequency-Modulated Continuous- Wave) modulation, or any modulation / frequency sweep suitable for the needs of the application. Said events, indicative of changes in the SMI signal, are detected by an event-based pipeline (not depicted in the figures) for SMI detection while it processes said modulated laser beam. The event-based pipeline is comprised in or coupled to the event-based camera 4. Further on, the detected events are processed by a processor (not depicted in the figures), which is configured to analyze the detected events and the captured images to determine characteristics of the scene. The processor is in this embodiment coupled to the event-based pipeline and the event- based camera 4. In this embodiment, the VCSEL 2 and the event-based camera 4 are arranged on the same ASIC 6 and on the same substrate 8, which is in this embodiment a printed circuit board (PCB). The ASIC 6 and the PCB 8 are connected via connection elements 16. These connection elements 16 are in this embodiment bumps. The bumps 16 are formed by bumping, which involves forming raised regions of metal over bonding pads to 2023PF01790 - 13 - allow the simultaneous bonding of the bumps 16 to the PCB 8. This process can lead to many advantages such as lower inductance, better electrical performance, higher current carrying capacity, a potentially much smaller footprint, and lower cost of the sensing system 1. However, the choice of the connection element for connecting ASIC 6 and PCB 8 depends on the specific requirements of the embodiment of the sensing system 1 and can be chosen according to these. The ASIC 6 is arranged in that way above the PCB 8, whereas the event-based camera 4 is arranged on top of the ASIC 6 on one end of the ASIC 6. Also, on top of said ASIC 6, yet on another end of it, the VCSEL 2 is arranged. The VCSEL 2 is connected via connections 18 to the ASIC 6. In this embodiment, the connections 18 are made by wire-bonding. A cover 14 is arranged over the assembly of event-based camera 4, VCSEL 2 and the ASIC 6 on the PCB 8, so that it covers the event-based camera 4, the VCSEL 2 and the ASIC 6. The cover 14 has two passages 12 for letting the laser light pass and a bar 10 is placed underneath it, extending between the cover 14 and the ASIC 6. The bar 10 also separates the ASIC 6 in two parts, on one of it the event-based camera 4 is arranged and on the other one the VCSEL 2 is arranged. The bar 10 forms further essentially two chambers together with the cover 14 and the PCB 8. The bar 10 further inhibits crosstalk that may occur between said two chambers. To eliminate the need for extra and / or external calibration, the sensing system 1 comprises in this embodiment a plurality 2023PF01790 - 14 - of VCSELs 2 (only one VCSEL 2 is depicted in the figures, though). Each one or at least two of the VCSELs has / have a different carrier frequency. With this, the problem of identifying the dots with parallax or in adversary systems is solved, as the sensing system will look for these preestablished carrier frequencies on the image. Therefore, no calibration for the dot position is thus needed. Also, a frequency comparison and flagging block (not depicted in the figures) is further comprised in this embodiment. A frequency comparison and flagging block further enables the detection of very close-range targets, which is not possible with other systems and helps to render the system immune to electromagnetic interference or noise. Furthermore, since the sensing systems transmits continuously, the frequency and comparison block simplifies the process of information extraction. Furthermore, the sensing system 1 may comprise a reference neuromorphic sensor 20, as it can be seen in FIG. 2. In this embodiment, the reference neuromorphic sensor 20 is an event- based camera. This helps to further simplify the detection of the pixels on the scene, which can be linked spatially and in a time domain. It also reduces the power consumption. Together with a frequency correlation block the events can be located more easily (at the edge) and a flagging of the events related to the illumination compared to those related to the scene can be done. This reference event-based camera 20 is able to do analogue demodulation. Without this reference event-based camera 20, the sensing system 1 can only do digital demodulation. 2023PF01790 - 15 - In another embodiment, which is depicted in FIG. 3, the bar 10 extends from the cover 14 to the PCB 8, so that two separate ASICs 6a and 6b are to be used. Again, on one of it the event-based camera 4 is arranged and on the other one the VCSEL 2 is arranged. The bar 10 forms further two chambers together with the cover 14 and the PCB 8. It is also possible to omit the ASIC in the chamber of the VCSEL 2, so that the VCSEL 2 is arranged on the PCB 8 without said ASIC 6, as it can be seen in FIG. 4. Another embodiment constituting another possibility to arrange the sensing system 1, is depicted in FIG. 5. As it can be seen in FIG. 5, two covers 14a and 14b are used instead of one cover as in FIG. 4 to be placed on one PCB 8. In between cover 14a and the PCB 8, the event-based camera 4 is arranged on ASIC 6a, whereas in between cover 14b and the PCB 8, the VCSEL 2 is arranged on ASIC 6b. Yet, if said one PCB of FIG. 5 is exchanged for two PCBs 8a and 8b, one under each cover 14, the embodiment of FIG. 6 is met. Here are two separate assemblies 22 and 24 forming the sensing system 1. The assembly 22 comprises the event-based camera 4 on ASIC 6a on PCB 8a covered by cover 14a, whereas assembly 24 comprises the VCSEL 2 on ASIC 6b on PCB 8b covered by cover 14b. Signals are the shared between those two assemblies for sensing distance and speed. A method for sensing distance and speed involves several steps. First a laser beam is emitted onto the scene using at least one VCSEL 2. Then the emitted laser beam is modulated with a FMCW (Frequency-Modulated Continuous-Wave) modulation unit. 2023PF01790 - 16 - Furthermore, the so modulated laser beam is processed with an event-based pipeline for SMI (self-mixing interferometry) detection to detect events indicative of changes in the SMI signal. The event-based camera 4 is then used to capture images of the scene in response to the detected events. Furthermore, the detected events and the captured images are analyzed with a processor to determine characteristics of the scene, such as speed and distance of objects in the scene. The steps described need not to be in a certain order or in the order described above. Some can be done before others, or vice versa, depending on the needs of the application.

[0002] 2023PF01790 - 17 - List of abbreviations: Frequency-Modulated FMCW Continuous-Wave Self-mixing interferometry SMI Application-Specific ASIC Integrated Circuit Vertical Cavity Surface VCSEL Emitting Laser Edge Emitting Laser EEL Light Detection and RangingLIDARTime-of-FlightToFPrinted circuit boardPCBAnalog-to-Digital ConverterADC

[0003] 2023PF01790 - 18 - LIST OF REFERENCE SIGNS Sensing system 1 Laser SMI illuminator 2 Neuromorphic sensor 4 ASIC 6 ASIC 6a ASIC 6b Substrate 8 Substrate 8a Substrate 8b Bar 10 Passage 12 Cover 14 Cover 14a Cover 14b Connection element 16 Connection 18 Reference neuromorphic sensor 20 Assembly 22 Assembly 24

Claims

2023PF01790 - 19 - CLAIMS 1. A sensing system (1) for sensing distance and speed comprising: ^ at least one laser SMI (self-mixing interferometry) illuminator (2) configured to emit a laser beam onto a scene, ^ a modulation unit configured to modulate the emitted laser beam,   ^ an event-based pipeline for SMI detection configured to process the modulated laser beam to detect events indicative of changes in the SMI signal,   ^ a neuromorphic sensor (4) configured to capture images of the scene in response to the detected events, and ^ a processor configured to analyze the detected events and the captured images to determine characteristics of the scene.

2. A sensing system (1) according to claim 1, wherein the modulation is a FMCW (Frequency-Modulated Continuous-Wave) modulation.

3. A sensing system (1) according to claim 1 or 2, wherein the neuromorphic sensor (4) is an event-based camera.

4. A sensing system (1) according to any of the preceding claims,2023PF01790 - 20 - wherein the at least one laser SMI illuminator is a VCSEL (Vertical Cavity Surface Emitting Laser).

5. A sensing system (1) according to any of claims 1 to 3, wherein the at least one laser SMI illuminator is an EEL (Edge Emitting Laser).

6. A sensing system (1) according to any of the preceding claims, comprising a plurality of laser SMI illuminators (2), wherein at least two laser SMI illuminators (2) out of the plurality of laser SMI illuminators (2) have a different carrier frequency.

7. A sensing system (1) according to any of the preceding claims, further comprising a reference neuromorphic sensor (20).

8. A sensing system (1) according to claim 7, wherein the reference neuromorphic sensor (20) is an event-based camera.

9. A sensing system (1) according to any of the preceding claims, further comprising a frequency comparison and flagging block.

10. A sensing system (1) according to any of the preceding claims, wherein the at least one laser SMI illuminator (2) or the plurality of laser SMI illuminators (2) and the neuromorphic sensor (4) are arranged on the same ASIC (6) (Application-Specific Integrated Circuit).2023PF01790 - 21 - 11. A sensing system (1) according to any of claims 1 to 9, wherein the at least one laser SMI illuminator (2) or the plurality of laser SMI illuminators (2) and the neuromorphic sensor (4) are arranged on different ASICs (6a, 6b).

12. A sensing system (1) according to any of the preceding claims, wherein the at least one laser SMI illuminator (2) or the plurality of laser SMI illuminators (2) and the neuromorphic sensor (4) are arranged on the same substrate (8).

13. A sensing system (1) according to any of claims 1 to 9 and 11, wherein the at least one laser SMI illuminator (2) or the plurality of laser SMI illuminators (2) and the neuromorphic sensor (4) are arranged on different substrates (8a, 8b).

14. A method for sensing distance and speed, the method comprising the following steps: ^ emitting a laser beam onto the scene using at least one laser SMI illuminator (2), ^ modulating the emitted laser beam with a modulation unit,   ^ processing the modulated laser beam with an event- based pipeline for SMI detection to detect events indicative of changes in the SMI signal,  2023PF01790 - 22 - ^ capturing images of the scene in response to the detected events using a neuromorphic sensor (4), and  ^ analyzing the detected events and the captured images with a processor to determine characteristics of the scene.

Images