An image sensor, a computer vision device, a method and a computer program for capturing event-based image data for computer vision

Abstract

Examples relate to an image sensor, a computer vision device, a method and a computer program for capturing event-based image data for computer vision. The image sensor is configured to capture event-based image data. The image sensor is configured to capture event data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum.

Description

[0001] An Image Sensor, a Computer Vision Device, a Method and a Computer Program for Capturing Event-Based Image Data for Computer Vision

[0002] Field

[0003] Examples relate to an image sensor, a computer vision device, a method and a computer program for capturing event-based image data for computer vision, more specifically, but not exclusively, to a concept for capturing multispectral event-based image data also in non-visual bands.

[0004] Background

[0005] Event-based image sensors are a type of camera technology that provides a new approach to image acquisition. Instead of capturing images at regular intervals, event-based sensors capture images only when there is movement or a significant change in the scene. This approach offers several benefits over traditional cameras, such as higher temporal resolution and lower power consumption.

[0006] One of the significant features of event-based sensors is their high temporal resolution. Instead of capturing images at fixed intervals, event-based sensors capture information precisely when changes occur. For real-time applications, such as motion tracking, this means that event-based sensors can capture and process data with much higher accuracy and speed than traditional cameras. Additionally, event-based sensors can be used in environments with low lighting conditions, making them ideal for use in applications such as surveillance and robotics. Another key feature of event-based sensors is their low power consumption. Because traditional cameras capture a large volume of data, they require a significant amount of processing power to compress, store, and transmit that information. In contrast, event-based sensors only capture information when changes occur, drastically reducing the amount of data that needs to be processed. This can lead to significant energy savings, particularly in battery- powered devices such as mobile phones and drones.

[0007] In conclusion, event-based image sensors provide a new approach to image acquisition, offering higher temporal resolution and lower power consumption. This technology is particularly well-suited for real-time applications such as motion tracking and robotics, as well as situations requiring low-light sensitivity. As the need for high-quality, real-time image processing continues to grow, event-based sensors may become a tool in many different industries and applications.

[0008] Summary

[0009] Examples are based on the finding that multispectral event-based vision sensors may be used for better sensing 3-dimensional (3D) scenes and for efficiently solving computer vision problems. On the one hand, event-based sensors (EVS) allow fast, low power, and high dynamic range imaging at high temporal resolution. On the other hand, multispectral sensors (MS) capture more information from the scene reflectance by combining different spectral bands in visible or near-infrared, for example. Particularly events outside the visible band contribute to the efficiency of computer vision. The combination of both, event-based sensing and multispectral sensing, particularly also outside the visible band, was found to improve computer vision. Improvements when considering multiple spectral bands showed to be higher for event data, as more information is lost with event-based sensing with respect to conventional imaging if contrast in the scene is not sufficient. By considering multiple bands, information becomes more selective.

[0010] Examples provide an image sensor configured to capture event-based image data. The image sensor is configured to capture event data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum. The additional event data in the band outside the visible spectrum may enhance computer vision.

[0011] For example, the at least one band has a center frequency above the frequencies of the visible spectrum. The at least one band may be in the ultraviolet light band or at even higher frequencies, shorter wavelengths, respectively. Events in these bands may comprise valuable information on objects, their movements etc. In further examples the at least one band may have a center frequency below the frequencies of the visible spectrum. The at least one band may be in the infra-red or near infra-red light band or at even lower frequencies, longer wavelengths, respectively. Likewise, event-information about these bands may provide further or even key information about a scene for computer vision. Furthermore, at least in some examples, the at least one band may have a center frequency above the frequencies of the visible spectrum and another one of the plurality of spectral bands may have a center frequency below the frequencies of the visible spectrum. In some scenes or for some objects, spectral information outside the visible spectrum on both sides may be considered and provide valuable additional information.

[0012] The image sensor may comprise a plurality of spectral filters configured to filter the spectrum to obtain spectral bands. Such a filter may be of any kind and may provide efficient means for separating the spectrum in the plurality of spectral bands. For example, the image sensor may further comprise a plurality of event-based sensor elements, which are coupled to the plurality of spectral filters and which are configured to capture the event-based data in the plurality of spectral bands. For example, the image sensor may comprise pixelwise sensor elements (EVS pixels), which are stacked with different spectral filters, similar to techniques applied to CMOS (Complementary Metal-Oxide-Semiconductor) sensors. Examples may provide an efficient implementation for event-based sensing in different spectral bands.

[0013] In examples, the plurality of spectral bands may comprise two or more sub-bands in a visible spectrum. Subdividing the visible spectrum in multiple sub-bands may enable more precise band-specific event detection, which may contribute to detection precision. For example, the two or more sub-bands in the visible spectrum may have the same bandwidths, which enables efficient spectral filtering.

[0014] Examples also provide a computer vision device comprising one or more image sensors according to the present description. The computer vision device may further comprise a neural network with two or more processing layers. For example, one processing layer of the neural network is configured to combine event-based image data from the plurality of spectral bands to obtain combined data. Additionally, another processing layer of the neural network may be configured to process the combined data. Examples may hence provide enhanced computer vision by combining event-based spectral imaging and machine learning.

[0015] Examples also provide a method for capturing event-based image data for computer vision. The method comprises capturing event-based image data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum. The method may further comprise combining the event-based image data of the plurality of spectral bands using a processing layer of a neural network. Yet another example is a computer program having a program code for performing the method described herein, when the computer program is executed on a computer, a processor, or a programmable hardware component.

[0016] Brief description of the Figures

[0017] Some examples of apparatuses and / or methods will be described in the following by way of example only, and with reference to the accompanying figures, in which

[0018] Fig. 1 shows a block diagram of an example of an image sensor and an example of a computer vision device;

[0019] Fig. 2 shows a block diagram of an example of a method for capturing event-based image data;

[0020] Fig. 3 illustrates spectral event-based data for different spectral bands;

[0021] Fig. 4 illustrates how face contrast on an image changes, depending on the spectral bands considered (e.g. Infrared (IR) or Near-Infrared (NIR));

[0022] Fig. 5 depicts improvements by considering multispectral EVS for a face detection example;

[0023] Fig. 6 illustrates some spectral bands in examples; and

[0024] Fig. 7 shows simulations results in a face tracking example.

[0025] Detailed Description

[0026] Some examples are now described in more detail with reference to the enclosed figures. However, other possible examples are not limited to the features of these examples described in detail. Other examples may include modifications of the features as well as equivalents and alternatives to the features. Furthermore, the terminology used herein to describe certain examples should not be restrictive of further possible examples. Throughout the description of the figures same or similar reference numerals refer to same or similar elements and / or features, which may be identical or implemented in a modified form while providing the same or a similar function. The thickness of lines, layers and / or areas in the figures may also be exaggerated for clarification.

[0027] When two elements A and B are combined using an “or”, this is to be understood as disclosing all possible combinations, i.e. only A, only B as well as A and B, unless expressly defined otherwise in the individual case. As an alternative wording for the same combinations, "at least one of A and B" or "A and / or B" may be used. This applies equivalently to combinations of more than two elements.

[0028] If a singular form, such as “a”, “an” and “the” is used and the use of only a single element is not defined as mandatory either explicitly or implicitly, further examples may also use several elements to implement the same function. If a function is described below as implemented using multiple elements, further examples may implement the same function using a single element or a single processing entity. It is further understood that the terms "include", "including", "comprise" and / or "comprising", when used, describe the presence of the specified features, integers, steps, operations, processes, elements, components and / or a group thereof, but do not exclude the presence or addition of one or more other features, integers, steps, operations, processes, elements, components and / or a group thereof.

[0029] Fig. 1 shows a block diagram of an example of an image sensor 10 and an example of a computer vision device 100. The image sensor 10 is configured to capture event-based image data. The image sensor 10 is configured to capture event data in a plurality of spectral bands., wherein at least one of the spectral bands lies outside a visible spectrum. In the example shown in Fig. 1 the image sensor 10 monitors a scene and provides event-based data per spectral band. For example, the at least one band has a center frequency above the frequencies of the visible spectrum, e.g. the at least one band is in the ultraviolet light band. Additionally or alternatively, the at least one band has a center frequency below the frequencies of the visible spectrum, e.g. the at least one band is in the infra-red or near infra-red light band. In some examples both is also possible: The at least one band has a center frequency above the frequencies of the visible spectrum and another one of the plurality of spectral bands has a center frequency below the frequencies of the visible spectrum. In Fig.l optional components are shown in broken lines. For example, the image sensor 10 further comprises a plurality of spectral filters 12, which are configured to filter the spectrum to obtain the spectral bands.

[0030] Here and in the following spectral filters can be optical components that allow only certain wavelengths of light to pass through them, while blocking others. Spectral filters separate specific wavelengths or bandwidths of light or radiation from a broader spectrum, enabling accurate and detailed measurements. They are available in a wide range of materials, sizes, and spectral properties to suit different applications.

[0031] There are several types of spectral filters available, each designed for a specific range of wavelengths. These include bandpass filters, long-pass filters, short-pass filters, and notch filters, among others. Bandpass filters allow a specific range of wavelengths to pass through them while blocking all other wavelengths. Long-pass filters allow longer wavelengths to pass through them, while blocking shorter wavelengths. Short-pass filters, on the other hand, allow short wavelengths to pass through, while blocking longer wavelengths. In summary, any spectral filters may be used in examples.

[0032] As further shown as optional components in Fig.l, the sensor 10 may comprise event-based sensor elements 14, which may correspond to EVS pixel sensors. For example, the image sensor 10 may further comprise a plurality of event-based sensor elements 14, which are coupled to the plurality of spectral filters 12 and which are configured to capture the event-based data in the plurality of spectral bands.

[0033] Examples make use of one or more event sensors. An event camera is an imaging sensor that responds to local changes, e.g. in brightness. Event cameras do not capture images using a shutter as conventional (frame) cameras do. Instead, each pixel inside an event camera operates independently and asynchronously, reporting changes in brightness as they occur, and staying silent otherwise. Event cameras have been seen as the next generation for computer vision tasks, because of their high speed, high dynamic range as well as the efficiency of data acquisition. Event cameras are used to capture the movement in the scene, and static objects will not activate the event sensor. This may allow for lower latencies, lower power consumption and lower data processing requirements as compared to frame-based sensors, which capture information on a full frame in each sampling step. An event-based sensor may have a higher dynamic range than framebased sensors, even if they use high-speed vision (high frame rates). For example, the optical event data has a sampling rate of at least 500 frames per second. Using event-based sensors, events may be recorded that would otherwise require conventional cameras to run at 10.000 images / second and more. Event-based sensors may allow latencies at 40-200ps at high bandwidths, e.g. 60 Meps (Megaevents per second) and more. Moreover, even 3D capturing may be enabled by event-based sensors. The capturing in examples may comprise capturing optical event data in 3D volume, comprising a time component, which may form a data basis for very efficient and precise movement detection.

[0034] Fig. 1 further illustrates a computer vision device 100 comprising one or more image sensors 10. For example, the computer vision device comprises a processing apparatus 20 with processing circuitry 22, which is coupled to one or more interfaces 24, which in turn receive the event-based data from the sensor 10. E.g. the event-based data may at least distinguish data from the visible and from the un-visible spectrum. However, in examples a higher number of bands is conceivable as will be laid out in more detail subsequently.

[0035] For example, the computer vision device may be integrated in a robot (industrial or home), a vehicle (car, bus truck, train, plane, etc.), an entertainment system (home, inflight, in-vehicle, etc.), or a smart home system (light control, temperature control etc.).

[0036] The circuitry 22 may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. In other words, the described function of the circuitry 22 may as well be implemented in software, which is then executed on one or more programmable hardware components. Such hardware components may comprise a general-purpose processor, a Digital Signal Processor (DSP), a Graphics Processing Unit (GPU), a micro-controller, etc. A further example is a device or entity 200, e.g. an automated system, e.g. for entertainment, gaming, exercising, vehicle, comprising an example of the apparatus 20. As further illustrated in Fig. 2, the apparatus 20 may optionally (shown in broken lines) comprise one or more interfaces 24, which are coupled to the respective circuitry 22. The one or more interfaces 24 may correspond to one or more inputs and / or outputs for receiving and / or transmitting information, which may be analog or in digital (bit) values (e.g. using converter technology) according to a specified code, within a module, between modules or between modules of different entities. For example, an interface 24 may comprise interface circuitry configured to receive and / or transmit information. In examples an interface 24 may correspond to any means for obtaining, receiving, transmitting or providing analog or digital signals or information, e.g. any connector, contact, pin, register, input port, output port, conductor, lane, etc., which allows providing or obtaining a signal or information, particularly the event-based data from one or more image sensors 10.

[0037] Fig. 2 shows a block diagram of an example of a method 30 for capturing event-based image data. The method 30 for capturing event-based image data for computer vision comprises capturing 34 event-based image data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum. As further shown in Fig. 2 the method 30 may comprise the optional step of filtering 32 the spectrum to obtain a plurality of spectral bands.

[0038] In examples, the sensor 10 may be applied to a computer vision problem by tackling the face detection problem using deep learning, as an example, based on simulation of this sensor 10 and a simpler real setup combining only two spectral bands. Trials show that early fusion of multispectral events significantly improves the face detection performance, greater than it does for early fusion of conventional multispectral images. In other examples other fusion techniques may be use (e.g. in a middle or output layer of a neural network).

[0039] In examples the computer vision device 100 may further comprise a neural network with two or more processing layers, wherein one processing layer of the neural network is configured to combine event-based image data from the plurality of spectral bands to obtain combined data. Another processing layer of the neural network may be configured to process the combined data.

[0040] Another example is a computer program having a program code for performing one of the methods described herein, when the computer program is executed on a computer, a processor, or a programmable hardware component. The computer program or the method 30 may further comprise combining the event-based image data of the plurality of spectral bands using a processing layer of a neural network.

[0041] Event-based vision is a new imaging modality that offers low latency, low power and high temporal resolution sensing. These properties make it a sensor of choice for edge applications on dynamic devices or in high dynamic range environments. While attracting a lot of attention, not many event-based cameras are on the market yet and sensors are monochromatic (grayscale), capturing light over all the visible spectrum at once. EVS cameras suffer in a situation, where there is low contrast in the scene, where no or very few events would be generated. The thresholding operation for generating events further reduces the data obtained in case the signal (in this case the different of brightness) is too low. In the worst case, the sensor would not sense anything but noise.

[0042] This may be suboptimal compared to a system that would perceive higher contrast in the scene, which in turn would generate more data and improve the signal to noise ratio. It is done for conventional imaging by considering spectral bands, e.g. multispectral or hyperspectral sensing, to capture enough information or the information that matters. However, so far event sensing is monochromatic (or at most RGB(red / Green / Blue). Examples may provide a system, which applies the advantages brought by multispectral sensing to event sensors, where it makes even more sense due to the nature of event data, as the data can be sparse and not sensitive enough (depends heavily on the contrast in the scene and the parameters of the sensor).

[0043] Examples provide a multispectral event-based vision sensor for better sensing the world and efficiently solve computer vision problems. On one hand event-based sensors (EVS) allow fast, low power, and high dynamic range imaging at high temporal resolution. On the other hand, multispectral sensors (MS) capture more information from the scene reflectance by combining different spectral bands in visible or near-infrared spectra, for example. Examples may combine both modalities.

[0044] Examples may use an EVS sensor with multiple channels. It could be the visible and non- visible, e.g. near-infrared spectral regions but not limited to it. Such as sensor could be realized rather easily with existing techniques applied to CMOS sensors, by stacking spectral filters (of any kind, as done for RGB or multispectral imaging, e.g. Bayer or plasmonic filters) on top of EVS pixels (DAVIS (Dynamic Active-Pixel Vision Sensor) pixels, ATIS (Asynchronous Time-based Image Sensor) pixels, etc.) Such a sensor enables capturing multispec- tral events, specifically including near infrared data (or anything else than only the three RGB visible bands).

[0045] In some examples a sensor would generate different events for the different bands, depending on the motion and contrast in the scene, therefore capturing more information than a conventional EVS camera with a single channel or RGB (3 bands in visible range only). Some channels might not generate events, but other channels would take over to complete the information.

[0046] Fig. 3 illustrates spectral event-based data for different spectral bands. Fig. 3 shows a chart with wavelength to the right and intensity to the top. Along the abscissa center frequencies for B / G / R and for near-infra-red (NIR) are indicated. At the top of Fig. 3, a grayscale (GS) perceived edge is illustrated. For GS, the intensity difference between the object intensity and the background intensity are evaluated. If the overall (accumulated over the visible band) intensity difference is above a threshold, an event is detected. Therefore, intensity differences over the spectrum may superimpose constructively or destructively. This is illustrated for the B and R band in Fig. 3. While the intensity difference (Intensity Object - Intensity Background) is negative for B, it is positive for R (and about the same for G). Therewith, when evaluating GS (=sum of BGR) the intensity is below a trigger threshold an no event is detected. However, when distinguishing the bands (multi spectral perceived edge shown underneath the abscissa) then events are triggered for B and R separately. As further exemplified in Fig. 3 if the NIR is also considered, there is an extra event as the intensity difference is largest in the NIR band.

[0047] The sensor could be applied to a computer vision problem, using a full system based on deep learning. This potential is explored compared to conventional multispectral imaging for the face detection task. This task is chosen as the contrast of a face on an image is greatly modified when considering other spectral bands such as near infrared (due to skin reflectance).

[0048] While conventional cameras capture a series of frames at a fixed rate, event cameras capture asynchronous and continuous streams of events. Each event is generated independently, at the pixel level, when the pixel intensity changes by a certain threshold value, resulting in a high temporal resolution (microsecond) and a very low power consumption. Some other properties of the event-cameras are the absence of motion blur and a high dynamic range: it can operate well in very bright and very dark environments at the same time. This makes event cameras particularly suitable for high-speed and low-latency applications such as robotics, autonomous vehicles, and virtual / augmented reality.

[0049] Despite their advantages, event cameras have been mostly limited to monochromatic vision, which limits the range of information they can capture. Indeed, existing event-based cameras are primarily based on grayscale intensity changes. Imagine a scenario like in Fig. 3. Different objects have a different pixel intensity, depending on the scene illumination, the camera sensitivity and more importantly the object’s reflectance spectrum. The object and background’s intensity across wavelengths is shown in Fig. 3 for a single pixel. In this scene, a regular monochromatic grayscale event-camera (on top) would not perceive any edge (intensity change) because the intensity both increases and decreases in the selected range, averaging to a null intensity change. Therefore, no events would be generated and the object can’t be detected. At the bottom, a multispectral camera outputs events only for narrow bands in the visible (say Blue, Green, Red). From the different channels, intensity differences can now be perceived, in particular for the B and R channel, and events will be generated. Moreover, grayscale event-cameras would miss out on information beyond the visible spectrum: if the object and background have very different reflectance in the near infrared (such as plants), it would be a relevant band to use for this application. The situation described in the Fig. 3 is only illustrative and a bit extreme: very often the visible band (grayscale) is discriminative enough to work with. However, it still shows that more information can be obtained out of these smaller bands, especially for event-based cameras that heavily rely on intensity contrast.

[0050] Fig. 4 illustrates how face contrast on an image changes, depending on the spectral bands considered (e.g. Infrared (IR) or Near-Infrared (NIR)). Fig. 4 shows RGB bands (converted to GS) of a face image on the left and the corresponding NIR band on the left. An off-the- shelf face detection neural network was trained and finetuned on simulated multispectral events (9 channels in visible and near infrared range). The events from the different bands are combined in the first layer of the deep neural network (early fusion). In this study, it was found that early fusion of multispectral events significantly improves the face detection performance, greater than it does for early fusion of conventional multispectral images. This result shows that polychromatic events carry relatively more information about the scene than conventional multispectral / color images do, with respect to their monochromatic equivalent.

[0051] Fig. 5 depicts improvements by considering multispectral EVS for a face detection example. Fig. 5 shows a mean average precision of the face detection mAP. The higher the mAP the better or the more precise is the face detection. The mAP is depicted by pairs of bars, where the bar on the left of a pair shows the mAP for APS (Active Pixel Sensor) and the bar on the right of a pair shows the mAP for EVS. While mAP is shown on the ordinate, the abscissa shows different spectral band configurations. The spectral resolution increases from left to right, from GS to 9 channels or sub-bands in the visible spectrum plus IR. It can be seen that the mAP increases in all cases when the IR band is added. This is indicated in Fig. 5 by the sloped bars connecting adjacent bar diagrams. However, increases are higher for the EVS than for the APS. Fig.5 shows improvements by considering multispectral EVS for the face detection experiment. Using multiple channels (going towards the right the number of channels increases) improves the accuracy, in particular using near infrared data (IR).

[0052] The more selective or differentiated the spectral information the better the performance. Such a system would generate higher contrast in the scene, which in turn would generate more data and improve the signal-to-noise ratio. More information would be captured from the scene in case there is no contrast in the visible range but only in the near infrared range, for example.

[0053] In the following, some examples will be formalized, and a few experiments will be setup to compare their performance against regular grayscale events and conventional images for face detection. Face detection is an important task in computer vision and has a wide range of applications, such as surveillance, biometrics, human-robot interaction, etc. Also, the human skin spectrum is well studied and it has interesting properties beyond the visible spectrum that generalize to any skin type. Event-cameras have a lot of potential in handheld devices, mobile robots or in surveillance and monitoring applications, often interacting with humans. For these reasons, face detection is chosen as use-case task but observations and conclusions may generalize to other tasks.

[0054] In the following three bimodal datasets are proposed with polychromatic events, from simulation, labeled for face detection. Two of them are large-scale color and grayscale datasets; the third is smaller but features data over 10 spectral bands, from blue to infrared. Here, bimodal means temporally and spatially aligned conventional images (APS) and simulated event-based data (EVS), 1 : 1.

[0055] A fair comparison of APS and EVS face-detection will be provided, quickly comparing the robustness of events over conventional images when both use the same neural architecture and have been trained on the same data. This differs from usual approaches where APS and EVS models are either trained on different imbalanced data or using hardly comparable algorithms. Multispectral events will be used, capturing events over multiple spectral bands, and benefits are explored they could have over monochromatic grayscale events. Event-based performance is significantly improved by the use of polychromatic events, beyond the effects observed with conventional multispectral images.

[0056] While conventional face detection can be considered a solved problem, event-based face detection is in very early stages of research. It has a lot of potential for always-on applications thanks to the low latency and low power consumption properties of event cameras, but it is a new problem to solve due to the change in data paradigm (frames to events). Two approaches can be distinguished: detection over reconstructed images or over events. While detection over reconstructed images is convenient and enables the use of well-known face detectors, it negates most advantages of event cameras - although the reconstructed image has a higher dynamic range. Equivalently to image-based face detection, over the time, research shifted from the use of hand-crafted features to deep learning approaches. Object detection architectures can be applied to 2D event representations, mostly encoding temporal information, too. The fact that events intrinsically only account for changes in the scene can be used and memory can be added to an architecture to remember static objects after they stopped moving. These recurrent networks significantly improve the mAP on the IMPx data set, but LSTM- variants (Long Short-Term Memory) are computationally expensive and slower in inference while visual transformers need a huge amount of training data and are slow to train. Examples may explore the effects of multispectral events for face detection. Training recurrent neural networks can be tricky and dealing with the huge amount of data RVTs (Recurrent Vision Transformer) need is impractical. Therefore, in the following a non-recurrent CNN (Convolutional Neural Network) architecture will be used, known for face detection, to perform experiments: RetinaFace. As the same network architecture and training data for both APS and EVS face detection, it can be argued that this approach is valid and results should generalize to recurrent neural networks.

[0057] Color / Multispectral face detection: Most concepts involving multispectral data and faces are focused on two areas, either face recognition using multispectral biometrics or face detection in thermal infrared.

[0058] Furthermore, in the following early fusion of multispectral data (images / events) will be explored, i.e. by stacking the different channels in a single input tensor. Another approach would have been to perform late fusion, i.e. compute features on each hyperspectral channel and fuse them in a deeper layer. The late fusion approach is quite common in the multispectral deep learning research and effective ways have been found to fuse features. Late fusion of RGB and NIR features in a classification neural network significantly improves the performance over RGB alone.

[0059] The light intensity received by a pixel, I(x, y), follows equation (1), where L(k) is the scene illumination, R(k) is the object reflectance and S(k) is the pixel sensitivity, at a particular wavelength . An event (x, y, t, p) is generated whenever log(I(x, y)) increases or decreases by a constant C.

[0060] Most commercial event cameras are monochromatic, this means their pixel sensitivity S(k) spans across all visible wavelengths. One approach is to play with the spectral sensitivity of the sensor to create multispectral events. For example, if an ideal long-pass filter is place in front of the event-camera with a cut-off frequency c, the new intensity equation becomes: because S(k) would be null for < . This concept can be generalized to any kind of filters and the sensitivity of the sensor can be shaped accordingly. As can be seen in conventional imaging, multispectral (MS) images can look very different from grayscale (GS) and this difference in intensity contrast certainly affects the count and distribution of events in a scene. Note that the narrower the band is, the less light intensity the sensor will capture, as a part of the Quantum Efficiency curve is simply truncated and therefore the final integral (eq. 2) is less or equal to the initial integral (eq. 1). While low light is an issue for regular cameras, the high dynamic range property of event-cameras makes them especially suitable for multispec- tral imaging.

[0061] One issue when working with event-based data is the lack of available datasets, particularly when trying to explore multispectral events while no multispectral event-based camera exists. Moreover, as a learning-based approach is used for face detection, a large-scale dataset is needed, with enough samples, diversity and challenging poses. Finally, for a fair comparison of an example event-based face detector with an image-based one, both should be trained on a similar domain. A scalable way of building a large event-based face detection dataset is to use simulated events, e.g. using open-source event simulation software.

[0062] • To train from scratch, open-source datasets with color videos (RGB) have been selected.

[0063] • For finetuning, own datasets have been captured with two multispectral cameras.

[0064] • To validate with simulated events, a few sequences have been captured with real events (grayscale and infrared).

[0065] The training datasets are

[0066] N-YoutubeFaces, cf. Lior Wolf, Tai Hassner, and Itay Maoz, Face recognition in unconstrained videos with matched background similarity, in CVPR 2011, pages 529-534, 2011, and

[0067] N-MobiFace, cf. Yiming Lin, Shiyang Cheng, Jie Shen, and Maja Pantic, Mobiface: A novel dataset for mobile face tracking in the wild, 2019, following the renaming convention set for neuromorphic data. YoutubeFaces consists of 3.4k very short color videos of 1.6k different people. YoutubeFaces perfectly suits the need of a diverse dataset for face detection in the wild. At the same time, it is not too challenging as the sequences mainly come from movies or tv studios broadcast, where the image standards are high.

[0068] On the other hand, MobiFace consists of 80 unedited mobile live color video recordings (95k frames) by smartphone users. It is challenging by design but all these challenges (fast motion, camera rotations, motion blur, scale and illumination variations, etc.) are situations where event cameras excel. However, because N-MobiFace is made of simulated events from conventional images, if the source image has motion blur, the simulated events will also suffer from it. These sequences have not been filtered out as it improved the generalization capabilities of the face detector. After conversion to events, both datasets were automatically labelled to complete for the missing ground truths, using existing face detection networks and NonMaximum Suppression over all predictions.

[0069] The following examples use deep-learning based face detection. Experiments with the blue, green and red channels of N-MobiFace and N-YoutubeFaces are a good start but not enough to draw conclusions and it does not help with exploring infrared bands. Therefore, there is a need specifically for multispectral data, enough to finetune the baseline model and to carry the multispectral experiments. Therefore, N-SpectralFace captures 69 sequences around the office in different places, with different faces and lighting conditions. Each sequence is recorded simultaneously with two cameras on a beamsplitter setup: A hyperspectral camera and a Basler dart grayscale camera with a long-pass IR filter in front.

[0070] In total, 10 multispectral bands are obtained in parallel: 8 narrow bands in the visible, 1 grayscale band over all the visible spectrum (from 430nm to 700nm) and 1 near-infrared band (over 850nm). The data from both cameras is synced temporally and spatially through calibration so it can be used together in single inputs. For conversion and labeling, the same approach as for N-MobiFace and N-YoutubeFaces has been used.

[0071] All the event data used for training or finetuning are simulated events. To make sure that the trained models and the conclusions derived are also valid for real events, a few multispectral sequences with real events, Real-SpectralFace, were captured. One camera is mounted with an Infrared long-pass filter, the other one with an IR cut-off filter. Therefore, two event channels have been captured simultaneously: grayscale and infrared, enough to do a simple comparison and validate some observations.

[0072] To examine the advantages of multispectral events for face detection, the RetinaFace architecture is used as presented in

[0073] Jiankang Deng, Jia Guo, Yuxiang Zhou, Jinke Yu, Irene Kotsia, and Stefanos Zafeiriou, Retinaface: Single-stage dense face localisation in the wild, CoRR, abs / 1905.00641, 2019, and the training has been adapted to the experiments. To make annotation easier and because it does not significantly improve the mAP face, landmarks have not been used. For simplicity, binary image 2D representation of events were used: for each pixel, if there was an event within the last time window (50ms in our case), it is represented as white, else it is black, regardless of their polarity.

[0074] RGB and hyperspectral datasets (frames and events) are available, but the RGB data is by far the most abundant. So, a baseline model has been trained from scratch first, using the RGB data and then it was finetuned using the right channel(s) of the final domain. Two experiments have been carried out: Single-channel and Multi-channel.

[0075] In the single channel experiment, the performances of individual channels are compared one to another by training a model for each band and modality (APS / EVS). In the Multi-channel experiment, the effect of combining different single channels together is interesting. In particular, channel combinations can be selected to evaluate two aspects: the number of channels and the effect of near-infrared on the face detection performance. In conclusion, for all models, a RetinaFace network is trained from scratch first on N-MobiFace and N-YoutubeFaces, and then finetuned on the channels of interest from N-SpectralFace. As bimodal datasets (APS and EVS) are available, all of the 42 final models are trained on the same number of samples, in the same order, for fairness. Finally, the performance can only benefit from an evaluation on real events to show that results can generalize to a real multispectral event-based imaging sensor if it is developed. So, grayscale(GS), infrared(IR) and GS+IR models have also been tested on real multispectral events from Real-SpectralFace. Results are indicated in Fig. 5.

[0076] In examples, any wavelength outside of the visible range, which can be defined as between 380nm and 750nm may be used. It can be Ultraviolet (100-380nm), near infrared (750nm to lOOOnm), Short-wave infrared (lOOOnm to 2500nm), middle or long-wave infrared up to thermal imaging (14000nm), or any other.

[0077] Examples of the event-based multispectral image sensor may be used in object detection, object recognition, motion detection, object tracking, robotics, autonomous vehicles, etc.

[0078] In examples, the more spectral bands the higher the selectivity and the better the recognition. For example, the plurality of spectral bands comprises two or more sub-bands in the visible spectrum. The two or more sub-bands may have the same bandwidths. In examples, the narrower the bandwidth of a spectral band (going along with the number of spectral bands) the better in general (more selective). Typically for multispectral sensing it is around 50 or lOOnm wide, while for hyperspectral sensing it can go down to 10 or 5 nm (there is no universal definition). In examples, it can be both, and they can have some gaps in between or they can be overlapping.

[0079] The bands used for the face detection study on simulated events with multiple visible and near infrared channels are 8 bands about lOOnm wide plus one grayscale band i.e. 400 to 700- 800nm plus one band further in the near-infrared range using a long-pass filter in front a second camera 850nm to lOOOnm.

[0080] Fig. 6 illustrates some spectral bands in examples. Fig. 6 shows quantum efficiency in % on the ordinate and wavelength in nm on the abscissa. Fig. 6 shows an example of selected spectral bands in the visible spectrum in an example, approximately reaching from 400nm (IR) up to 730nm.

[0081] The bands used for the face detection study on real events (two bands only) are one band the near-infrared range using a long-pass filter (850nm to lOOOnm) and a visible band (400 to 650nm) using a NIR cut-off filter. This study shows that the results obtained with simulated events generalize to real events by confirming the trend observed, two EVS cameras were used two capture data with two different spectral filters (one band the near-infrared range using a long-pass filter (850nm to lOOOnm) and a visible band (400 to 650nm) using a NIR cut-off filter).

[0082] Fig. 7 shows simulations results in this example. Fig. 7 shows three triplet bar diagrams. In each triplet the very left bar shows results for GS, the middle bar shows results for IR and the bar on the right shows the results for GS+IR. The accuracy slightly increases by combining the two bands. The overall accuracy increases (on the left, mAP=mean average precision), and it can be noticed that the accuracy at intersection over union threshold of 75% (AP.75 on the right, AP.5 in the middle shows 50% ) increases even more, which shows that the detection is much more accurate in terms of where the face was located.

[0083] In the following some examples are summarized: (1) An image sensor configured to capture event-based image data, the image sensor being configured to capture event data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum.

[0084] (2) The image sensor of (1), wherein the at least one band has a center frequency above the frequencies of the visible spectrum.

[0085] (3) The image sensor of (1) or (2), wherein the at least one band is in the ultraviolet light band.

[0086] (4) The image sensor of (1) to (3), wherein the at least one band has a center frequency below the frequencies of the visible spectrum.

[0087] (5) The image sensor of (1) to (4), wherein the at least one band is in the infra-red or near infra-red light band.

[0088] (6) The image sensor of (1) to (5), wherein the at least one band has a center frequency above the frequencies of the visible spectrum and another one of the plurality of spectral bands has a center frequency below the frequencies of the visible spectrum.

[0089] (7) The image sensor of (1) to (6), further comprising a plurality of spectral filters configured to filter the spectrum to obtain spectral bands.

[0090] (8) The image sensor of (1) to (7), further comprising a plurality of event-based sensor elements, which are coupled to the plurality of spectral filters and which are con-figured to capture the event-based data in the plurality of spectral bands.

[0091] (9) The image sensor of (1) to (8), wherein the plurality of spectral bands comprises two or more sub-bands in a visible spectrum.

[0092] (10) The image sensor of (1) to (9), wherein the two or more sub-bands have the same bandwidths.

[0093] (11) A computer vision device comprising one or more image sensors according to (1) to (10). (12) The computer vision device of (11), further comprising a neural network with two or more processing layers, wherein one processing layer of the neural network is configured to combine event-based image data from the plurality of spectral bands to obtain combined data and wherein, optionally, another processing layer of the neural network is configured to process the combined data.

[0094] (13) A method for capturing event-based image data for computer vision, the method comprising capturing event-based image data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum.

[0095] (14) The method of (13), further comprising combining the event-based image data of the plurality of spectral bands using a processing layer of a neural network.

[0096] (15) A computer program having a program code for performing the method according to (13) or (14), when the computer program is executed on a computer, a processor, or a programmable hardware component.

[0097] The aspects and features described in relation to a particular one of the previous examples may also be combined with one or more of the further examples to replace an identical or similar feature of that further example or to additionally introduce the features into the further example.

[0098] Examples may further be or relate to a (computer) program including a program code to execute one or more of the above methods when the program is executed on a computer, processor or other programmable hardware component. Thus, steps, operations or processes of different ones of the methods described above may also be executed by programmed computers, processors or other programmable hardware components. Examples may also cover program storage devices, such as digital data storage media, which are machine-, processor- or computer-readable and encode and / or contain machine-executable, processor-executable or computer-executable programs and instructions. Program storage devices may include or be digital storage devices, magnetic storage media such as magnetic disks and magnetic tapes, hard disk drives, or optically readable digital data storage media, for example. Other examples may also include computers, processors, control units, (field) programmable logic arrays ((F)PLAs), (field) programmable gate arrays ((F)PGAs), graphics processor units (GPU), application-specific integrated circuits (ASICs), integrated circuits (ICs) or system-on-a-chip (SoCs) systems programmed to execute the steps of the methods described above.

[0099] It is further understood that the disclosure of several steps, processes, operations or functions disclosed in the description or claims shall not be construed to imply that these operations are necessarily dependent on the order described, unless explicitly stated in the individual case or necessary for technical reasons. Therefore, the previous description does not limit the execution of several steps or functions to a certain order. Furthermore, in further examples, a single step, function, process or operation may include and / or be broken up into several sub-steps, - functions, -processes or -operations.

[0100] If some aspects have been described in relation to a device or system, these aspects should also be understood as a description of the corresponding method. For example, a block, device or functional aspect of the device or system may correspond to a feature, such as a method step, of the corresponding method. Accordingly, aspects described in relation to a method shall also be understood as a description of a corresponding block, a corresponding element, a property or a functional feature of a corresponding device or a corresponding system.

[0101] The following claims are hereby incorporated in the detailed description, wherein each claim may stand on its own as a separate example. It should also be noted that although in the claims a dependent claim refers to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent or independent claim. Such combinations are hereby explicitly proposed, unless it is stated in the individual case that a particular combination is not intended. Furthermore, features of a claim should also be included for any other independent claim, even if that claim is not directly defined as dependent on that other independent claim.

Claims

ClaimsWhat is claimed is:

1. An image sensor configured to capture event-based image data, the image sensor being configured to capture event data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum.

2. The image sensor of claim 1, wherein the at least one band has a center frequency above the frequencies of the visible spectrum.

3. The image sensor of claim 2, wherein the at least one band is in the ultraviolet light band.

4. The image sensor of claim 1, wherein the at least one band has a center frequency below the frequencies of the visible spectrum.

5. The image sensor of claim 4, wherein the at least one band is in the infra-red or near infra-red light band.

6. The image sensor of claim 1, wherein the at least one band has a center frequency above the frequencies of the visible spectrum and another one of the plurality of spectral bands has a center frequency below the frequencies of the visible spectrum.

7. The image sensor of claim 1, further comprising a plurality of spectral filters configured to filter the spectrum to obtain spectral bands.

8. The image sensor of claim 7, further comprising a plurality of event-based sensor elements, which are coupled to the plurality of spectral filters and which are configured to capture the event-based data in the plurality of spectral bands.

9. The image sensor of claim 1, wherein the plurality of spectral bands comprises two or more sub-bands in a visible spectrum.

10. The image sensor of claim 9, wherein the two or more sub-bands have the same bandwidths.

11. A computer vision device comprising one or more image sensors according to claim12. The computer vision device of claim 11, further comprising a neural network with two or more processing layers, wherein one processing layer of the neural network is configured to combine event-based image data from the plurality of spectral bands to obtain combined data.

13. A method for capturing event-based image data for computer vision, the method comprising capturing event-based image data in a plurality of spectral bands, wherein at least one of the spectral bands lies outside a visible spectrum.

14. The method of claim 13, further comprising combining the event-based image data of the plurality of spectral bands using a processing layer of a neural network.

15. A computer program having a program code for performing the method according to claim 13, when the computer program is executed on a computer, a processor, or a programmable hardware component.

Images