Event camera arrangement and motor vehicle having the event camera arrangement

Abstract

The invention relates to an event camera arrangement (20) comprising an event camera (30,30'; 130,130') having a dynamic vision sensor (34) and a camera lens (32), wherein a separate masking means (60; 60') is allocated to the event camera (30,30';130,130'), said masking means having a masking device (62) which, as required, selectively optically covers a partial region of the vision sensor (34) with an opaque masking element (64), and wherein a camera controller (24) having an electronic masking controller (41) is provided which controls the masking means (60; 60').

Description

[0001] DESCRIPTION

[0002] Event camera arrangement and motor vehicle with the event camera arrangement

[0003] The invention relates to an event camera arrangement and to a motor vehicle with such an event camera arrangement.

[0004] An event camera, in this context, is a camera with a dynamic vision sensor, also known as a neuromorphic camera or silicon retina. The dynamic vision sensor reacts pixel by pixel to local brightness changes, with each pixel sensor remaining electronically silent without a brightness change. Since this method generates significantly less image data over time than conventional full-frame sensors, in which each sensor pixel of the full-frame sensor is regularly read and processed, an event camera potentially offers a hundred times higher frame rate and a significantly higher dynamic range with much lower power consumption.

[0005] A fundamental disadvantage of an event camera is the fact that no image data is available in image areas without any movement or brightness variation. However, in many technical applications, for example in robotics or in the field of driver assistance systems for motor vehicles, a representation of static image areas is desired or necessary, so that an event camera cannot be used in many technical applications. It is known from the prior art to set an event camera in motion using a vibration device when required in order to make static image areas visible in this way. However, vibrations are fundamentally detrimental to the health of the event camera. Against this background, the object of the invention is to provide a low-wear event camera arrangement displaying static image areas, comprising a dynamic vision sensor orto create a motor vehicle with such an event camera arrangement.

[0006] This object is achieved according to the invention with an event camera arrangement having the features of claim 1 or with a motor vehicle having such an event camera arrangement having the features of claim 9.

[0007] The event camera arrangement according to the invention comprises a dynamic vision sensor and camera optics, which together form an event camera. The camera optics are understood here to mean lenses and mirrors that project an image onto the vision sensor. The vision sensor is a two-dimensional imaging sensor that reacts pixel by pixel only to brightness changes. This occurs, for example, by storing a reference brightness level for each pixel and continuously comparing this with the current actual brightness level. As soon as the brightness difference between the reference and actual brightness levels exceeds a threshold value, a corresponding signal containing a timestamp and the pixel address is sent to an image processing unit. The vision sensor typically has a high temporal resolution in the kilohertz to megahertz range and a high sensitivity of over 100 dB.

[0008] The event camera is assigned a separate masking device, which includes a masking device that selectively optically shades a portion of the vision sensor with a masking element as needed. The masking element can, for example, be an opaque bar that is quickly moved across the entire field of view of the event camera. After the opaque masking element has completely passed an image pixel, its brightness is again adjusted by the static image regions, so that a brightness signal is output from the respective image pixel. In this way, the static image regions are also made visible if necessary.

[0009] According to the invention, a camera control with an electronic masking control is provided, which controls the masking means in order to generate and move the masking element as required.

[0010] The event camera array or the event camera is therefore not subjected to vibration, significantly improving the reliability and durability of the event camera. This allows event cameras to be used in applications where visibility of static image areas is also required.

[0011] Preferably, a camera stereo array is provided that comprises two rigidly connected identical event cameras whose fields of view largely overlap within the desired distance range. This also allows spatial images to be generated, as is particularly required in robotics and driver assistance systems for motor vehicles. Particularly preferably, the two masking devices of the two event cameras are controlled synchronously by the masking controller, since this ensures high accuracy of the depth information, particularly for the static image areas.

[0012] Preferably, the masking control activates the masking means based on events, i.e., not necessarily at rigid, regular intervals. Particularly in camera-based motor vehicle assistance systems, activation of the masking means is generally only required when the vehicle is stationary, whereas activation of the masking means is generally not required in ferry operation. Preferably, the masking means is arranged spatially between the vision sensor and the camera optics, particularly preferably directly adjacent to the vision sensor proximally. In this way, the masking element can be masked, i.e., shaded, very precisely and, if necessary, even pixel-by-pixel or pixel-line-by-pixel.

[0013] Alternatively, the masking device can be positioned distal to the camera optics, i.e., between the camera optics and the observed scene. This significantly simplifies the conversion or integration of the masking device into a conventional event camera.

[0014] Preferably, the marking device is an LCD screen or a physically different but similarly functioning screen, with a non-activated LCD image pixel being largely transparent and an activated LCD image pixel being largely opaque. The masking device, designed as an LCD screen, has no moving parts, thus protecting the camera mechanically and allowing a relatively fast-moving masking element of any shape to be generated.

[0015] In principle, the masking element can have any shape or form. However, the masking element is particularly preferably a moving bar, which is oriented horizontally or vertically, for example, and is moved vertically or horizontally through the image. By successively masking or shading the entire detection area of ​​the imaging vision sensor, a temporary data overflow is avoided, in particular. Furthermore, by selectively masking any section of the detection area, a part of the statistical scene can be made visible as needed. According to patent claim 9, a motor vehicle is equipped with an event camera arrangement having the features of one of claims 1-8, wherein the camera control has a vehicle dynamics evaluation unit that is informationally connected to the masking control via a data connection.If a specific and predefined vehicle dynamic is present, the vehicle dynamics evaluation unit instructs the masking control to activate the masking agent or the opaque masking element.

[0016] Particularly preferably, the vehicle dynamics evaluation unit is informationally assigned an acceleration sensor and / or a wheel speed sensor, which in turn is assigned to a vehicle wheel. In this way, the vehicle dynamics evaluation unit can regularly instruct the masking control, for example when the vehicle is stationary, to activate the masking means so that the static image areas are also made visible.

[0017] Two exemplary embodiments of the invention are explained in more detail below with reference to the figures. They show:

[0018] Figure 1 schematically shows a motor vehicle with an event camera arrangement with two identical event cameras,

[0019] Figure 2 shows the event camera arrangement of Figure 1 in an enlarged and detailed view with a first embodiment of the two event cameras, in which the masking means is arranged proximal to the camera optics,

[0020] Figure 3 is a plan view of the masking means of the event camera of the

[0021] Figures 1 and 2, and Figure 4 show the event cameras of Figure 1 in a second embodiment, each with a masking means distal to the camera optics.

[0022] Figure 1 schematically illustrates a motor vehicle 100 with four vehicle wheels 110-113. Behind its windshield 102, the vehicle has an event camera array 20 that records events in the longitudinal direction x in front of the motor vehicle 100. Each vehicle wheel 110-113 is assigned a wheel speed sensor 50-53. A monitor 26 is arranged in the dashboard of the motor vehicle 100 below the windshield 102.

[0023] The event camera arrangement 20 has a camera stereo arrangement 22 with two identical event cameras 30, 30' which are rigidly connected to each other and to the motor vehicle 100, the respective fields of view S, S' of which largely overlap at the target distance, as indicated in Figures 1 and 2.

[0024] A central electronic camera control unit 24 is informationally assigned to the two event cameras 30, 30', which includes an image processing unit 44, a vehicle dynamics evaluation unit 40 with an integrated masking control unit 41, and an acceleration sensor 42. The image processing unit 44 is connected to the monitor 26 via a data connection in order to display images obtained from the signals of the event cameras 30, 30'.

[0025] The driving dynamics evaluation unit 40 is informationally connected via signal and / or data connections to the four wheel speed sensors 50-53, the acceleration sensor 42 and the image processing unit 44.

[0026] The acceleration sensor 42 is mounted on the vehicle and provides acceleration values ​​of the motor vehicle 100 along all three spatial axes x, y, and z. The wheel speed sensors 50-53 each provide a wheel speed of the respective vehicle wheel 110-113.

[0027] Figures 2 and 3 show the completely identical event cameras 30, 30' in more detail: each event camera 30, 30' has a dynamic vision sensor 34, an optical masking device 60, 60', and a camera lens 32. The vision sensor 34 is a two-dimensional imaging sensor that responds pixel by pixel to brightness changes. The vision sensor 34 has a temporal resolution in the kilohertz range and a sensitivity of over 100 dB. The signals from the two vision sensors 34 of the two event cameras 30, 30' are processed in the image processing unit 44 and fused into a two-dimensional or three-dimensional image.

[0028] The masking means 60; 60' is formed by a masking device 62 embodied as an LCD screen, which is positioned approximately in a vertical transverse plane xy, and can represent an opaque masking element 64, for example, a bar-shaped masking element 64 that moves from left to right across the entire LCD screen within a few milliseconds. In the non-activated state, the masking device 62 is optically transmissive across its entire surface, so that the vision sensor 34 located behind it captures the entire field of view S, S'. In the exemplary embodiment of the event cameras 30, 30' shown in Figure 2, the masking device 60 is arranged between the camera optics 32 and the vision sensor 34, and in this case is arranged directly adjacent to the vision sensor 34.

[0029] In the second embodiment of the event cameras 130, 130' shown in Figure 4, the masking means 60, 60' is arranged distal to the respective camera optics 32.

[0030] The dynamic masking of the event cameras 30, 30'; 130, 130' is performed in this case on an event-based basis. For this purpose, the vehicle dynamics evaluation unit 40 continuously evaluates the signals from the acceleration sensor 42 and the wheel speed sensors 50-53 to determine whether the motor vehicle 100 is stationary. Since the event cameras 30, 30'; 130, 130' cannot capture the static scenery in front of the motor vehicle 100, the vehicle dynamics evaluation unit 40 activates the masking control 41 when the vehicle is completely stationary. This control, in turn, synchronously controls the masking means 60, 60' of the two event cameras 30, 30'; 130, 130' such that the masking element 64 successively moves across the entire image area within a few milliseconds. As a result, when the motor vehicle 100 is stationary, the static scenery in front of the motor vehicle 100 is also made visible.

Claims

CLAIMS 1. Event camera arrangement (20) with an event camera (30, 30'; 130, 130') having a dynamic vision sensor (34) and a camera lens (32), wherein the event camera (30, 30'; 130, 130') is assigned a separate masking means (60; 60') which has a masking device (62) which selectively optically shades a partial area of ​​the vision sensor (34) with an opaque masking element (64) as needed, and wherein a camera control (24) with an electronic masking control (41) is provided which controls the masking means (60; 60').

2. Event camera arrangement (20) according to claim 1, wherein a Camera stereo arrangement (22) of two rigidly connected event cameras (30,30'; 130,130') is provided, the fields of view (S, S') of which overlap.

3. Event camera arrangement (20) according to claim 2, wherein the two masking means (60; 60') of the two event cameras (30, 30'; 130, 130') are controlled synchronously by the masking control (41).

4. Event camera arrangement (20) according to one of the preceding claims, wherein the masking control (41) activates the masking means (60; 60') in an event-based manner.

5. Event camera arrangement (20) according to one of the preceding claims, wherein the masking means (60) is arranged between the vision sensor (34) and the camera optics (32).

6. Event camera arrangement (20) according to one of the preceding claims 1-3, wherein the masking means (60') is arranged distal to the camera optics (32).

7. Event camera arrangement (20) according to one of the preceding claims, wherein the masking device (62) is a pixel-by-pixel opaque switchable screen, preferably an LCD screen.

8. Event camera arrangement (20) according to one of the preceding claims, wherein the masking element (64) is a moving bar.

9. Motor vehicle (100) with an event camera arrangement (20) having the features of one of the preceding claims, wherein the camera control (24) has a vehicle dynamics evaluation unit (40) which is informationally connected to the masking control (41) and instructs the masking control (41) to activate the masking means (60; 60') when a certain vehicle dynamic is present.

10. Motor vehicle (100) according to the preceding claim, wherein an acceleration sensor (42) is informationally assigned to the driving dynamics evaluation unit (40).

11. Motor vehicle (100) according to one of claims 9 or 10, wherein the driving dynamics evaluation unit (40) is informationally assigned at least one wheel speed sensor (50 - 53) which is assigned to a vehicle wheel (110 - 113).

Images