Event-based camera to support object detection using a frame-based camera

Abstract

The invention relates to a system for an automated vehicle for visual environment perception, comprising a frame-based camera (1) for generating camera images and an event-based camera (3) for generating event-based image information, and comprising a computing unit (5) configured to generate event images from image information of the event-based camera (3), to use the event images in a first machine learning model (M1) for a respective first object analysis, and to use the camera images of the frame-based camera (1) in a second machine learning model (M2) for a respective second object analysis, and to adapt the second machine learning model (M2) depending on the result of the first object analysis and / or to augment the camera images of the frame-based camera (1) used for the second machine learning model (M2).and to use the result of the first object analysis and the result of the second object analysis, which is dependent on the first object analysis, as data for an environmental assessment.

Description

[0001] The invention relates to a system for an automated vehicle for visual environment detection, as well as a vehicle with such a system.

[0002] As the level of automation increases, the requirement for a highly automated vehicle to recognize its surroundings—especially people, objects, lanes, accessible space, etc.—increases. This requires the evaluation of a multitude of sensors. In addition to the accuracy of the environment detection, short latency and energy efficiency are also important, so that objects are not only recognized correctly, but also as quickly and energy-efficiently as possible.

[0003] In this context, US 2020 / 0175401 A1 relates to a method comprising the following steps: generating a plurality of images at a cutoff frequency, wherein the images are obtained from one or more image sensors arranged around a vehicle; determining location information of objects classified in images, wherein the determination of the location information is based on the analysis of the images via a first machine learning model at the cutoff frequency, wherein a subset of the images is analyzed via a second machine learning model at less than the cutoff frequency, and wherein the first machine learning model is configured to periodically receive output information from the second machine learning model, wherein the received output information can be used to increase an accuracy associated with the determination of location information;and outputting the determined location information, wherein the determined location information is configured for use in autonomous driving of the vehicle;

[0004] In US 2020 / 0175401 A1, two camera systems with different frame rates are used to enable high accuracy and short latency.

[0005] Information from a first camera with a high frame rate is evaluated using a comparatively inaccurate but fast machine learning model in order to detect objects with the shortest possible latency, in order to combine this initial information with the information from a second camera with a low frame rate in order to detect objects with a comparatively higher accuracy.

[0006] The minimum achievable latency is limited. Firstly, frame-based cameras are used, whose maximum frame rate is technically limited to, for example, 30 frames per second. Furthermore, machine learning models that evaluate frame-based image data with a high number of pixels per frame are very computationally intensive, for example, due to the number of multiply-accumulate arithmetic operations per frame.

[0007] The following information, however, is generally known prior art in the field of specialized cameras and therefore does not require citation of a specific prior art document: Frame-based cameras and event-based cameras are known. An event-based camera, also known as a neuromorphic camera, silicon retina, or dynamic image sensor (e.g., a dynamic vision sensor, abbreviated to "DVS," dynamic-and-active-pixel vision sensor, or an asynchronous time-based image sensor), comprises an image sensor that responds to local changes in brightness. Event-based cameras do not capture images with a shutter like frame-based cameras. Instead, the pixels of an event-based camera are asynchronous and independent. They remain essentially silent but respond independently of one another to changes in brightness in the environment of the event-based camera.For this purpose, each pixel stores a reference brightness and compares it with the new brightness level. If the brightness difference exceeds a threshold, this pixel sets its reference level to the new brightness value and generates an event. The event-based cameras therefore process local relative light intensity changes directly and send these information packets as events to the connected control devices. Here, these can be aggregated temporally and / or spatially to form so-called event images. In contrast to frame-based cameras, the event-based cameras typically only provide binary information about relative light intensity changes, with the information on the direction of change, i.e.However, compared to frame-based cameras, they tend to have a higher temporal resolution and typically a higher dynamic range. This allows for very precise tracking of object movements in their surroundings.

[0008] These properties of event-based cameras, particularly with regard to frame-based cameras, which are commonly used in the state of the art, can be used for automated driving applications, particularly for vehicles on public roads, which require appropriate environmental detection.

[0009] The object of the invention is to make visual environment detection of an automated vehicle more secure.

[0010] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0011] A first aspect of the invention relates to a system for an automated vehicle for visual environmental detection, comprising a frame-based camera for generating camera images and an event-based camera for generating event-based image information, and comprising a computing unit designed to generate event images from image information of the event-based camera, as well as to use the event images in a first machine learning model for a respective first object analysis and to use the camera images of the frame-based camera in a second machine learning model for a respective second object analysis, and wherein the computing unit is designed to adapt the second machine learning model depending on the result of the first object analysis and / or to augment the camera images of the frame-based camera used for the second machine learning model,and to use the result of the first object analysis and the result of the second object analysis carried out dependently on the first object analysis as data for an environment recording.

[0012] The computing unit is designed to generate an event image from image information of the event-based camera. In general, the respective "event images" are temporally aggregated event information (e.g., a histogram of all events over a specific period of time for each pixel of the event-based camera). For this purpose, incoming image information with events is temporally and / or spatially aggregated in the computing unit to form the respective event images. Local aggregation is an aggregation of events for individual pixel positions or regions around a pixel position, and temporal aggregation can weight and take into account the time difference of the event from the current time t0, for example, using exponential temporal memory loss (e.g., so-called "eligibility traces"), etc.For example, with a temporal-spatial aggregation, a histogram can be created for each pixel of the event-based camera (e.g., 5 events at position x1, y1 in the last 5 seconds), whereby the events of the event-based camera are summed over a fixed period of time dt (possibly time-weighted), and thus an event image is generated in time steps dt. For example, a temporal aggregation can be the number of all events of the event-based camera (e.g., 128 events in the last 5 seconds), whereby the events of the event-based camera are recorded over a fixed period of time dt (possibly time-weighted). For example, a spatial aggregation can be the number of all events of the event-based camera at a position of the sensor of the event-based camera or a region of the sensor at the current time t0.It is irrelevant whether the part of the computing unit that generates the respective event image from the image information of the event-based camera is part of the event-based camera itself, or belongs to a control unit of the vehicle itself, or whether the transformation of the image information into a respective event image takes place elsewhere. In addition to the previously described method of temporal and / or spatial event aggregation into an event image, high-dimensional embedding of event information trained using machine learning methods is also known from the state of the art. For example, a multitude of different temporal and / or spatial event aggregations can be generated or mapped using such a trained high-dimensional embedding, thus expanding the concept of event information and the definition of event images.

[0013] For the purposes of this document, the term "visual environment detection" also includes visual interior detection of a vehicle interior. The data from the environment detection serves as input data for assistance systems such as a device for at least partially automated driving, a steering assistant, or a device for interior and / or driver monitoring.

[0014] According to the invention, two different systems are used for the visual environment detection of an automated vehicle, namely an event-based camera and a frame-based camera, which enable the execution of identical or at least similar functions with the execution of the machine learning models (e.g., object detection, gesture recognition, etc.). The event-based camera typically processes local relative light intensity changes directly and asynchronously (e.g., on neuromorphic hardware, analogous to a spiking neural network, etc.), which are aggregated to form the event images. The frame-based camera (e.g., charged coupled device (CCD) or active pixel sensor (APS)), on the other hand, typically processes the information about the light intensity on the sensor on an image- or frame-based basis with a fixed or adaptive frame rate. However, both the camera images and the event images are fed to their respective machine learning models.

[0015] Event-based cameras are advantageously used, as they generate short latencies and a significantly higher rate of (e.g., time-aggregated) images than frame-based cameras due to direct event processing, e.g., greater than 10,000 fps. Furthermore, methods and computation units for processing event-based data are presented that are comparatively advantageous in terms of latency, computational operations, and energy efficiency.

[0016] The event-based camera exhibits low latency (the latency of a single pixel to intensity changes is less than or equal to 1 microsecond), high temporal resolution (approximately 100 microseconds in realistic applications), a comparably high dynamic range (i.e., recordings at very high and very low light intensities are possible), the ability to adapt to changing lighting conditions (e.g., when driving through a tunnel; through an adaptive intensity threshold), and comparably low energy consumption (particularly with few intensity changes), all in relation to the frame-based camera. The frame-based camera, however, has a comparatively higher pixel density and can process information about stationary objects (i.e., objects whose light intensity does not change over time) and color information (which is currently not processed by event cameras).

[0017] Preferably, events from the event camera are temporally aggregated or converted into so-called "discretized event streams." Various methods, such as histograms, binary histograms, event volume, event embedding, etc., can be used to generate event images, i.e., to aggregate the events over a period of time in such a way that an image of the intensity changes is created. These event images can be generated with high temporal resolution, similar to a frame-based camera with a high frame rate, and evaluated using the first model of machine learning, particularly for the purpose of computer vision, for example, with an artificial neural network.

[0018] The event-based camera and the frame-based camera together form a camera system to enable both high accuracy and short latency in environment detection. Information from the event-based camera with high temporal resolution is preferentially evaluated using a comparatively inaccurate but fast first machine learning model in order to detect and, if necessary, track objects, people, or highly dynamic areas, etc., with the shortest possible latency. While the information from the frame-based camera with a low frame rate and comparatively lower temporal resolution is obtained in order to detect and, if necessary, track objects, people, etc. with comparatively higher accuracy using a relatively accurate second machine learning model.

[0019] Both the first and second machine learning models are specifically designed for object detection and recognition. Artificial neural networks are particularly well-suited for this. Object detection and recognition typically involve image segmentation, meaning that each pixel of a given image is classified into a category, for example, "cyclist." Objects should therefore be understood as classes of segments of the environment, such as "cyclist," "traffic light," "pedestrian crossing," or "meadow." However, the speeds and accelerations of such objects, as well as those of other road users, can also be determined using the respective machine learning model.

[0020] Due to the high temporal resolution, the speed of fast objects can be determined very precisely from the sensor data of the event camera using the first model of machine learning. Advantageously, this first step also allows areas in the image with high dynamics, e.g., due to moving objects such as vehicles, pedestrians, animals, etc., to be identified. Alternatively or additionally, events from the event-based camera are processed asynchronously in time to detect and, if necessary, track objects. For this purpose, these events can advantageously be evaluated directly on asynchronous information processing hardware (neuromorphic hardware, analog spiking neural network, etc.). For example, abstracted analog or digital spiking / pulsed neurons (e.g., integrate and fire, leaky integrate and fire, etc.) can be used.) which are advantageously strongly and recurrently connected to temporally aggregate and process information (e.g., to generate an attractor state that characterizes a certain object category).

[0021] In order for the results of the first machine learning model to be used to adjust the second machine learning model or to augment the camera data for the second machine learning model, a temporal connection between the information acquisition of the event-based camera and the frame-based camera must be established, since it can be assumed that the host vehicle, as the carrier of these environmental sensors, is moving, and that objects in the environment can also move relative to the rest of the environment. Therefore, the computing unit preferably performs temporal synchronization in order to provide an augmented camera image with information that was acquired at the same time, in particular at the same time, by the first machine learning model.

[0022] The result of the first object analysis includes, in particular, information about objects, their positions, their position changes, etc., obtained by executing the first machine learning model. Entire image regions in the event-based camera image with high dynamics may have been assigned to an object, whereby a high speed / acceleration of objects moving relative to the event-based camera in the captured area may also be detected.

[0023] This advantageously enables improved object detection and recognition, as well as object tracking. Further possible applications for improved environment detection include depth determination (image-based estimation of the distance to objects in front of the vehicle) and driver monitoring (hypercardia, pupils, gaze direction, etc.). Furthermore, combining different sensor systems with different system properties (diverse redundancy in image acquisition) in a common camera system and increasing the energy efficiency of the camera analysis can also be advantageous.Further advantages for the system design can be the diverse redundancy in the computing unit through the use of neuromorphic and non-neuromorphic hardware and the diverse redundancy in the methodology through the use of mentioned methods such as spiking neural networks in combination with other methods of machine learning or image processing in general.

[0024] According to an advantageous embodiment, the computing unit is designed to augment camera images used for the second machine learning model depending on the result of the first object analysis by at least one of the following steps: - Limiting the camera images to a respective cut-out image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis; - limiting the camera images to a respective cut-out image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis, wherein the computing unit is designed to determine the size of the image area cut out in the respective camera images depending on a change in position and / or speed and / or acceleration of the object detected by means of the respective event image determined by means of the first object analysis; - Filtering the camera images in an image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis; - Using an event image as an additional channel in a temporally and preferably spatially associated camera image in an image area for which an object was detected in this event image by means of the first object analysis; - Highlighting by enlarging an image area in a respective one of the camera images for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis; - Enhancing an image area of ​​a camera image used for the second machine learning model, comprising an object recognized by the second machine learning model, with information from the result of the first object analysis about an object recognized in the first machine learning model in the common image area of ​​a respective event image and a respective temporally associated camera image; - Masking out an image area in a respective image area of ​​the camera images, for whose spatially associated pixels in a respective event image an assignment to a respective known object with a probability below a predetermined limit has been made by means of the first object analysis; - Overlaying information of a respective event image with original data of the camera images to generate at least one augmented camera image in an area of ​​a detected object of interest.

[0025] Augmenting camera images specifically means expanding / enhancing and / or highlighting information in a given camera image. This occurs depending on the result of the calculations in the first machine learning model, meaning that augmentation does not have to occur for all camera images, but generally occurs for some results of the calculations of the first machine learning model and not for others. However, configurations are also possible in which all camera images are generally augmented depending on the result of the first machine learning model. The adaptation of the second machine learning model, on the other hand, is preferably only temporary.

[0026] To limit the camera images to a respective cropped image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis, a respective camera image is, so to speak, cropped. The area around an object in the surroundings of the vehicle detected with the first machine learning model can be delimited, for example, with a bounding box. Alternatively or additionally, an area with high dynamic range can be cropped out and only this area of ​​the camera images of the frame-based camera can be processed in the second machine learning model, wherein the size of the section can depend in particular on the change in position and / or speed and / or acceleration of the object detected with the first object analysis carried out with the first machine learning model, in particular relative to a respective event image.The second machine learning model can then process a smaller input space, e.g., an image with fewer pixels. This can reduce computing time, memory requirements, and energy consumption in the second object analysis, or allow a more accurate second machine learning model to be calculated with comparable computing requirements.

[0027] When filtering the camera images in an image area for which an object was detected in a respective temporally and spatially associated event image using the first object analysis, the image information from the frame-based camera can, for example, be additionally filtered locally before the image information is processed by the second machine learning model. For example, the contrast around an object detected with the first machine learning model is increased or decreased in the image information of the frame-based camera. Alternatively or in addition to the contrast, the saturation (in particular color saturation) can be changed. It is also conceivable to improve the illumination of the detected object by focusing the light beam using preferably controllable pixel spotlights.

[0028] A further augmentation can be that information from the event image is used as an additional channel (RGB + BE) for each of the camera images in the area of ​​an object of interest detected (in the same event image by the first machine learning model).

[0029] In addition, an enhancement of an image area of ​​a camera image used for the second machine learning model, comprising an object recognized by the second machine learning model, can be carried out with information from the result of the first object analysis about an object recognized in the first machine learning model in the common image area of ​​a respective event image and a respective temporally associated camera image. If the second machine learning model is an artificial neural network that suggests object categories for predefined sub-areas of an image, as well as optional shifts and adjustments of the sub-area (e.g.If a single-shot detector (SSD) predicts a predefined region, where the predefined regions are referred to as prior boxes or default boxes, the information from the first machine learning model about detected objects at specific positions can be used to strengthen the predefined region that covers the regions of these positions. For example, the confidence of the second machine learning model for these regions can be increased by a constant value or by a value that depends on the prediction certainty of the first machine learning model.

[0030] In addition, highlighting can be performed by enlarging an image area in a respective camera image for which an object was detected in a respective temporally and spatially related event image using the first object analysis. For example, highlighting can be achieved by increasing the contrast only in the corresponding area in the camera image and using it for further processing. For example, extreme highlighting can be achieved by only enlarging the area of ​​a detected object of interest in the camera image and using it for further processing.

[0031] If the second model detects an object of particular relevance, e.g. a vulnerable road user, this information can be used in the first model via a feedback loop, whereby methods such as contrasting, highlighting, magnification in the image areas containing the object with particular intensity, clustering, length, etc. are carried out in order to obtain as much information as possible about the object of particular relevance and not to lose the object, for example, during tracking.

[0032] A further augmentation may be that information of the event image, e.g. a contour of the event image, is overlaid in a respective one of the camera images in the area of ​​an object of interest detected (with the first machine learning model).

[0033] According to a further advantageous embodiment, the computing unit is designed to adapt the second machine learning model depending on the result of the first object analysis by one of the following: - Applying priming of an attention mechanism by adjusting attention weights in the second machine learning model with information from the result of the first object analysis; - Temporarily using an alternative and specially parameterized second machine learning model; - Identifying areas in layers in the second machine learning model that correlate with information from the first machine learning model about detected objects and increasing feature maps in these areas; - Identifying regions in layers in the second machine learning model that correlate with information from the first machine learning model about detected objects and reducing feature maps outside these regions;

[0034] The adaptation of the second machine learning model can also be called "priming." By applying priming of an attention mechanism by adjusting attention weights in the second machine learning model with information from the result of the first object analysis, such a second machine learning model that uses an attention mechanism can be used, and by priming the attention mechanism with information from the application of the first machine learning model, better accuracy in object detection can be achieved. If the second machine learning model is an artificial neural network with attention mechanisms, the attention weights can be increased for those areas that match information from the first machine learning model about detected objects.Such an attention mechanism (referred to as "attention" in the technical literature, particularly in connection with transformers), allows for attentional priming, which can involve giving more weight to those network areas of a neural network (if such a network is used as the second machine learning model) that process the area of ​​the camera image and were identified as important in the event image. Especially with convolutional neural networks, the spatial information is initially retained across layers, but can be expanded with different resolutions / information from other areas (so-called skip connections can be used for this), whereby the resolution (given by kernel size and stride) changes across layers. The advantage is that information from an event image can be used to process the camera image, and, for example,Objects detected in the event image can be more easily found in the camera image. If the second machine learning model is an artificial neural network (e.g., ResNet or Highway Network) with so-called skip connections (or shortcut connections), the weights of the skip connections can be increased forwardly for those areas that match information from the first machine learning model about detected objects. In particular, areas can also be masked out in which no objects or objects were detected only with very low certainty / certainty in the first machine learning model.

[0035] Furthermore, by simply calculating the processing of the image information, it is possible to determine the regions in the various layers of a second machine learning model, implemented as an artificial neural network, that correspond to information from the first machine learning model about detected objects. The values ​​of the feature maps in these regions for the respective layer can then be increased, for example, by a constant value or a value dependent on the predictive certainty of the first machine learning model. Alternatively, feature maps that do not lie in these regions can be reduced.

[0036] According to a further advantageous embodiment, the computing unit is designed to evaluate the result of the second object analysis depending on the result of the first object analysis, to link the evaluation to the environment detection and to operate a decision logic of a driving control depending on the evaluated environment detection.

[0037] For evaluation purposes, information about the speed of an object can be determined in the first machine learning model, and evaluations of the second machine learning model can be evaluated based on the object's speed. For example, if a detected object in the environment is traveling at a very high speed relative to the image capture range of the event-based and frame-based cameras, the model prediction of the second machine learning model can be evaluated as less reliable than if the object's speed is lower. This evaluation is particularly useful for decision-making in a driving control system.

[0038] According to a further advantageous embodiment, the computing unit is designed to evaluate the result of the second object analysis depending on a speed of an object detected by the first object analysis in the surroundings of the vehicle, said speed being determined by the first object analysis.

[0039] According to a further advantageous embodiment, the computing unit is designed to merge the result of the first object analysis and the result of the second object analysis carried out as a function of the first object analysis into common data of an environment detection.

[0040] The results of the first and second machine learning models can be fused. For example, object positions can be calculated from the weighted averages of the two results. The results of the second machine learning model, which uses the camera images, can be given greater weight, or an object category (e.g., dog or cat) can be used by the machine learning model with the higher prediction reliability. Furthermore, the velocities and accelerations of objects relative to the ego vehicle can be taken into account when fusing the model results.For example, if an object is moving at very high speed, the model prediction of the first machine learning model can be given higher weight due to motion blur of the frame-based camera, and if the object is moving at very low speeds, the model prediction of the second machine learning model can be given higher weight due to smaller intensity changes.

[0041] According to a further advantageous embodiment, the computing unit is designed to merge the result of the first object analysis and the result of the second object analysis carried out as a function of the first object analysis in a weighted manner, and to determine the weightings themselves as a function of the result of the first object analysis.

[0042] For example, if the first machine learning model predicts a vehicle at the environment position x+ / -dx,y+ / -dy, while the second machine learning model does not, a vehicle will be detected with an 80:20 weighting in favor of the first machine learning model, and no vehicle will be predicted or detected with a 20:80 weighting.

[0043] It is also possible to consider the reliability of object detection or recognition. For example, if the first machine learning model detects a vehicle at the surrounding position x+ / -dx,y+ / -dy with only a confidence of 40%, and the second machine learning model predicts it with a confidence of 100%, a vehicle will be predicted or predicted even with an 80:20 weighting if the overall certainty (0.8 x 0.4 + 0.2 x 1 = 0.52) is greater than a threshold, for example, 50%.

[0044] According to a further advantageous embodiment, the computing unit is designed to determine the weightings as a function of a speed of an object detected by means of the first object analysis, ascertained by means of the first object analysis, and / or as a function of a probability of an object detection and / or object recognition carried out by means of the first object analysis, ascertained by means of the first object analysis.

[0045] According to a further advantageous embodiment, the environment detection comprises at least one of the following: object detection, semantic segmentation of a detected object, object classification, object recognition, object tracking.

[0046] A further aspect of the invention relates to a vehicle with a system as described above and below.

[0047] Advantages and preferred developments of the proposed vehicle result from an analogous and analogous transfer of the statements made above in connection with the proposed system.

[0048] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0049] They show: Fig. 1: A vehicle with a dual camera system according to an embodiment of the invention. Fig. 2: A schematic structure of the cameras and the machine learning models of the vehicle according to an embodiment of the invention.

[0050] The representations in the figures are schematic and not to scale.

[0051] Fig. 1 shows a bird's-eye view of a vehicle with a system for visual environment detection, which has a frame-based camera 1 for generating a camera image and an event-based camera 3 for generating event-based image information. A computing unit 5 of the vehicle generates a respective event image from the image information of the event-based camera 3. This occurs at a predetermined frequency at repeated time intervals, so that a respective video is effectively generated by both the frame-based camera and the event-based camera. After the computing unit 5 has combined the image information from the event-based camera 3 into a respective event image, the computing unit 5 has access to an event image and a camera image, each relating to one of the many points in time during the ongoing detection of the vehicle's environment.In the next step, the computing unit 5 evaluates the event image for each point in time for which an event image is available using a first machine learning model M1. The first machine learning model M1 is an artificial neural network and is used to process the respective event image and to perform object detection and object recognition. For this purpose, for example, a semantic segmentation of the pixels is carried out in order to classify the pixels that depict the surroundings of the vehicle, i.e., to divide them into classes such as "tree", "cyclist", "guardrail", etc.; a second machine learning model, which also has an artificial neural network, serves the same purpose. The second machine learning model M2, however, processes camera images from the frame-based camera 1. Thus, object detection and object recognition are carried out via two channels, for example, usingsemantic segmentation is carried out in the computing unit 5. However, in order to combine the advantages and disadvantages of the frame-based camera 1 and the event-based camera 3 in such a way that the advantages of both are combined as much as possible and the disadvantages fade into the background, the result of the execution of the artificial neural network of the first machine learning model M1 is used to augment at least a subset of the camera images of the frame-based camera 1 using information from the event images themselves or the result of the execution of the first machine learning model M1, and / or to adapt the second machine learning model M2 for the subset of the camera images depending on the result of the first machine learning model M1.For this purpose, the computing unit 5 must ensure that there is a very close temporal relationship, in particular a simultaneity, between the evaluated event images and the respective camera images, which are augmented or processed by an adapted second machine learning model. Finally, the event images evaluated by the first machine learning model M1 and the (possibly augmented) camera images evaluated by the (possibly adapted) second machine learning model M2 are used together for environment detection, so that automatic vehicle guidance can be carried out using the information about objects in the vehicle's surroundings obtained by interpreting the event images and the camera images.In particular, the advantages of the event-based camera 3, namely its high bandwidth and short latency, are exploited to use information about fast-moving objects in the vehicle's surroundings to augment the camera images of the frame-based camera 1. Thus, with the first machine learning model M1 based on the event images, high-frequency information can be reliably obtained, while the second machine learning model M2 exploits the advantages of the camera images, namely high resolution and color information, to obtain information of corresponding depth. The second machine learning model M2 itself can also be adapted; for example, a second version of the second machine learning model M2 with different weightings can be temporarily used if the first object analysis performed by the first machine learning model M1 suggests a reason.

[0052] Fig. Figure 2 shows the logical structure of the cameras 1,3 and the machine learning models M1,M2 for the system of Fig. 1. A frame-based camera 1 generates camera images and transmits them to a second machine learning model M2 in the form of an artificial neural network. An event-based camera 3 generates event-based image information, which is converted into event images by a computing unit 5. The event images are fed to a first machine learning model M1, also in the form of an artificial neural network. The first artificial neural network M1 performs a first object analysis, whereby a classification of areas of the vehicle's surroundings is preferably carried out in the event images of the event-based camera 3. The second artificial neural network M2 performs a second object analysis, whereby a classification of the vehicle's surroundings is carried out pixel by pixel, analogous to the first object analysis.Depending on the result of the first object analysis, which is based on the event images, the second artificial neural network M2 is now adapted and / or the camera images of the frame-based camera 1 used for the second artificial neural network M2 are augmented, as described under . Fig. 1. The augmentation of the camera images as well as the adaptation of the second artificial neural network M2 depending on the result of the first artificial neural network M1 is described in Fig. 2 symbolized by dashed arrows 7,9.

[0053] Computing unit 5 is further configured to merge, preferably in a weighted manner, the result of the first object analysis and the result of the second object analysis, which is performed dependently on the first object analysis. This fusion is performed in a module 11 associated with computing unit 5, preferably after the augmentation and adaptation of the second artificial neural network M2 have been at least partially completed. The data generated by module 11 are used as input data for an assistance system, such as a brake assist system or an automated driving system.

[0054] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that exemplary embodiments are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. List of reference symbols 1 frame-based camera 3 Event-based camera 5 Computing unit 11 Module of the computing unit for fusion M1 first model of machine learning M2 second model of machine learning QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 2020 / 0175401 A1 [0003, 0004]

Claims

[1] A system for an automated vehicle for visual environmental detection, comprising a frame-based camera (1) for generating camera images and an event-based camera (3) for generating event-based image information, and comprising a computing unit (5) designed to generate event images from image information of the event-based camera (3), as well as to use the event images in a first machine learning model (M1) for a respective first object analysis and to use the camera images of the frame-based camera (1) in a second machine learning model (M2) for a respective second object analysis, and wherein the computing unit (5) is designed to adapt the second machine learning model (M2) depending on the result of the first object analysis and / or to augment the camera images of the frame-based camera (1) used for the second machine learning model (M2),and to use the result of the first object analysis and the result of the second object analysis carried out dependently on the first object analysis as data for an environment recording. [2] System according to claim 1, wherein the computing unit (5) is designed to augment camera images used for the second machine learning model (M2) depending on the result of the first object analysis by one of the following: - Limiting the camera images to a respective cut-out image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis; - limiting the camera images to a respective cut-out image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis, wherein the computing unit (5) is designed to determine the size of the image area cut out in the respective camera images depending on a change in position and / or speed and / or acceleration of the object detected by means of the respective event image, determined by means of the first object analysis; - Filtering the camera images in an image area for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis; - Using an event image as an additional channel in a temporally associated camera image in an image area for which an object was detected in this event image by means of the first object analysis; - Enhancing an image area of ​​a camera image used for the second machine learning model (M2) comprising an object recognized by the second machine learning model (M2) with information from the result of the first object analysis about an object recognized in the first machine learning model (M1) in the common image area of ​​a respective event image and a respective temporally associated camera image; - Highlighting by enlarging an image area in a respective one of the camera images for which an object was detected in a respective temporally and spatially associated event image by means of the first object analysis; - Masking out an image area in the camera images for whose spatially associated pixels in a respective event image an assignment to a respective known object with a probability below a predetermined limit has been made by means of the first object analysis; - Overlaying information of a respective event image with original data of the camera images to generate at least one augmented camera image in an area of ​​a detected object of interest. [3] System according to one of the preceding claims, wherein the computing unit (5) is designed to adapt the second machine learning model (M2) depending on the result of the first object analysis by one of the following: - Applying priming of an attention mechanism by adjusting attention weights in the second machine learning model (M2) with information from the result of the first object analysis; - Temporarily using an alternatively and specifically parameterized second machine learning model (M2); - Determining areas in layers in the second machine learning model (M2) that correlate with information from the first machine learning model (M1) about detected objects and increasing feature maps in these areas; - Determining areas in layers in the second machine learning model (M2) which correlate with information from the first machine learning model (M1) about detected objects and reducing feature maps outside these areas; [4] System according to one of the preceding claims, wherein the computing unit (5) is designed to evaluate the result of the second object analysis depending on the result of the first object analysis, to link the evaluation to the environment detection and to operate a decision logic of a driving control depending on the evaluated environment detection. [5] System according to claim 4, wherein the computing unit (5) is designed to evaluate the result of the second object analysis depending on a speed of an object detected by the first object analysis in the surroundings of the vehicle, said speed being determined by the first object analysis. [6] System according to one of the preceding claims, wherein the computing unit (5) is designed to merge the result of the first object analysis and the result of the second object analysis carried out as a function of the first object analysis into common data of an environment detection. [7] System according to claim 6, wherein the computing unit (5) is designed to merge the result of the first object analysis and the result of the second object analysis carried out as a function of the first object analysis in a weighted manner, and to determine the weightings themselves in turn as a function of the result of the first object analysis. [8] System according to claim 7, wherein the computing unit (5) is designed to determine the weightings as a function of a speed of an object detected by means of the first object analysis, ascertained by means of the first object analysis, and / or as a function of a probability of an object detection and / or object recognition carried out by means of the first object analysis, ascertained by means of the first object analysis. [9] System according to one of the preceding claims, wherein the environment detection comprises at least one of the following: object detection, semantic segmentation of a detected object, object classification, object recognition, object tracking. [10] Vehicle with a system according to one of the preceding claims.

Images