Environmental perception of a vehicle for a large number of object classes

Abstract

The invention relates to a system for vehicle environment sensing, comprising a sensor unit (1) for transmitting image data about the environment to a computing unit (3), which is designed to analyze the image data in a primary artificial neural network with instance segmentation, and to determine for each detected object (5) a first uncertainty value regarding the detection of the object (5) and a second uncertainty value regarding the assignment of the object (5) to a class, and to transmit for objects (5) with a second uncertainty value above the second boundary condition a geometric area of ​​the image data comprising only this respective object (5) to a secondary pre-trained artificial neural network for object recognition.

Description

[0001] The invention relates to a system for detecting the surroundings of a vehicle, a vehicle with such a system, and a method for supporting a driving control function of an automated vehicle by means of automated detection of the surroundings.

[0002] The following information does not necessarily arise from a single specific document from the prior art, but rather arises from general expert considerations and the current but generally known specialist knowledge itself: Particularly in the field of automated driving of vehicles such as passenger cars or trucks, it is crucial to accurately analyze the vehicle's surroundings and detect objects such as pedestrians. Objects are conceptual elements that are to be detected by machines in a scene. Such object detection is typically performed based on visual data, for example, from a camera, but can also be performed alternatively or additionally with the help of other types of sensors. In all cases, a respective data set containing information about the vehicle's surroundings is recorded at a certain repetition frequency for the purpose of ongoing object detection.To detect objects from the information of such a current data set, a pre-trained artificial neural network can be used. This network receives the information from the above-mentioned current data set as input data and, as output data, provides object recognition with classification of the object into one of several predefined classes of objects. These classes, such as pedestrians, cyclists, vehicles, etc., are already predefined in the design of the artificial neural network and must therefore be included in a sufficient and representative manner in a training data set.

[0003] The parameters of the artificial neural network are defined in a training process prior to its subsequent use in regular operation. Typically, machine learning methods such as supervised learning are used to train the artificial neural network with specified input variables and corresponding specified output variables. Real sensor data or synthetic image data of the vehicle's surroundings, modeled on the sensor data, are used primarily for the input data, while the output variable is specified depending on the subsequent result. For classifying objects in the environment of an automated vehicle, the following classes can be specified, for example: road, traffic sign, tree, traffic light, warning beacon, pylon, dog, road marking, pedestrian, cyclist, passenger car, construction vehicle, etc.For automated driving at Level 3 or 4, particularly in less structured traffic areas such as urban traffic, a large number of classes are required for training and inference. Inference is generally the application and execution of a fully trained artificial neural network. Input data is assigned to the artificial neural network, which processes this input data and then delivers output data as the result of the inference. An inference step is therefore the execution of an inference for certain input data using a certain artificial neural network.When applying environment sensing to an automated vehicle, rare, particularly static, objects on and near the road, such as a child's stroller, a lying bicycle, or a beverage crate, must also be considered due to their safety relevance for maneuver planning. Against this background, the industrialization and efficient implementation of automated environment sensing is hampered by the ongoing increase in requirements, including with regard to the number of object classes to be considered.

[0004] Furthermore, applying a fully trained artificial neural network does not always produce clear or error-free results. Reasons for this include poor lighting conditions, reflections, obscuration by other objects, or deceptive appearances such as images of people on posters or people in disguises. Objects not taken into account in the training data can also appear in later operation. Due to this problem, artificial neural networks have been developed in the state of the art. When used, a measure of the uncertainty in object recognition (i.e. classification) can be output, particularly on a pixel-by-pixel basis. These are artificial neural networks with an uncertainty estimation, i.e. they are functionally designed to provide an individual uncertainty value, particularly for each pixel of an image.Already known methods for uncertainty estimation include the Monte Carlo dropout and the deep ensemble method. However, the ILVI approach (ILVI stands for "intermediate layer variational inference"), which is also well-known in the state of the art, is significantly less computationally intensive and therefore significantly less time-consuming. Such an artificial neural network with an intermediate layer for variational inference is discussed in detail in the publication “Real-time Uncertainty Estimation Based On Intermediate Layer Variational Inference” by the authors Ahmed Hammam, Seyed Eghbal Ghobadi, Frank Bonarens and Christoph Stiller, CSCS'21, November 30, 2021, Ingolstadt, Germany, ACM ISBN 978-1-4503-9139-9 / 21 / 11 (available at the time of writing at https: / / doi.org / 10.1145 / 3488904.3493381).The approach presented there uses an additional layer compared to a conventional neural network, which serves to directly obtain a probability estimate for the assignment by reading the data from the additional layer. This additional layer advantageously avoids having to analyze data from the entire neural network, which would be significantly more time-consuming than collecting data from just the additional layer. With the help of this additional layer, a probability per pixel related to a respective class is obtained as an additional output variable of the artificial neural network.

[0005] The ILVI approach is briefly outlined below: The additional layer of the artificial neural network, whose data is read out, serves to model the uncertainty for pixel classification. Two constructs are used within this layer, namely the tensors "mean" and "variance." However, no averaging or variance calculation is performed in the true sense; rather, these are determined, among other things, via convolutional layers corresponding to one of the structures of the first artificial neural network. This is achieved with the help of the additional layer, which generates the mean tensor after a 1x1 convolutional transformation and, in parallel, the variance tensor via a second 1x1 convolutional transformation. In a further step, the data from the additional layer is read out, in which the mean tensor is added to the variance tensor multiplied by a random value.The aim of the training is to train all weights with the help of the loss function so that these two tensors represent the parameters of a Gaussian distribution. For each inference, the conversion is then carried out several times for each scene image across the levels of the first artificial neural network, preferably in the range of 8 to 16 times. Confidence values ​​equal to the number of classes are then obtained, preferably by applying a logistic function such as the softmax function. After carrying out n inference repetitions, the n confidence values ​​are averaged for each class. Each pixel of the semantic segmentation is assigned the class with the highest confidence value. To determine the respective probability measure for each pixel, a standard entropy formula is preferably used (this corresponds to equation (9) in the above-mentioned publication regardingof the ILVI approach) the uncertainty for each pixel and thus the respective probability measure is calculated.

[0006] During the inference step of an unmodified artificial neural network, typically, due to the properties of the output layer (e.g., with softmax functions), unknown real objects that do not belong to any of the classes specified during training of the artificial neural network are always incorrectly assigned to one of the classes used for training. While uncertainty modeling methods, as explained above, are capable of simultaneously estimating the detection uncertainty of a pixel while simultaneously estimating its class, this does not yet provide any information as to whether the object is known or unknown.Consequently, a conventional, unmodified artificial neural network, even with uncertainty estimation (as in the ILVI approach explained above), is typically unable to recognize unknown objects (i.e., objects for which no specific class was specified during training of the artificial neural network). A known object is an object that belongs to an object class whose pixels in the data were annotated as such and taken into account during training. This is not the case for an unknown object.

[0007] In this regard, it is known in the prior art to extend an artificial neural network with semantic segmentation and pixel-by-pixel uncertainty estimation, in particular based on the ILVI approach, such that an additional output layer is introduced. Using input data based on the respective current image data, object recognition is carried out in the image data in a respective inference step. In the artificial neural network, a vector of concentration parameters of a Dirichlet distribution is determined for each pixel. In an additional layer, an assignment of a respective pixel to a class of objects and a pixel-related uncertainty value regarding the assignment are determined from the respective vector of concentration parameters. The assigned class is then classified as belonging to a known or unknown object.Based on these vectors with the concentration parameter components, the pixel is assigned to a class, the uncertainty values ​​are calculated, and binary information about whether a pixel belongs to a known or unknown class is obtained for each pixel. The basic principles can be found in the publication "Towards Improved Intermediate Layer Variational Inference for Uncertainty Estimation" by Ahmed Hammam, Frank Bonarens, Seyed Eghbal Ghobadi, and Christoph Stiller (https: / / link.springer.com / chapter / 10.1007 / 978-3-031-25072-9_36). The publication "Predictive Uncertainty Quantification of Deep Neural Networks using Dirichlet Distributions" by Ahmed Hammam, Frank Bonarens, Seyed Eghbal Ghobadi, and Christoph Stiller also describes this. 2022. In Proceedings of the 6th ACM Computer Science in Cars Symposium (CSCS '22). Association for Computing Machinery, New York, NY, USA, Article 4, 1-10. https: / / doi.org / 10.1145 / 3568160.3570233 describes the application of the ILVI approach together with this modeling of Dirichlet distributions and introduces a cost function to train an artificial neural network for improved uncertainty estimation without specific annotation. In other words, an image dataset can be transformed by the pre-trained artificial neural network during its operation (in a respective inference step), resulting in a per-pixel Dirichlet distribution for the output values ​​of the artificial neural network. In semantic segmentation, the predicted class is specifically the one with the highest concentration parameter. After normalizing the concentration parameters, the entropy can be calculated and used as an estimate of the uncertainty of a pixel's class prediction.Semantic segmentation involves assigning each pixel to a respective class. Furthermore, semantic segmentation outputs a respective uncertainty measure for each pixel. This uncertainty measure is derived from the concentration parameters, while the information for this uncertainty estimate can only be obtained with the help of the intermediate layer provided for this purpose in the artificial neural network. Using Dirichlet distributions, uncertainties for the assignment of classes to i) known and ii) unknown objects are thus possible.This advantageously avoids the situation where, as is often the case in conventional approaches, pixels are classified by assigning them to one of a plurality of object classes. This occurs by determining the probability of a correct assignment for each pixel with respect to each of the given classes of known objects and selecting the assignment with the highest probability. Thus, a pixel is always assigned to a class of a known object, without a class representing a possible assignment for unknown objects. Therefore, a separate class, the so-called unknown class, can be used for unknown objects.

[0008] The object of the invention is to make machine detection of the environment of a vehicle more efficient.

[0009] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0010] A first aspect of the invention relates to a system for detecting the surroundings of a vehicle, comprising a sensor unit and a computing unit, wherein the sensor unit is designed to transmit image data about the surroundings of the vehicle to the computing unit, and wherein the computing unit is designed to analyze the image data in a primary pre-trained artificial neural network, wherein the primary artificial neural network is designed for instance segmentation in order to detect objects in the surroundings of the vehicle based on the image data, to assign each detected object to a class, to localize the detected objects by means of localization, and to determine, for each detected object, a first uncertainty value regarding the detection of the object and a second uncertainty value regarding the assignment of the object to a class, wherein each assigned class is an element from a finite set of classes,which were used to train the primary artificial neural network, and wherein the computing unit is designed to transmit, for each such object whose first uncertainty value falls below a first predetermined boundary condition and simultaneously whose second uncertainty value exceeds the second predetermined boundary condition, a geometric region of the image data determined by means of the localization and comprising only this respective object to a secondary pre-trained artificial neural network, wherein the secondary pre-trained artificial neural network is designed to classify one object per image region.

[0011] The sensor unit is preferably located on the vehicle itself and detects the vehicle's surroundings, continuously provides current information about the vehicle's surroundings during operation, and transmits this information to the processing unit in the form of image data. The sensor unit is preferably a camera unit, but in principle, alternative sensor units such as lidar or the result of a sensor fusion of similar or dissimilar sensors can also be used.

[0012] As explained at the beginning, state-of-the-art methods are known for evaluating recognized classes of objects with accuracy values ​​(uncertainty values ​​regarding the correctness of the assigned class can be defined equivalently). These express the extent to which the executing system is certain that the respective object actually belongs to the assigned class. A possible and preferred approach for determining accuracy values ​​(alternatively expressible in uncertainty values) is the ILVI approach explained at the beginning. Naturally, the uncertainty for the rare objects mentioned at the beginning will be higher than that for common object classes that are therefore extensively considered in training, such as cyclists or cars.

[0013] These accuracy values ​​or uncertainty values ​​are now used to feed a first subset of reliably classified objects (i.e. the accuracy values ​​are high, in particular above a predetermined limit, i.e. the uncertainty values ​​are correspondingly low) directly to a driving control function, and to use a second subset of unreliably classified objects for further downstream classification by the secondary pre-trained artificial neural network.

[0014] If the first uncertainty value falls below the first specified boundary conditions, in particular a first specified limit value, the uncertainty regarding the presence of an object is low. This means that an object has been detected by the vehicle with a high degree of reliability, i.e., such an object has been found. If the second uncertainty value falls below a second specified boundary condition, in particular a second specified limit value, the object has been correctly assigned to the correct class with a high degree of reliability. These are typically standard objects that correspond exactly to the objects with which the primary artificial neural network was trained in the context of the traffic scenes used.If, however, the second uncertainty value exceeds the second predefined boundary condition, the object is likely to be rare and was not present in the set of objects used to train the primary artificial neural network. For such a rare object, an image section containing this object, determined using localization, is transmitted to the secondary artificial neural network for object recognition. The classification of objects into these standard / special objects during training is preferably done manually and is then experience-based. The criteria for this are the probability of occurrence in the specific domain and the availability of sufficient scene data sets containing the respective objects.

[0015] It is irrelevant whether the respective uncertainty value is replaced by a reliability value, which leads to the exchange of the terms “undershoot” and “exceed”.

[0016] The differences between the primary pre-trained artificial neural network and the secondary pre-trained artificial neural network are as follows: The primary is a highly complex artificial neural network designed for instance segmentation. This means that image regions in which objects are suspected are marked as such, with the objects being handled in a way that is distinguishable from one another. As a result of the execution of the primary network, a respective pixel-based instance assignment, such as an instance number, is obtained. Like semantic segmentation, instance segmentation provides a class assignment, but also allows a differentiation between different instances of a class of objects, i.e., individual objects of the same class can be distinguished.When implemented, this can, for example, lead to a situation where “Cyclist 1”, “Cyclist 2”, etc. were detected in the image areas. Typically, the primary network is also designed in such a way that it can perform semantic segmentation to classify each individual image pixel into one of a large number of pre-trained object classes. The semantic segmentation and instance segmentation are then preferably implemented in parallel (i.e. [quasi-]simultaneously) executed branches in the primary pre-trained artificial neural network. For this purpose, the primary pre-trained artificial neural network is fed with a large number of objects relevant to traffic situations in the input data during training. Such objects are objects from classes that can frequently occur in traffic situations, for example cyclists, passenger cars, traffic signs.However, considering a large number of rare objects for training the primary network is hardly feasible. In contrast, the secondary pre-trained artificial neural network typically has a simpler structure, as it only pursues the purpose of object recognition. For this purpose, the secondary network is trained with an extremely large number of individual objects and a corresponding class of objects is specified. Unlike the primary network, these object classes for the training purposes of the secondary network are not limited to traffic situations. Rather, everyday objects, animals, and a multitude of other object classes from the world are used here, which can be recognized by the secondary network in object recognition. The number of trained classes in the secondary network can therefore exceed that of the primary network by several times.Accordingly, diverse and large databases are used in order to be able to use input data to train the secondary system in an appropriate manner. These databases can be generally accessible and, advantageously, do not come from traffic scenarios, meaning that no test drives are necessary to obtain the data. The complexity of the secondary system can be lower in that the selection of a geometric area around an object with classification with high uncertainty carried out by the computing unit during instance segmentation means that, on the one hand, only a portion of an overall image from the sensor unit is transmitted to the secondary system, and, on the other hand, this portion therefore only contains a single object that the secondary system is to recognize through classification.This significantly simplifies the task to be performed by the secondary, as this approach ensures that the image area transmitted to the secondary only contains a single object - this represents a significant simplification of the task compared to the task of the primary.

[0017] Considering a large number of rare objects for perception training, as is the case with the primary artificial neural network, is hardly feasible with the current state of the art. However, extensive image datasets can be more easily made available for training such an artificial neural network. The large number of rare, special objects in the vehicle's environment is advantageously eliminated from the computationally intensive perception function of the primary pre-trained artificial neural network by supplementing the inference by the primary pre-trained artificial neural network with a downstream process.According to the invention, the primary pre-trained artificial neural network performs the computationally intensive perception function with localization and classification for all identified objects, but performs the classification for a smaller number of pre-trained object classes. Furthermore, the image sections containing additional (potential) objects are identified. The class of these additional objects is determined by a subsequent, pure object recognition process for the respective image section. The secondary artificial neural network serves this purpose. As a result, for example, a stationary bicycle is recognized as a standard object and part of the training data set. A lying bicycle, on the other hand, is classified as a special object, and the corresponding image area is passed on to the secondary pre-trained artificial neural network.

[0018] The smaller the number of classes with which the primary artificial neural network is pre-trained, the smaller the selection of classes into which objects are classified by the primary pre-trained artificial neural network in the inference. Reducing the number of these classes significantly simplifies the task of the primary pre-trained artificial neural network, i.e., computing time and energy are saved when executing the primary pre-trained artificial neural network. However, as explained above, the primary network is designed to transfer image regions containing objects with uncertain class assignment to the secondary network in such a way that each image region contains only one object. The execution of the secondary network is therefore significantly more energy- and time-efficient than the execution of an artificial neural network, which receives image data with a (potentially) large number of objects per image.This allows a much larger number of classes to be considered for such special objects in the secondary pre-trained artificial neural network. A further advantage arises from the application's ability to implicitly identify image sections for explicitly unknown objects. Even image regions with unknown objects against an unknown background are thus transferred to the secondary pre-trained artificial neural network. Previously known methods using artificial neural networks, however, are unable to identify unknown objects without specific annotation.

[0019] In summary, the separate treatment of objects frequently encountered in road traffic and rare objects, which are typically more unusual in traffic scenes, creates reliable environment detection and, moreover, energy-efficient environment detection. The number of classes for the perception task of the primary artificial neural network is therefore significantly reduced, allowing computing time and energy consumption during inference to be saved. Nevertheless, all safety-relevant objects can be detected. This also implicitly contributes to the detection of unknown objects—objects that were not yet known during development, are very novel, or have unusual "poses" of objects.

[0020] According to an advantageous embodiment, the computing unit is designed to transmit the result of the localization and classification of a respective object whose first uncertainty value falls below the first predetermined limit condition and simultaneously whose second uncertainty value falls below the second predetermined limit condition, and the result of the classification of the secondary pre-trained artificial neural network to an automatic driving control system.

[0021] According to a further advantageous embodiment, the primary artificial neural network, when executed on the computing unit, performs a pixel-by-pixel semantic segmentation in addition to the instance segmentation.

[0022] Semantic segmentation involves assigning a specific pixel to a specific class. During training of the artificial neural network, prior to regular operation, a large number of predefined, specific classes of known objects are used as input, for example, traffic sign, tree, traffic light, pedestrian, cyclist, and passenger car. The result of semantic segmentation during regular operation of the artificial neural network is thus the output of the class assignment per pixel.

[0023] According to a further advantageous embodiment, the primary pre-trained artificial neural network has an ILVI additional layer and is designed to model a pixel-related output value of the primary predefined artificial neural network as a Dirichlet distribution during pixel-wise semantic segmentation and to determine a respective pixel-related uncertainty value from respective concentration parameters of the respective Dirichlet distribution, and to determine the respective object-related first uncertainty value and second uncertainty value from the plurality of pixel-related uncertainty values.

[0024] Based on the ILVI approach, an additional layer is used to model the uncertainty of perception and an output layer for additional estimation of the Dirichlet distribution. A respective pixel-related output value is modeled as a Dirichlet distribution, and the respective pixel-related uncertainty value is determined from concentration parameters of the Dirichlet distribution.

[0025] According to a further advantageous embodiment, the computing unit is designed to carry out a classification into known objects and unknown objects by means of the concentration parameters of the respective Dirichlet distribution, and to transmit only a respective area of ​​the image data with unknown objects to the secondary pre-trained artificial neural network for object recognition.

[0026] According to a further advantageous embodiment, the primary pre-trained artificial neural network is pre-trained with image data of traffic situations comprising a plurality of objects within these traffic situations and an environment, and the secondary pre-trained artificial neural network is pre-trained with information about objects without a traffic situation and without an environment.

[0027] According to a further advantageous embodiment, the computing unit is designed to carry out a depth estimation for determining the distance between the vehicle and a respective object by executing the primary pre-trained artificial neural network.

[0028] Depth estimation allows for distance estimation of sections in an image captured by the sensor unit depicting the vehicle's surroundings, eliminating the need to analyze more distant areas of the environment using the secondary neural network. This allows distant environmental areas and objects to be ignored in this network, as they are not relevant for automatic driving control anyway.

[0029] According to a further advantageous embodiment, the computing unit is designed to assign a meta-annotation to objects and to transmit the respective meta-annotation to an automatic driving control system.

[0030] This process implicitly preserves meta-annotations relevant for maneuver planning. The objects with the meta-annotations can then be considered by the vehicle control system for maneuver planning.

[0031] A further aspect of the invention relates to a vehicle having a system as described above and below, wherein the computing unit of the system is designed to use the result of the environment detection for an automatic driving control function of the vehicle.

[0032] Advantages and preferred developments of the proposed vehicle result from an analogous and analogous transfer of the statements made above in connection with the proposed system.

[0033] A further aspect of the invention relates to a method for supporting a driving control function of an automated vehicle by automated environment detection, comprising the steps: - Over a plurality of consecutive points in time: Repeatedly determining image data about the surroundings of the vehicle by a sensor unit and transmitting the image data to a computing unit; - At each of the points in time: Determination of a critical distance of a minimum-risk maneuver based on known latency times, the current vehicle speed and an estimated braking deceleration on the current road layout; - Performing a respective inference for the image data of each time point using a primary pre-trained artificial neural network to perform instance segmentation by means of the computing unit, so that an instance list with instances of objects and assigned classes, with distances of the objects to the sensor unit, with assignment of a pixel mask, with the associated number of pixels, and with a second uncertainty value regarding the assignment of an object to a class is obtained; - Eliminating instances from the instance list with a mean distance greater than a first predetermined threshold, and / or with a number of pixels less than a second predetermined threshold, and / or with a second uncertainty value less than a third predetermined threshold; - Creating an object list, which is a list of detected objects, by copying the instances of the instance list remaining after elimination with a second uncertainty value regarding the assignment of an object to a class smaller than a predefined fourth threshold into the object list together with associated meta-annotations, and by performing the following steps in a loop over all remaining object instances with a second uncertainty value greater than the fourth threshold: Determining the smallest rectangular image section that completely contains an area consisting of the pixel mask of the respective instance and a predefined geometric overset, Inference with a secondary pre-trained artificial neural network based on the image data of the area,and if a confidence value determined by the secondary artificial neural network exceeds a predetermined fifth threshold: copy this object with meta-annotation into the object list; and, - Passing the object list to the drive control function.

[0034] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0035] They show: Fig. 1: A system for detecting the surroundings of a vehicle according to an embodiment of the invention. Fig. 2: A method for supporting a driving control function of an automated vehicle by automated environment detection according to an embodiment of the invention.

[0036] The representations in the figures are schematic and not to scale.

[0037] Fig. Figure 1 shows a system for detecting the surroundings of a vehicle. A sensor unit 1 and a processing unit 3 are integrated into the vehicle. The system is described using a perception task based on image data from sensor unit 1 with a camera for image data without temporal linking (single shot): Sensor unit 1 generates and transmits image data about the vehicle's surroundings to processing unit 3 at a specified frame rate in equidistant time steps. Processing unit 3 feeds this image data as input data to a primary pre-trained artificial neural network for instance segmentation analysis.Based on the image data, objects 5 in the vehicle’s surroundings are identified, i.e. searched for, using instance segmentation. The objects 5 found are each assigned to a class, but can still be distinguished from one another by the computing unit 3, even within the same class. Furthermore, the primary artificial neural network carries out a depth estimation in order to estimate the distance between objects and the vehicle. The church steeple in the background is identified as being too far away from the vehicle to be relevant for the vehicle and its maneuver planning. For this reason, the secondary artificial neural network does not carry out any further analysis of the church steeple, nor is it transmitted to the driving control system as object 5. The core idea is also to identify and classify the large number of special objects 5, such as the one in . Fig. 1 and the beverage crate are to be classified by the secondary artificial neural network. Standard objects 5 are understood to be typical objects known for road traffic, and in particular dynamic objects 5 such as pedestrians, cyclists, motorcycles, cars, trucks, etc. The category of special objects 5 primarily refers to static, rarely occurring objects and items that are nevertheless safety-relevant for maneuver planning. For this purpose, for each identified object 5, a first uncertainty value regarding the identification of the object 5 and a second uncertainty value regarding the assignment of the object 5 to a class are determined. These classes are the ones used to train the primary artificial neural network.For objects 5 that were identified as such with sufficient reliability and for which sufficient reliability could be determined in the classification into the respective classes using the concentration parameters of the Dirichlet distribution, they are used in an automatic driving control system. For other objects 5 that were identified as such with sufficient reliability but whose assignment to their respective class was unreliable, a geometric region is isolated from the image data that contains only one of these uncertain objects 5, and the isolated region is transmitted to a secondary pre-trained artificial neural network. These are shown in . Fig. 1 the child's vehicle and the beverage crate. For this purpose, the primary pre-trained artificial neural network has a shared feature extraction (“shared backbone”) with an additional layer for modeling uncertainty, based on the ILVI approach. Its output consists of latent representations and serves as input for the perception layers in different branches (“branches” or “heads”). It also has a branch for semantic segmentation with subsequent calculation of the concentration parameters of a Dirichlet function. For each pixel, a semantic segmentation is obtained for class assignment, a confidence assignment as a value for the estimated certainty of detection (100% for certain), and a value for the classification of known or unknown objects. The latter is referred to below as special object assignment.An advantageous implementation results from positioning the ILVI layer between the extraction and perception layers. Another branch is used for instance segmentation to distinguish between different instances of a class. The result of this branch is a respective pixel-based instance number. A third branch is used for depth estimation, which assigns a depth or distance value per pixel to the sensor unit. The primary artificial neural network is trained using a cost function exclusively for standard objects. To achieve this, it is advisable to mark annotated special objects as areas to be ignored (so-called "ignore areas"). Special objects are detected and fed into the classification algorithm of the secondary pre-trained artificial neural network.The result of the execution of the secondary pre-trained artificial neural network is a class assigned to the respective object and a corresponding confidence value. The secondary pre-trained artificial neural network is thus used to classify special objects for a reduced image section with essentially only one object per section. Therefore, the secondary network is only trained with image data that always contains only one object that covers a large part of the image (image section). The task of localization is eliminated.

[0038] Fig. 2 shows a method for supporting a driving control function of an automated vehicle by automated environment detection, comprising the steps: - Over a plurality of successive points in time: Repeatedly determining S1 image data about an environment of the vehicle by a sensor unit 1 and transmitting the image data to a computing unit 3; - At each of the points in time: Determination S2 of a critical distance of a minimum-risk maneuver based on known latencies, the current vehicle speed and an estimated braking deceleration on the current road layout; - Executing S3 a respective inference for the image data of each time point with a primary pre-trained artificial neural network to execute an instance segmentation by means of the computing unit 3, so that an instance list with instances of objects 5 and assigned classes, with distances of the objects 5 to the sensor unit 1, with assignment of a pixel mask, with the associated number of pixels, and with a second uncertainty value regarding the assignment of an object 5 to a class is obtained; - Eliminating S4 instances from the instance list with a mean distance greater than a first predetermined limit, and / or with a number of pixels less than a second predetermined limit, and / or with a second uncertainty value less than a third predetermined limit; - Creating S5 an object list, which is a list of detected objects 5, by copying the instances of the instance list remaining after elimination with a second uncertainty value regarding the assignment of an object 5 to a class smaller than a predefined fourth threshold into the object list together with associated meta-annotations, and by performing the following steps in a loop over all remaining object instances with a second uncertainty value greater than the fourth threshold: Determining the smallest rectangular image section that completely contains an area consisting of the pixel mask of the respective instance and a predefined geometric overset, Inference with a secondary pre-trained artificial neural network based on the image data of the area,and if a confidence value determined by the secondary artificial neural network exceeds a predetermined fifth threshold: Copy this object 5 with meta-annotation into the object list; and, - Pass S6 of the object list to the drive control function.

[0039] Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of variations exist. It is also clear that the embodiments mentioned by way of example are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description. List of reference symbols 1 sensor unit 3 Computing unit 5 objects S1 Determine S2 Determine S3 Execute S4 Eliminate S5 Create S6 Handover QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] Real-time Uncertainty Estimation Based On Intermediate Layer Variational Inference” by Ahmed Hammam, Seyed Eghbal Ghobadi, Frank Bonarens, and Christoph Stiller, CSCS'21, November 30, 2021, Ingolstadt, Germany, ACM ISBN 978-1-4503-9139-9 / 21 / 11 (available at the time of writing at https: / / doi.org / 10.1145 / 3488904.3493381

[0004] Towards Improved Intermediate Layer Variational Inference for Uncertainty Estimation” by Ahmed Hammam, Frank Bonarens, Seyed Eghbal Ghobadi, and Christoph Stiller (https: / / link.springer.com / chapter / 10.1007 / 978-3-031-25072-9_36

[0007] Predictive Uncertainty Quantification of Deep Neural Networks using Dirichlet Distributions“ der Autoren Ahmed Hammam, Frank Bonarens, Seyed Eghbal Ghobadi, and Christoph Stiller. 2022. In Proceedings of the 6th ACM Computer Science in Cars Symposium (CSCS '22). Association for Computing Machinery, New York, NY, USA, Article 4, 1-10. https: / / doi.org / 10.1145 / 3568160.3570233

[0007]

Claims

[1] A system for detecting the surroundings of a vehicle, comprising a sensor unit (1) and a computing unit (3), wherein the sensor unit (1) is designed to transmit image data about the surroundings of the vehicle to the computing unit (3), and wherein the computing unit (3) is designed to analyze the image data in a primary pre-trained artificial neural network, wherein the primary artificial neural network is designed for instance segmentation in order to detect objects (5) in the surroundings of the vehicle based on the image data, to assign each detected object (5) to a class, to localize the detected objects (5) by means of localization, and to determine, for each detected object (5), a first uncertainty value regarding the detection of the object (5) and a second uncertainty value regarding the assignment of the object (5) to a class, wherein a respective assigned class is an element from a finite set of classes,which were used to train the primary artificial neural network, and wherein the computing unit (3) is designed to transmit, for a respective such object (5) whose first uncertainty value falls below a first predetermined limit condition and simultaneously whose second uncertainty value exceeds a second predetermined limit condition, a geometric region of the image data determined by means of the localization and comprising only this respective object (5) to a secondary pre-trained artificial neural network, wherein the secondary pre-trained artificial neural network is designed to classify one object per image region. [2] System according to claim 1, wherein the computing unit (3) is designed to transmit the result of the localization and classification of a respective object (5) whose first uncertainty value falls below the first predetermined limit condition and at the same time whose second uncertainty value falls below the second predetermined limit condition, and the result of the classification of the secondary pre-trained artificial neural network to an automatic driving control system. [3] System according to one of the preceding claims, wherein the primary artificial neural network, when executed on the computing unit (3), performs a pixel-wise semantic segmentation in addition to the instance segmentation. [4] System according to claim 3, wherein the primary pre-trained artificial neural network has an ILVI additional layer and is designed to model a pixel-related output value of the primary predefined artificial neural network as a Dirichlet distribution during pixel-wise semantic segmentation and to determine a respective pixel-related uncertainty value from respective concentration parameters of the respective Dirichlet distribution, and to determine the respective object-related first uncertainty value and second uncertainty value from the plurality of pixel-related uncertainty values. [5] System according to claim 4, wherein the computing unit (3) is designed to carry out a classification into known objects (5) and unknown objects (5) by means of the concentration parameters of the respective Dirichlet distribution, and to transmit only a respective region of the image data with unknown objects (5) to the secondary pre-trained artificial neural network for object recognition. [6] System according to one of the preceding claims, wherein the primary pre-trained artificial neural network is pre-trained with image data of traffic situations comprising a plurality of objects (5) within these traffic situations and an environment, and the secondary pre-trained artificial neural network is pre-trained with information about objects (5) without a traffic situation and without an environment. [7] System according to one of the preceding claims, wherein the computing unit (3) is designed to carry out a depth estimation for determining the distance between the vehicle and a respective object (5) by executing the primary pre-trained artificial neural network. [8] System according to one of the preceding claims, wherein the computing unit (3) is designed to assign a meta-annotation to objects (5) and to transmit the respective meta-annotation to an automatic driving control system. [9] Vehicle with a system according to one of the preceding claims, wherein the computing unit (3) of the system is designed to use the result of the environment detection for an automatic driving control function of the vehicle. [10] Method for supporting a driving control function of an automated vehicle by automated environment detection, comprising the steps: - Over a plurality of successive points in time: repeatedly determining (S1) image data about an environment of the vehicle by a sensor unit (1) and transmitting the image data to a computing unit (3); - At each of the points in time: determining (S2) a critical distance of a minimum-risk maneuver based on known latency times, the current vehicle speed and an estimated braking deceleration on the current road course; - Carrying out (S3) a respective inference for the image data of each time point with a primary pre-trained artificial neural network for carrying out an instance segmentation by means of the computing unit (3), so that an instance list with instances of objects (5) and assigned classes, with distances of the objects (5) to the sensor unit (1), with assignment of a pixel mask for localization, with the associated number of pixels, and with a second uncertainty value regarding the assignment of an object (5) to a class is obtained; - eliminating (S4) instances from the instance list with a mean distance greater than a first predetermined limit, and / or with a number of pixels less than a predetermined second limit, and / or with a second uncertainty value less than a predetermined third limit; - Creating (S5) an object list, which is a list of recognized objects (5), by copying the instances of the instance list remaining after elimination with a second uncertainty value regarding the assignment of an object (5) to a class smaller than a predetermined fourth limit value into the object list, and by performing the following steps in a loop over all remaining object instances with a second uncertainty value greater than the fourth limit value: determining the smallest rectangular image section that completely contains an area consisting of the pixel mask of the respective instance and a predefined geometric overset, inference with a secondary pre-trained artificial neural network based on the image data of the area, and if a confidence value determined by the secondary artificial neural network exceeds a predetermined fifth limit value: copying this object (5) into the object list; and - Transfer (S6) the object list to the drive control function.

Images